Get ClickPipe

One of the following:

• ClickPipeKafkaSource{ … }

  View ClickPipeKafkaSource schema →

  12 properties

  • typeoptionalkafkaorredpandaormskorgcmkorconfluentorwarpstream+2 more

    Type of the Kafka source.

  • formatoptionalJSONEachRoworAvroorAvroConfluentorProtobuf

    Format of the Kafka source.

  • brokersoptionalstring

    Brokers of the Kafka source.

  • topicsoptionalstring

    Topics of the Kafka source.

  • consumerGroupoptionalstring | null

    Consumer group of the Kafka source. If not provided "clickpipes-<>" will be used.

    Example: "my-clickpipe-consumer-group"
  • authenticationoptionalPLAINorSCRAM-SHA-256orSCRAM-SHA-512orIAM_ROLEorIAM_USERorMUTUAL_TLS+1 more

    Authentication method of the Kafka source. SERVICE_ACCOUNT_WORKLOAD_IDENTITY is in Private Preview. ClickPipes uses the GCP service account returned in gcpWorkloadIdentity.principal by the operation with operationId clickPipesServiceContextGet; grant it access to the source resources. Supported authentication methods: kafka: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, MUTUAL_TLS, msk: SCRAM-SHA-512, IAM_ROLE, IAM_USER, MUTUAL_TLS, gcmk: PLAIN, MUTUAL_TLS, SERVICE_ACCOUNT_WORKLOAD_IDENTITY, confluent: PLAIN, MUTUAL_TLS, warpstream: PLAIN, azureeventhub: PLAIN, redpanda: SCRAM-SHA-256, SCRAM-SHA-512, MUTUAL_TLS, dokafka: SCRAM-SHA-256, MUTUAL_TLS

  • iamRoleoptionalstring | null

    IAM role for the Kafka source. Use with IAM role authentication. Read more in ClickPipes documentation: https://clickhouse.com/docs/en/integrations/clickpipes/kafka#iam

    Example: "arn:aws:iam::123456789012:role/MyRole"
  • offsetoptionalClickPipeKafkaOffsetornull

    2 variants

    One of the following:

    • ClickPipeKafkaOffset
    • null
  • schemaRegistryoptionalClickPipeKafkaSchemaRegistryornull

    2 variants

    One of the following:

    • ClickPipeKafkaSchemaRegistry
    • null
  • caCertificateoptionalstring | null

    PEM encoded CA certificates to validate the broker's certificate.

  • reversePrivateEndpointIdsoptionalarray ofstring

    Reverse private endpoint UUIDs used for a secure private connection to the Kafka source.

  • exactlyOnceoptionalboolean | null

    Enable exactly-once delivery. Guarantees every Kafka record is inserted exactly once across restarts and rebalances. Can only be set at pipe creation.

• null

One of the following:

• ClickPipeObjectStorageSource{ … }

  View ClickPipeObjectStorageSource schema →

  14 properties

  • typeoptionals3orgcsordospacesorazureblobstorageorcloudflarer2orovhobjectstorage

    Type of the ObjectStorage source.

  • formatoptionalJSONEachRoworJSONAsObjectorCSVorCSVWithNamesorTabSeparatedorTabSeparatedWithNames+2 more

    Format of the files.

  • urloptionalstring

    Provide a path to the file(s) you want to ingest. You can specify multiple files using bash-like wildcards. For more information, see the documentation on using wildcards in path: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#limitations

    Example: "https://datasets-documentation.s3.eu-west-3.amazonaws.com/http/**.ndjson.gz"
  • delimiteroptionalstring | null

    Delimiter used in the files.

    Example: ","
  • compressionoptionalnoneorgziporgzorbrotliorbrorxz+3 more

    Compression algorithm used for the files.

    Example: "auto"
  • isContinuousoptionalboolean | null

    If set to true, the pipe will continuously read new files from the source. If set to false, the pipe will read the files only once. New files have to be uploaded lexically order.

  • queueUrloptionalstring | null

    Queue URL for event-based continuous ingestion. For S3, provide an SQS queue URL. For GCS, provide a Pub/Sub subscription (e.g. projects/{project}/subscriptions/{name}). When provided, files are ingested based on event notifications rather than lexicographical order. Only applicable when isContinuous is true and authentication is not public.

    Example: "https://sqs.us-east-1.amazonaws.com/123456789012/MyQueue"
  • skipInitialLoadoptionalboolean | null

    If set to true, skips the initial load and only ingests files delivered by queue notifications. Only applicable when queueUrl is provided.

  • startAfteroptionalstring | null

    Skip all files up to and including this object key during the initial load. Cannot be provided when skipInitialLoad is true.

    Example: "events/2026-06-01/"
  • authenticationoptionalIAM_ROLEorIAM_USERorCONNECTION_STRINGorSERVICE_ACCOUNTorSERVICE_ACCOUNT_WORKLOAD_IDENTITY

    Authentication method. IAM_USER is for S3, GCS, and DigitalOcean Spaces. IAM_ROLE is for S3 only. SERVICE_ACCOUNT is for GCS only. For GCS, SERVICE_ACCOUNT_WORKLOAD_IDENTITY is in Private Preview. ClickPipes uses the GCP service account returned in gcpWorkloadIdentity.principal by the operation with operationId clickPipesServiceContextGet; grant it access to the source resources. CONNECTION_STRING is for Azure Blob Storage. PUBLIC uses no authentication.

  • iamRoleoptionalstring | null

    IAM role to be used with IAM role authentication. Read more in ClickPipes documentation: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#authentication

    Example: "arn:aws:iam::123456789012:role/MyRole"
  • connectionStringoptionalstring | null

    Connection string for Azure Blob Storage authentication. Required when authentication is CONNECTION_STRING.

    Example: "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net"
  • pathoptionalstring | null

    Path to the file(s) within the Azure container. Used for Azure Blob Storage sources. You can specify multiple files using bash-like wildcards. For more information, see the documentation on using wildcards in path: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#limitations

    Example: "data/logs/*.json"
  • azureContainerNameoptionalstring | null

    Container name for Azure Blob Storage. Required when type is azureblobstorage.

    Example: "mycontainer"
• null

One of the following:

• ClickPipeKinesisSource{ … }

  View ClickPipeKinesisSource schema →

  8 properties

  • formatoptionalJSONEachRoworAvroorAvroConfluentorProtobuf

    Format of the Kinesis stream.

  • streamNameoptionalstring

    Name of the Kinesis stream.

    Example: "my-stream"
  • regionoptionalstring

    AWS region of the Kinesis stream.

    Example: "us-east-1"
  • useEnhancedFanOutoptionalboolean | null

    Use enhanced fan-out for the Kinesis stream.

  • iteratorTypeoptionalTRIM_HORIZONorLATESTorAT_TIMESTAMP

    Type of iterator to use when reading from the Kinesis stream. If AT_TIMESTAMP is used, the timestamp field must be provided.

  • timestampoptionalinteger | null

    UNIX timestamp to start reading from the Kinesis stream. Required if iteratorType is AT_TIMESTAMP.

    Example: 1615766400
  • authenticationoptionalIAM_ROLEorIAM_USER

    Authentication method to use with the Kinesis stream.

  • iamRoleoptionalstring | null

    IAM role to use for authentication. Required if IAM_ROLE is used.

    Example: "arn:aws:iam::123456789012:role/MyRole"
• null

One of the following:

• ClickPipePubSubSource{ … }

  View ClickPipePubSubSource schema →

  9 properties

  • formatJSONEachRoworAvroorProtobufrequired

    Format of messages in the Pub/Sub topic. GCP Pub/Sub ClickPipes are in limited preview — contact support to enable this feature for your organization.

    Example: "JSONEachRow"
  • projectIdstringrequired

    GCP project ID that owns the Pub/Sub topic.

    Example: "my-gcp-project"
  • topicstringrequired

    Pub/Sub topic name (not the fully-qualified path).

    Example: "my-topic"
  • authenticationSERVICE_ACCOUNTorSERVICE_ACCOUNT_WORKLOAD_IDENTITYrequired

    Authentication method to use with GCP Pub/Sub. SERVICE_ACCOUNT_WORKLOAD_IDENTITY is in Private Preview. ClickPipes uses the GCP service account returned in gcpWorkloadIdentity.principal by the operation with operationId clickPipesServiceContextGet; grant it access to the source resources.

    Example: "SERVICE_ACCOUNT"
  • seekTypelatestorearliestortimestamprequired

    Starting position strategy for consuming the subscription. The seekTimestamp companion is required only when seekType is "timestamp"; setting it for a mismatched seek type is rejected.

    Example: "earliest"
  • seekTimestampoptionalstring | null

    RFC 3339 / ISO 8601 timestamp to seek to. Required when seekType is "timestamp"; must be omitted otherwise.

    format: date-time

    Example: "2026-04-10T12:00:00Z"
  • filteroptionalstring | null

    Optional Pub/Sub subscription filter expression (CEL). Maximum 256 characters.

    maxLength: 256
  • enableOrderingoptionalboolean | null

    Whether to enable ordered delivery of messages (requires messages to be published with ordering keys).

  • ackDeadlineoptionalinteger | null

    Acknowledgement deadline for messages, in seconds. Must be between 10 and 600.

    maximum: 600, minimum: 10
• null

One of the following:

• ClickPipePostgresSource{ … }

  View ClickPipePostgresSource schema →

  12 properties

  • typeoptionalpostgresorsupabaseorneonoralloydborplanetscaleorrdspostgres+5 more

    Type of the Postgres source. Defaults to "postgres" if not specified.

  • hostoptionalstring

    PostgreSQL server hostname or IP address. To use a reverse private endpoint, pass the endpoint hostname here.

    format: hostname

    Example: "my-postgres-server.example.com"
  • portoptionalinteger

    PostgreSQL server port.

    maximum: 65535, minimum: 1

    Example: 5432
  • databaseoptionalstring

    PostgreSQL database name to replicate from.

    Example: "production_db"
  • authenticationoptionalbasicorIAM_ROLE

    Authentication method for Postgres connection.

    Example: "IAM_ROLE"
  • iamRoleoptionalstring

    IAM role ARN for IAM authentication (required for IAM_ROLE authentication).

    Example: "arn:aws:iam::123456789012:role/MyApplicationRole"
  • tlsHostoptionalstring

    TLS/SSL host for secure connections.

    Example: "my-postgres-server.example.com"
  • caCertificateoptionalstring

    PEM encoded CA certificate to validate the Postgres server certificate.

    Example: "-----BEGIN CERTIFICATE-----\n..."
  • disableTlsoptionalboolean

    Disable TLS for the Postgres connection. Use with caution in production environments.

    Example: false
  • skipCertVerificationoptionalboolean

    Skip TLS certificate verification for the Postgres connection. Use with caution in production environments.

    Example: false
  • settingsoptionalobject{ … }

    11 properties

    • syncIntervalSecondsoptionalinteger

      Interval in seconds to sync data from Postgres during CDC replication.

      minimum: 1

      Example: 60
    • pullBatchSizeoptionalinteger

      Number of rows to pull in each batch during CDC replication.

      minimum: 1

      Example: 1000
    • publicationNameoptionalstring

      PostgreSQL publication name to use for CDC replication. If not provided, ClickPipes will create one automatically.

      Example: "clickpipes_publication"
    • replicationModeoptionalcdcorsnapshotorcdc_only

      Replication mode: "cdc" (change data capture with initial snapshot), "snapshot" (one-time snapshot only), or "cdc_only" (CDC without initial snapshot).

      Example: "cdc"
    • replicationSlotNameoptionalstring

      PostgreSQL replication slot name. Only valid for "cdc_only" mode. For "cdc" mode, ClickPipes creates the slot automatically.

      Example: "clickpipes_slot"
    • allowNullableColumnsoptionalboolean

      Preserve nullability from Postgres in the destination ClickHouse table. When true, columns without NOT NULL constraints are created as Nullable(...). When false, all columns are non-nullable and NULL values are replaced with the default value for the type. Note: Nullable types have performance overhead in ClickHouse.

      Example: false
    • initialLoadParallelismoptionalinteger

      Number of parallel workers to use per table in the initial snapshot phase.

      minimum: 1

      Example: 1
    • snapshotNumRowsPerPartitionoptionalinteger

      Number of rows per partition during the snapshot phase.

      minimum: 1000

      Example: 100000
    • snapshotNumberOfParallelTablesoptionalinteger

      Number of tables to snapshot in parallel during the initial load phase.

      minimum: 1

      Example: 1
    • enableFailoverSlotsoptionalboolean

      Enable failover support for the replication slot on PG17 and newer. Only applicable when ClickPipes creates the replication slot (i.e., replicationSlotName is NOT provided).

      Example: false
    • deleteOnMergeoptionalboolean

      Enable hard delete behavior in ReplacingMergeTree for PostgreSQL DELETE operations.

      Example: false
  • tableMappingsoptionalarray ofobject{ … }

    List of table mappings defining which PostgreSQL tables to replicate and how they map to ClickHouse tables.

    9 properties

    • sourceSchemaNameoptionalstring

      PostgreSQL source schema name.

      Example: "public"
    • sourceTableoptionalstring

      PostgreSQL source table name.

      Example: "users"
    • targetTableoptionalstring

      ClickHouse target table name, optionally prefixed with schema name (e.g., "my_schema_my_table"). The table will be created automatically if it does not exist. For snapshot mode, the target table must be empty.

      Example: "public_users"
    • excludedColumnsoptionalarray ofstring

      List of column names to exclude from replication. Column names must be unique within this list.

      Example: ["internal_id","temp_data"]
    • useCustomSortingKeyoptionalboolean

      Whether to use a custom sorting key. If true, sortingKeys must be provided. If false or omitted, the default sorting key is the PostgreSQL primary key.

      Example: false
    • sortingKeysoptionalarray ofstring

      Ordered list of column names to use as the sorting (ORDER BY) key in ClickHouse. Only used when useCustomSortingKey is true. Column names must be unique within this list.

      Example: ["created_at_date","event_id"]
    • tableEngineoptionalMergeTreeorReplacingMergeTreeorNull

      ClickHouse table engine: "ReplacingMergeTree" (handles updates/deletes), "MergeTree" (append-only), or "Null" (forward data to materialized views without storing it).

      Example: "ReplacingMergeTree"
    • partitionKeyoptionalstring

      Custom partitioning column used for parallel snapshotting. Only beneficial for PostgreSQL 13 (no benefit for PG14+, which supports indexed ctid scans). Must be an indexed column of type: smallint, integer, bigint, timestamp without time zone, or timestamp with time zone. Unrelated to ClickHouse partitioning.

      Example: "id"
    • partitionByExproptionalstring

      ClickHouse PARTITION BY expression applied to the destination table when ClickPipes creates it.

      Example: "toYYYYMM(created_at)"
• null

One of the following:

• ClickPipeMySQLSource{ … }

  View ClickPipeMySQLSource schema →

  12 properties

  • hoststringrequired

    MySQL server hostname or IP address. To use a reverse private endpoint, pass the endpoint hostname here.

    format: hostname

    Example: "my-mysql-server.example.com"
  • portintegerrequired

    MySQL server port.

    maximum: 65535, minimum: 1

    Example: 3306
  • settingsobject{ … }required

    10 properties

    • replicationModecdcorsnapshotorcdc_onlyrequired

      Replication mode: "cdc" (change data capture with initial snapshot), "snapshot" (one-time snapshot only), or "cdc_only" (CDC without initial snapshot).

      Example: "cdc"
    • syncIntervalSecondsoptionalinteger

      Interval in seconds to sync data from MySQL during CDC replication.

      minimum: 1

      Example: 60
    • pullBatchSizeoptionalinteger

      Number of rows to pull in each batch during CDC replication.

      minimum: 1

      Example: 1000
    • replicationMechanismoptionalGTIDorFILE_POS

      MySQL replication mechanism: "GTID" (Global Transaction Identifier) or "FILE_POS" (binary log file and position). Defaults to "GTID" if not specified. MariaDB supports "GTID" only. For "FILE_POS" on MySQL, contact support.

      Example: "GTID"
    • useCompressionoptionalboolean

      Enable compression for the MySQL connection.

      Example: false
    • allowNullableColumnsoptionalboolean

      Preserve nullability from MySQL in the destination ClickHouse table. When true, columns without NOT NULL constraints are created as Nullable(...). When false, all columns are non-nullable and NULL values are replaced with the default value for the type. Note: Nullable types have performance overhead in ClickHouse.

      Example: false
    • initialLoadParallelismoptionalinteger

      Number of parallel workers to use per table in the initial snapshot phase.

      minimum: 1

      Example: 1
    • snapshotNumRowsPerPartitionoptionalinteger

      Number of rows per partition during the snapshot phase.

      minimum: 1000

      Example: 100000
    • snapshotNumberOfParallelTablesoptionalinteger

      Number of tables to snapshot in parallel during the initial load phase.

      minimum: 1

      Example: 1
    • deleteOnMergeoptionalboolean

      Enable hard delete behavior in ReplacingMergeTree for MySQL DELETE operations.

      Example: false
  • tableMappingsarray ofobject{ … }required

    List of table mappings defining which MySQL tables to replicate and how they map to ClickHouse tables.

    9 properties

    • sourceSchemaNamestringrequired

      MySQL source database name.

      Example: "my_database"
    • sourceTablestringrequired

      MySQL source table name.

      Example: "users"
    • targetTablestringrequired

      ClickHouse target table name, optionally prefixed with schema name (e.g., "my_database_my_table"). The table will be created automatically if it does not exist. For snapshot mode, the target table must be empty.

      Example: "my_database_users"
    • excludedColumnsoptionalarray ofstring

      List of column names to exclude from replication. Column names must be unique within this list.

      Example: ["internal_id","temp_data"]
    • useCustomSortingKeyoptionalboolean

      Whether to use a custom sorting key. If true, sortingKeys must be provided. If false or omitted, the default sorting key is the MySQL primary key.

      Example: false
    • sortingKeysoptionalarray ofstring

      Ordered list of column names to use as the sorting (ORDER BY) key in ClickHouse. Only used when useCustomSortingKey is true. Column names must be unique within this list.

      Example: ["created_at_date","event_id"]
    • tableEngineoptionalMergeTreeorReplacingMergeTreeorNull

      ClickHouse table engine: "ReplacingMergeTree" (handles updates/deletes), "MergeTree" (append-only), or "Null" (forward data to materialized views without storing it).

      Example: "ReplacingMergeTree"
    • partitionKeyoptionalstring

      Custom partitioning column used for parallel snapshotting. Must be an indexed column of an integer, date, datetime or timestamp type. Unrelated to ClickHouse partitioning.

      Example: "id"
    • partitionByExproptionalstring

      ClickHouse PARTITION BY expression applied to the destination table when ClickPipes creates it.

      Example: "toYYYYMM(created_at)"
  • typeoptionalmysqlorrdsmysqlorauroramysqlormariadborrdsmariadb

    Type of the MySQL source. Defaults to "mysql" if not specified.

  • authenticationoptionalbasicorIAM_ROLE

    Authentication method for MySQL connection.

    Example: "basic"
  • iamRoleoptionalstring

    IAM role ARN for IAM authentication (required for IAM_ROLE authentication).

    Example: "arn:aws:iam::123456789012:role/MyApplicationRole"
  • tlsHostoptionalstring

    TLS/SSL host for secure connections.

    Example: "my-mysql-server.example.com"
  • caCertificateoptionalstring

    PEM encoded CA certificate to validate the MySQL server certificate.

    Example: "-----BEGIN CERTIFICATE-----\n..."
  • disableTlsoptionalboolean

    Disable TLS for the MySQL connection. Use with caution in production environments.

    Example: false
  • skipCertVerificationoptionalboolean

    Skip TLS certificate verification for the MySQL connection. Use with caution in production environments.

    Example: false
  • serverIdoptionalinteger

    Optional MySQL server_id the pipe declares itself as in the MySQL replication topology. Must be unique across replicas connected to the source. If omitted, one is assigned automatically.

    maximum: 4294967295, minimum: 1

    Example: 4242
• null

One of the following:

• ClickPipeBigQuerySource
• null

One of the following:

• ClickPipeMongoDBSource{ … }

  View ClickPipeMongoDBSource schema →

  8 properties

  • uristringrequired

    MongoDB connection URI. Supports both standard URIs (mongodb://...) and SRV URIs (mongodb+srv://...). Embedded credentials are redacted from API responses, so the returned value can differ from what was submitted.

    Example: "mongodb+srv://cluster0.example.mongodb.net/mydb"
  • readPreferenceprimaryorprimaryPreferredorsecondaryorsecondaryPreferredornearestrequired

    MongoDB read preference for replica set reads.

    Example: "secondaryPreferred"
  • tlsHostoptionalstring

    TLS/SSL host for secure connections.

    Example: "cluster0.example.mongodb.net"
  • disableTlsoptionalboolean

    Disable TLS for the MongoDB connection. Defaults to false (TLS enabled).

    Example: false
  • skipCertVerificationoptionalboolean

    Skip TLS certificate verification for the MongoDB connection. Use with caution in production environments.

    Example: false
  • caCertificateoptionalstring

    PEM encoded CA certificate to validate the MongoDB server certificate.

    Example: "-----BEGIN CERTIFICATE-----\n..."
  • settingsoptionalobject{ … }

    7 properties

    • replicationModecdcorsnapshotorcdc_onlyrequired

      Replication mode: "cdc" (change data capture with initial snapshot), "snapshot" (one-time snapshot only), or "cdc_only" (CDC without initial snapshot).

      Example: "cdc"
    • syncIntervalSecondsoptionalinteger

      Interval in seconds to sync data from MongoDB during CDC replication.

      minimum: 1

      Example: 60
    • pullBatchSizeoptionalinteger

      Number of rows to pull in each batch during CDC replication.

      minimum: 1

      Example: 100000
    • snapshotNumRowsPerPartitionoptionalinteger

      Number of rows per partition during the snapshot phase.

      minimum: 1000

      Example: 100000
    • snapshotNumberOfParallelTablesoptionalinteger

      Number of collections to snapshot in parallel during the initial load phase.

      minimum: 1

      Example: 1
    • deleteOnMergeoptionalboolean

      Enable hard delete behavior in ReplacingMergeTree for MongoDB DELETE operations.

      Example: false
    • useJsonNativeFormatoptionalboolean

      Store JSON values in native ClickHouse JSON format. When disabled, JSON data is stored as String.

      Example: true
  • tableMappingsoptionalarray ofobject{ … }

    List of collection mappings defining which MongoDB collections to replicate and how they map to ClickHouse tables.

    4 properties

    • sourceDatabaseNamestringrequired

      MongoDB source database name.

      Example: "mydb"
    • sourceCollectionstringrequired

      MongoDB source collection name.

      Example: "users"
    • targetTablestringrequired

      ClickHouse target table name. The table will be created automatically if it does not exist.

      Example: "mydb_users"
    • tableEngineoptionalMergeTreeorReplacingMergeTreeorNull

      ClickHouse table engine: "ReplacingMergeTree" (handles updates/deletes), "MergeTree" (append-only), or "Null" (forward data to materialized views without storing it).

      Example: "ReplacingMergeTree"
• null