Discover ClickPipe source schema

One of the following:

• ClickPipePostKafkaSource{ … }

  View ClickPipePostKafkaSource schema →

  14 properties

  • typeoptionalkafkaorredpandaormskorgcmkorconfluentorwarpstream+2 more

    Type of the Kafka source.

  • formatoptionalJSONEachRoworAvroorAvroConfluentorProtobuf

    Format of the Kafka source.

  • brokersoptionalstring

    Brokers of the Kafka source.

  • topicsoptionalstring

    Topics of the Kafka source.

  • consumerGroupoptionalstring | null

    Consumer group of the Kafka source. If not provided "clickpipes-<>" will be used.

    Example: "my-clickpipe-consumer-group"
  • authenticationoptionalPLAINorSCRAM-SHA-256orSCRAM-SHA-512orIAM_ROLEorIAM_USERorMUTUAL_TLS+1 more

    Authentication method of the Kafka source. SERVICE_ACCOUNT_WORKLOAD_IDENTITY is in Private Preview. ClickPipes uses the GCP service account returned in gcpWorkloadIdentity.principal by the operation with operationId clickPipesServiceContextGet; grant it access to the source resources. Supported authentication methods: kafka: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, MUTUAL_TLS, msk: SCRAM-SHA-512, IAM_ROLE, IAM_USER, MUTUAL_TLS, gcmk: PLAIN, MUTUAL_TLS, SERVICE_ACCOUNT_WORKLOAD_IDENTITY, confluent: PLAIN, MUTUAL_TLS, warpstream: PLAIN, azureeventhub: PLAIN, redpanda: SCRAM-SHA-256, SCRAM-SHA-512, MUTUAL_TLS, dokafka: SCRAM-SHA-256, MUTUAL_TLS

  • iamRoleoptionalstring | null

    IAM role for the Kafka source. Use with IAM role authentication. Read more in ClickPipes documentation: https://clickhouse.com/docs/en/integrations/clickpipes/kafka#iam

    Example: "arn:aws:iam::123456789012:role/MyRole"
  • offsetoptionalClickPipeKafkaOffsetornull

    2 variants

    One of the following:

    • ClickPipeKafkaOffset
    • null
  • schemaRegistryoptionalClickPipeMutateKafkaSchemaRegistryornull

    2 variants

    One of the following:

    • ClickPipeMutateKafkaSchemaRegistry
    • null
  • caCertificateoptionalstring | null

    PEM encoded CA certificates to validate the broker's certificate.

  • reversePrivateEndpointIdsoptionalarray ofstring

    Reverse private endpoint UUIDs used for a secure private connection to the Kafka source.

  • exactlyOnceoptionalboolean | null

    Enable exactly-once delivery. Guarantees every Kafka record is inserted exactly once across restarts and rebalances. Can only be set at pipe creation.

  • credentialsoptionalPLAINorMskIamUserorAzureEventHuborMutualTLS

    Credentials for Kafka source. Choose one that is supported by the authentication method.

    4 variants

    One of the following:

    • PLAIN
    • MskIamUser
    • AzureEventHub
    • MutualTLS
  • protobufSchemaoptionalstring

    Base64-encoded .proto source or serialized FileDescriptorSet. Supported only with Protobuf format and cannot be combined with schemaRegistry.

    maxLength: 1048576, minLength: 1

    Example: "c3ludGF4ID0gInByb3RvMyI7IG1lc3NhZ2UgRXZlbnQge30="
• null

One of the following:

• ClickPipePostKinesisSource{ … }

  View ClickPipePostKinesisSource schema →

  10 properties

  • formatoptionalJSONEachRoworAvroorAvroConfluentorProtobuf

    Format of the Kinesis stream.

  • streamNameoptionalstring

    Name of the Kinesis stream.

    Example: "my-stream"
  • regionoptionalstring

    AWS region of the Kinesis stream.

    Example: "us-east-1"
  • useEnhancedFanOutoptionalboolean | null

    Use enhanced fan-out for the Kinesis stream.

  • iteratorTypeoptionalTRIM_HORIZONorLATESTorAT_TIMESTAMP

    Type of iterator to use when reading from the Kinesis stream. If AT_TIMESTAMP is used, the timestamp field must be provided.

  • timestampoptionalinteger | null

    UNIX timestamp to start reading from the Kinesis stream. Required if iteratorType is AT_TIMESTAMP.

    Example: 1615766400
  • authenticationoptionalIAM_ROLEorIAM_USER

    Authentication method to use with the Kinesis stream.

  • iamRoleoptionalstring | null

    IAM role to use for authentication. Required if IAM_ROLE is used.

    Example: "arn:aws:iam::123456789012:role/MyRole"
  • accessKeyoptionalMskIamUserornull

    2 variants

    One of the following:

    • MskIamUser
    • null
  • protobufSchemaoptionalstring

    Base64-encoded .proto source or serialized FileDescriptorSet. Required with Protobuf format and not supported with other formats.

    maxLength: 1048576, minLength: 1

    Example: "c3ludGF4ID0gInByb3RvMyI7IG1lc3NhZ2UgRXZlbnQge30="
• null

One of the following:

• ClickPipePostPubSubSource
• null

One of the following:

• ClickPipePostObjectStorageSource{ … }

  View ClickPipePostObjectStorageSource schema →

  16 properties

  • typeoptionals3orgcsordospacesorazureblobstorageorcloudflarer2orovhobjectstorage

    Type of the ObjectStorage source.

  • formatoptionalJSONEachRoworJSONAsObjectorCSVorCSVWithNamesorTabSeparatedorTabSeparatedWithNames+2 more

    Format of the files.

  • urloptionalstring

    Provide a path to the file(s) you want to ingest. You can specify multiple files using bash-like wildcards. For more information, see the documentation on using wildcards in path: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#limitations

    Example: "https://datasets-documentation.s3.eu-west-3.amazonaws.com/http/**.ndjson.gz"
  • delimiteroptionalstring | null

    Delimiter used in the files.

    Example: ","
  • compressionoptionalnoneorgziporgzorbrotliorbrorxz+3 more

    Compression algorithm used for the files.

    Example: "auto"
  • isContinuousoptionalboolean | null

    If set to true, the pipe will continuously read new files from the source. If set to false, the pipe will read the files only once. New files have to be uploaded lexically order.

  • queueUrloptionalstring | null

    Queue URL for event-based continuous ingestion. For S3, provide an SQS queue URL. For GCS, provide a Pub/Sub subscription (e.g. projects/{project}/subscriptions/{name}). When provided, files are ingested based on event notifications rather than lexicographical order. Only applicable when isContinuous is true and authentication is not public.

    Example: "https://sqs.us-east-1.amazonaws.com/123456789012/MyQueue"
  • skipInitialLoadoptionalboolean | null

    If set to true, skips the initial load and only ingests files delivered by queue notifications. Only applicable when queueUrl is provided.

  • startAfteroptionalstring | null

    Skip all files up to and including this object key during the initial load. Cannot be provided when skipInitialLoad is true.

    Example: "events/2026-06-01/"
  • authenticationoptionalIAM_ROLEorIAM_USERorCONNECTION_STRINGorSERVICE_ACCOUNTorSERVICE_ACCOUNT_WORKLOAD_IDENTITY

    Authentication method. IAM_USER is for S3, GCS, and DigitalOcean Spaces. IAM_ROLE is for S3 only. SERVICE_ACCOUNT is for GCS only. For GCS, SERVICE_ACCOUNT_WORKLOAD_IDENTITY is in Private Preview. ClickPipes uses the GCP service account returned in gcpWorkloadIdentity.principal by the operation with operationId clickPipesServiceContextGet; grant it access to the source resources. CONNECTION_STRING is for Azure Blob Storage. PUBLIC uses no authentication.

  • iamRoleoptionalstring | null

    IAM role to be used with IAM role authentication. Read more in ClickPipes documentation: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#authentication

    Example: "arn:aws:iam::123456789012:role/MyRole"
  • connectionStringoptionalstring | null

    Connection string for Azure Blob Storage authentication. Required when authentication is CONNECTION_STRING.

    Example: "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net"
  • pathoptionalstring | null

    Path to the file(s) within the Azure container. Used for Azure Blob Storage sources. You can specify multiple files using bash-like wildcards. For more information, see the documentation on using wildcards in path: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#limitations

    Example: "data/logs/*.json"
  • azureContainerNameoptionalstring | null

    Container name for Azure Blob Storage. Required when type is azureblobstorage.

    Example: "mycontainer"
  • accessKeyoptionalMskIamUserornull

    2 variants

    One of the following:

    • MskIamUser
    • null
  • serviceAccountKeyoptionalstring | null

    Base64-encoded GCP service account JSON key. Required when authentication is SERVICE_ACCOUNT.

• null