Skip to content
ClickHouse Docs
ClickHouse DocsClickHouse Docs
49.6kGet Started

AWS PrivateLink setup to expose MSK for ClickPipes

View as MarkdownEdit this pageReport an issueSet up docs MCP

Overview

This guide will get you started with setting up a MSK multi-VPC to be used with ClickPipes reverse private endpoint.

ClickPipes owns the client VPC and creates the managed VPC connection. To establish the network connection, enable multi-VPC private connectivity on your MSK cluster and authorize the ClickPipes AWS account in the cluster policy.

Requirements

Your MSK cluster VPC must be located in one of our ClickPipes regions. See ClickPipes regions for the list of supported regions.

Enabling multi-VPC connectivity

  1. Navigate to the MSK cluster.
    • Choose “Clusters” from the left navigation pane in the Amazon MSK console.
    • Select the specific MSK cluster you want to configure for multi-VPC connectivity.
  2. Enable MSK multi-VPC connectivity
    • In the Connectivity tab, find the Multi-VPC connectivity section.
    • Click on Edit.
    • Enable the Turn-on MSK multi-VPC connectivity option.
    • Follow the instructions
  3. Add ClickPipes account principal into a cluster’s policy
    • Navigate to the Configuration tab.
    • Click on Edit in the Cluster policy section.
    • Include arn:aws:iam::072088201116:root in the IAM policy. Example: 
      {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::072088201116:root"
                ]
            },
            "Action": [
                "kafka:CreateVpcConnection",
                "kafka:GetBootstrapBrokers",
                "kafka:DescribeCluster",
                "kafka:DescribeClusterV2"
            ],
            "Resource": "<MSK_CLUSTER_ARN>"
        }
    ]
}

Enable your MSK authentication method for both the cluster and multi-VPC connectivity. The following example uses SASL/IAM; set scram = true instead to use SASL/SCRAM.

resource "aws_msk_cluster" "this" {
  # ... existing configuration ...

  client_authentication {
    sasl {
      iam = true
    }
  }

  broker_node_group_info {
    # ... existing configuration ...

    connectivity_info {
      vpc_connectivity {
        client_authentication {
          sasl {
            iam = true
          }
        }
      }
    }
  }
}

Authorize the ClickPipes AWS account in the MSK cluster policy:

resource "aws_msk_cluster_policy" "clickpipes" {
  cluster_arn = aws_msk_cluster.this.arn

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "ClickPipesMultiVpc"
      Effect    = "Allow"
      Principal = { AWS = "arn:aws:iam::072088201116:root" }
      Action = [
        "kafka:CreateVpcConnection",
        "kafka:GetBootstrapBrokers",
        "kafka:DescribeCluster",
        "kafka:DescribeClusterV2",
      ]
      Resource = aws_msk_cluster.this.arn
    }]
  })
}

If the cluster already has a policy, preserve its existing statements when adding the ClickPipes permissions.

Creating reverse private endpoint

Follow reverse private endpoint creation steps in the ClickPipes documentation.

Create the reverse private endpoint with the ClickHouse Terraform provider:

resource "clickhouse_clickpipes_reverse_private_endpoint" "msk" {
  service_id         = var.clickhouse_service_id
  description        = "MSK multi-VPC reverse private endpoint"
  type               = "MSK_MULTI_VPC"
  msk_cluster_arn    = aws_msk_cluster.this.arn
  msk_authentication = "SASL_IAM" # Or "SASL_SCRAM"

  depends_on = [aws_msk_cluster_policy.clickpipes]
}

The resource fields are immutable. Changing a field destroys and recreates the reverse private endpoint.

Navigation