Integrate Teleport with ClickHouse Private

Set up Teleport Enterprise for secure database access to ClickHouse Private clusters in government/FIPS environments. This guide covers deploying Teleport on EKS, configuring database access, SAML authentication, and certificate rotation.

Prerequisites

The FIPS/Government deployment has been completed with SERVER_VERIFICATION_MODE=relaxed and KEEPER_VERIFICATION_MODE=relaxed (required for Teleport mTLS)
Teleport Enterprise license
The server certificate SAN must include DNS.2 = c-${CLUSTER_NAME}-server-any.ns-${CLUSTER_NAME}.svc.${KUBERNETES_DOMAIN} (see Generate FIPS certificates)

Create a Node Group for Teleport

A dedicated node group for Teleport is optional. If you do not have Bottlerocket ARM FIPS nodes available, create a node group with the following configuration in the EKS console under Compute:

Name: fips-arm64
Node IAM role: create a new recommended role
Kubernetes taints:
- Key: group, Value: fips-arm64, Effect: NoSchedule
AMI type: Bottlerocket ARM FIPS
Instance types: m7g.large
Scaling: Desired 2, Min 2, Max 3

Install Teleport Cluster on EKS

Create Namespace and License Secret

kubectl create namespace teleport
kubectl label namespace teleport 'pod-security.kubernetes.io/enforce=baseline'

# Download the Teleport Enterprise license, then:
kubectl --namespace teleport create secret generic license --from-file=/Downloads/license.pem

Generate FIPS Certificates for Teleport

Create the following script (generate_fips_certs_in_container_for_teleport.sh):

#!/bin/bash
set -e

CLUSTER_FQDN=${CLUSTER_FQDN:-"teleport.example.com"}
NAMESPACE=${NAMESPACE:-"teleport"}
KUBERNETES_DOMAIN=${KUBERNETES_DOMAIN:-"cluster.local"}
COUNTRY=${COUNTRY:-"US"}
STATE=${STATE:-"YourState"}
CITY=${CITY:-"YourCity"}
ORG=${ORG:-"YourOrganization"}
ORG_UNIT=${ORG_UNIT:-"YourOrganizationalUnit"}
CN=${CN:-"YourRootCA"}

# Only run inside a container, prevent accidental execution on host.
if [ ! -f /.dockerenv ]; then
    echo "Not running inside Docker"
    exit 1
fi

dnf -y install openssl crypto-policies-scripts
fips-mode-setup --enable

mkdir -p /certs/{ca_teleport,server_teleport}

# Generate CA certificate
cd /certs/ca_teleport
openssl genrsa -out ca.key 3072

cat > ca.cnf << EOF
[ req ]
distinguished_name = req_distinguished_name
req_extensions     = v3_ca
prompt             = no

[ req_distinguished_name ]
C  = ${COUNTRY}
ST = ${STATE}
L  = ${CITY}
O  = ${ORG}
OU = ${ORG_UNIT}
CN = ${CN}

[ v3_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
EOF

openssl req -new -x509 -days 3650 -key ca.key -out ca.crt -config ca.cnf

# Generate server certificate
cd /certs/server_teleport
cat > server.cnf << EOF
[ req ]
distinguished_name = req_distinguished_name
req_extensions     = v3_req
prompt             = no

[ req_distinguished_name ]
C  = ${COUNTRY}
ST = ${STATE}
L  = ${CITY}
O  = ${ORG}
OU = ${ORG_UNIT}
CN = clickhouse-server

[ v3_req ]
subjectKeyIdentifier = hash
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = ${CLUSTER_FQDN}
DNS.2 = teleport.${NAMESPACE}.svc.${KUBERNETES_DOMAIN}
EOF

openssl genrsa -out server.key 3072
openssl req -new -key server.key -out server.csr -config server.cnf
openssl x509 -req -days 365 -in server.csr -CA /certs/ca_teleport/ca.crt -CAkey /certs/ca_teleport/ca.key \
    -CAcreateserial -out server.crt -extensions v3_req -extfile server.cnf

echo 'Verifying FIPS compliance of generated certificates:'
openssl version
openssl rsa -in /certs/server_teleport/server.key -text -noout | grep 'Private-Key'
openssl x509 -in /certs/server_teleport/server.crt -text -noout | grep 'Signature Algorithm'
echo 'FIPS-compliant certificates have been generated successfully!'

Run it (replace <CLUSTER_FQDN> with the FQDN of the Teleport service):

export CLUSTER_FQDN=<CLUSTER_FQDN>
export NAMESPACE=teleport
export KUBERNETES_DOMAIN=cluster.local
export COUNTRY=US
export STATE=YourState
export CITY=YourCity
export ORG=YourOrganization
export ORG_UNIT=YourOrganizationalUnit
export CN=YourRootCA

docker run -it --rm \
    -e CLUSTER_FQDN=${CLUSTER_FQDN} \
    -e NAMESPACE=${NAMESPACE} \
    -e KUBERNETES_DOMAIN=${KUBERNETES_DOMAIN} \
    -e COUNTRY=${COUNTRY} \
    -e STATE=${STATE} \
    -e CITY=${CITY} \
    -e ORG=${ORG} \
    -e ORG_UNIT=${ORG_UNIT} \
    -e CN=${CN} \
    -v $(pwd):/certs \
    registry.access.redhat.com/ubi8/ubi \
    /certs/generate_fips_certs_in_container_for_teleport.sh

Create Certificate Secrets

kubectl create secret generic teleport-proxy-cert \
  --from-file=tls.crt=server_teleport/server.crt \
  --from-file=tls.key=server_teleport/server.key \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=ca_teleport/ca.crt \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -

Create Helm Values

Create eks_teleport_values.yaml. Replace the placeholders:

<CLUSTER_NAME>: The FQDN of the cluster
<VPN_SUBNETS>: VPN private subnets, comma-delimited (e.g., private-subnet-us-east-2a,private-subnet-us-east-2b,private-subnet-us-east-2c)
<BUCKET_NAME>: S3 bucket for Teleport session records
<NODE_SELECTOR>: Node selector value (e.g., fips-arm64)

chartMode: aws
clusterName: "<CLUSTER_NAME>"

enterprise: true
enterpriseImage: public.ecr.aws/gravitational/teleport-ent-fips-distroless
authentication:
  localAuth: false

service:
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    service.beta.kubernetes.io/aws-load-balancer-subnets: "<VPN_SUBNETS>"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

proxyListenerMode: multiplex
aws:
  region: us-east-2
  backendTable: teleport-helm-backend
  auditLogTable: teleport-helm-events
  auditLogMirrorOnStdout: false
  sessionRecordingBucket: <BUCKET_NAME>
  backups: true
  dynamoAutoScaling: false

highAvailability:
  replicaCount: 2

nodeSelector:
  eks.amazonaws.com/nodegroup: <NODE_SELECTOR>

publicAddr: [<CLUSTER_NAME>:443, teleport.teleport.svc.cluster.local:443]

tls:
  existingSecretName: teleport-proxy-cert
  existingCASecretName: teleport-proxy-ca

tolerations:
- key: "group"
  operator: "Equal"
  value: "teleport"
  effect: "NoSchedule"

podSecurityPolicy:
  enabled: false

log:
  level: INFO
  output: stderr

operator:
  enabled: true
  installCRDs: dynamic

serviceAccount:
  create: true
  name: "teleport"

rbac:
  create: true

This example deploys Teleport with FIPS on EKS. For other cloud providers, refer to the Teleport documentation.

Deploy Teleport Cluster

helm repo add teleport https://charts.releases.teleport.dev
helm repo update

helm upgrade --install teleport teleport/teleport-cluster \
  --namespace teleport \
  --values eks_teleport_values.yaml \
  --version 18.2.2

Get the Teleport Service External IP

kubectl get svc teleport -n teleport -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Take the external IP and create a CNAME record in Route53 or your DNS provider. If the load balancer does not have a hostname:

kubectl get svc teleport -n teleport -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Then create an A record instead.

Create a Teleport Provision Token

A provision token allows Teleport agents to join the cluster. Create teleport_provision_token_for_clickhouse.yaml:

apiVersion: resources.teleport.dev/v2
kind: TeleportProvisionToken
metadata:
  name: kube-agent-token
  namespace: teleport
spec:
  roles:
    - Db
  join_method: kubernetes
  kubernetes:
    type: in_cluster
    allow:
      - service_account: teleport-agent:teleport-kube-agent

kubectl apply -f teleport_provision_token_for_clickhouse.yaml

Set Up the Teleport Agent

Create Agent Namespace

kubectl create namespace teleport-agent
kubectl label namespace teleport-agent 'pod-security.kubernetes.io/enforce=baseline'

Create Agent Secrets

Create a secret for the Teleport proxy CA in the agent namespace:

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=ca_teleport/ca.crt \
  --namespace=teleport-agent \
  --dry-run=client -o yaml | kubectl apply -f-

Create a secret for the database CA (concatenate CAs if using multiple databases with different CAs). Create a file called db_cas.pem, then:

kubectl create secret generic db-ca \
  --from-file=ca.pem=db_cas.pem \
  --namespace=teleport-agent \
  --dry-run=client -o yaml | kubectl apply -f-

Configure Agent Values

Create teleport_agent_values.yaml. Replace:

<CLICKHOUSE_CLUSTER_NAME>: Name of the provisioned cluster (e.g., default-aa-01)
<ENV>: Static label for this database (e.g., dev)

roles: db
proxyAddr: teleport.teleport.svc.cluster.local:443
enterprise: true
enterpriseImage: public.ecr.aws/gravitational/teleport-ent-fips-distroless
joinParams:
  method: kubernetes
  tokenName: kube-agent-token
tls:
  existingCASecretName: teleport-proxy-ca
databases:
  - name: <CLICKHOUSE_CLUSTER_NAME>-https
    uri: c-<CLICKHOUSE_CLUSTER_NAME>-server-any.<CLICKHOUSE_CLUSTER_NAME>.svc.cluster.local:8443
    protocol: clickhouse-https
    tls:
      ca_cert_file: "/etc/teleport-tls-db/db-ca/ca.pem"
    static_labels:
      env: <ENV>
  - name: <CLICKHOUSE_CLUSTER_NAME>-native
    uri: c-<CLICKHOUSE_CLUSTER_NAME>-server-any.<CLICKHOUSE_CLUSTER_NAME>.svc.cluster.local:9440
    protocol: clickhouse
    tls:
      ca_cert_file: "/etc/teleport-tls-db/db-ca/ca.pem"
    static_labels:
      env: <ENV>
extraVolumes:
  - name: db-ca
    secret:
      secretName: db-ca
extraVolumeMounts:
  - name: db-ca
    mountPath: /etc/teleport-tls-db/db-ca
    readOnly: true

The YAML creates 2 database resources for 1 database because ClickHouse supports 2 protocols (HTTPS on 8443 and native TLS on 9440). Remove whichever you do not need.

Deploy the Agent

VERSION="18.2.2"
helm upgrade --install teleport-kube-agent teleport/teleport-kube-agent \
  --namespace teleport-agent \
  --version ${VERSION} \
  -f teleport_agent_values.yaml

Add Teleport CA to ClickHouse Trusted List

Fetch the Teleport root CA and add it to the ClickHouse server certificate secret:

kubectl exec -it -n teleport deployment/teleport-auth -- tctl auth export --type=db-client > teleport_ca.crt

cat teleport_ca.crt  # Ensure it is not empty

CLUSTER_NAME="<CLUSTER_NAME>"
NAMESPACE="ns-${CLUSTER_NAME}"

kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath="{.data.ca\.crt}" | base64 -d > current_ca.crt
cat current_ca.crt teleport_ca.crt > combined_ca.crt

kubectl create secret generic -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret \
  --from-file=server.crt=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.crt}' | base64 -d) \
  --from-file=server.key=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.key}' | base64 -d) \
  --from-file=client.crt=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.crt}' | base64 -d) \
  --from-file=client.key=<(kubectl get secret -n ${NAMESPACE} ${CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.key}' | base64 -d) \
  --from-file=ca.crt=combined_ca.crt \
  --namespace=${NAMESPACE} \
  --dry-run=client -o yaml | kubectl apply -f -

rm teleport_ca.crt current_ca.crt combined_ca.crt

Restart the ClickHouse server pods to apply the new CA.

Configure SAML Authentication

Create a Database Access Role

Create saml_clickhouse_user_role.yaml. Replace the user trait key with the attribute from your identity provider (e.g., http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress):

apiVersion: resources.teleport.dev/v1
kind: TeleportRoleV8
metadata:
  name: saml-clickhouse-user-dev-role
  namespace: teleport
spec:
  allow:
    db_labels:
      env: dev
    db_users:
      - '{{external["http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"]}}'
    db_names:
    - '*'

kubectl apply -f saml_clickhouse_user_role.yaml

For more information on roles, see the Teleport documentation.

Create a SAML Connector

Create saml_connector.yaml. Replace <DOMAIN> with your Route53 domain and <Entity Descriptor> with the value from your identity provider:

# Example Auth0 integration -- rename and adjust role mappings as needed
kind: TeleportSAMLConnector
apiVersion: resources.teleport.dev/v2
metadata:
  name: samlconnector
  namespace: teleport
spec:
  acs: https://teleport.<DOMAIN>/v1/webapi/saml/acs/samlconnector
  service_provider_issuer: https://teleport.<DOMAIN>
  display: Auth0
  attributes_to_roles:
  - name: http://schemas.xmlsoap.org/claims/Roles
    value: clickhouse-dev
    roles:
    - saml-clickhouse-user-dev-role
  - name: http://schemas.xmlsoap.org/claims/Roles
    value: admin
    roles:
    - access
    - editor
  entity_descriptor: <Entity Descriptor>

For more information on SAML connectors, see the Teleport documentation.

kubectl apply -f saml_connector.yaml

Create Database Users

Exec into a ClickHouse server pod:

kubectl exec -ti <CLICKHOUSE_SERVER_POD> -n ns-<CLUSTER_NAME> -- /bin/bash

Connect to the database:

cat <<EOF > client-config.xml
<config>
    <openSSL>
        <client>
            <caConfig>/etc/clickhouse-server/certs/ca.crt</caConfig>
        </client>
    </openSSL>
</config>
EOF
PASSWORD='My super secret p@$$w0rd'
clickhouse client --host localhost --port 9440 --password "$PASSWORD" --secure --config=client-config.xml

Create a user authenticated via SSL certificate CN (replace <DB_USER> with the username – typically the user’s email address from the identity provider):

CREATE USER '<DB_USER>' IDENTIFIED WITH ssl_certificate CN '<DB_USER>';
GRANT SELECT ON default.* TO '<DB_USER>';

Connect via Teleport

Add the Teleport CA to your system’s trusted CAs, or use the --insecure flag for development.

# Log in to Teleport
tsh login --proxy=teleport.<DOMAIN>:443 --auth=<AUTH_TYPE> --user=<TELEPORT_USER> teleport.<DOMAIN>

# Create a proxy tunnel
tsh proxy db --tunnel <CLICKHOUSE_CLUSTER_NAME>-native --db-user=<DB_USER> --port 59215 &

# Connect
clickhouse client --host localhost --port 59215 --user <DB_USER>

Replace:

<DOMAIN>: Your Route53 domain
<AUTH_TYPE>: Authentication type (e.g., samlconnector)
<TELEPORT_USER>: Teleport username
<DB_USER>: Database username
<CLICKHOUSE_CLUSTER_NAME>: ClickHouse cluster name (e.g., default-aa-01)

Day 2: Certificate Rotation

Teleport certificates expire and must be rotated before expiry.

Rotate Teleport Proxy Certificates

Generate new certificates using the script from Generate FIPS Certificates for Teleport. Ensure CLUSTER_FQDN and other variables match your cluster.

The script must run on a FIPS-enabled host. Docker automatically mounts /proc, so as long as the container runs on a FIPS-enabled host, the generated certificates will be FIPS-compliant.

Append the new CA to the existing bundle in both namespaces:

kubectl get secret teleport-proxy-ca -n teleport -o jsonpath='{.data.ca\.pem}' | base64 -d > current_teleport_ca.pem
cat current_teleport_ca.pem ca_teleport/ca.crt > combined_ca.pem

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=combined_ca.pem \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=combined_ca.pem \
  --namespace=teleport-agent \
  --dry-run=client -o yaml | kubectl apply -f -

Rolling restart Teleport:

kubectl rollout restart deployment -n teleport
kubectl rollout restart deployment -n teleport-agent

Add the new Teleport CA to ClickHouse trusted list:

kubectl exec -it -n teleport deployment/teleport-auth -- tctl auth export --type=db-client > teleport_ca.crt

CLICKHOUSE_CLUSTER_NAME="<CLICKHOUSE_CLUSTER_NAME>"
NAMESPACE="ns-${CLICKHOUSE_CLUSTER_NAME}"

kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath="{.data.ca\.crt}" | base64 -d > current_ca.crt
cat current_ca.crt teleport_ca.crt > combined_ca.crt

kubectl create secret generic -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret \
  --from-file=server.crt=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.crt}' | base64 -d) \
  --from-file=server.key=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.server\.key}' | base64 -d) \
  --from-file=client.crt=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.crt}' | base64 -d) \
  --from-file=client.key=<(kubectl get secret -n ${NAMESPACE} ${CLICKHOUSE_CLUSTER_NAME}-server-cert-secret -o jsonpath='{.data.client\.key}' | base64 -d) \
  --from-file=ca.crt=combined_ca.crt \
  --namespace=${NAMESPACE} \
  --dry-run=client -o yaml | kubectl apply -f -

rm -f teleport_ca.crt current_ca.crt combined_ca.crt

kubectl rollout restart statefulset -n ${NAMESPACE}

Replace the proxy certificate:

kubectl create secret generic teleport-proxy-cert \
  --from-file=tls.crt=server_teleport/server.crt \
  --from-file=tls.key=server_teleport/server.key \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart deployment -n teleport

Remove the old CA from the bundle:

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=ca_teleport/ca.crt \
  --namespace=teleport \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic teleport-proxy-ca \
  --from-file=ca.pem=ca_teleport/ca.crt \
  --namespace=teleport-agent \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart deployment -n teleport
kubectl rollout restart deployment -n teleport-agent

Clean up:

rm -f current_teleport_ca.pem combined_ca.pem

Reference

Teleport Documentation