Deploy ClickHouse with FIPS Compliance (Government)

This tutorial covers deploying ClickHouse Private with FIPS 140-3 compliance on AWS. It builds on the standard AWS deployment, with stricter node architecture and AMI requirements, then adds the FIPS-specific Kubernetes configuration.

Prerequisites

Before starting this tutorial, you must have completed the following from tutorials/deploy-aws.md:

Step 1: IAM role for ECR access
Step 2: Container images copied to your ECR using the FIPS version tags below
Step 3: VPC created
Step 4: EKS cluster created
Step 5: FIPS node groups created using the x86_64 requirements below instead of the standard ARM64 recommendations
Step 6: S3 bucket and IAM roles created

FIPS node group requirements

Use x86_64 nodes for ClickHouse Server, ClickHouse Keeper, and the ClickHouse Operator. The cryptographic provider validation for the current Server and Keeper FIPS images does not include Linux on ARM64. A FIPS-enabled ARM node image does not extend that validation to the cryptographic module inside the ClickHouse containers.

AWS supports two x86_64 node image paths for this deployment:

Use BOTTLEROCKET_x86_64_FIPS, which is available directly as an EKS managed node group AMI type.
Build a custom EKS-optimized Amazon Linux 2023 x86_64 AMI with FIPS mode enabled, then select it through an EC2 launch template. AWS publishes AL2023_x86_64_STANDARD, but does not expose a separate AL2023_x86_64_FIPS managed node group AMI type. See the EKS AMI build scripts.

The Bottlerocket option avoids maintaining a custom AMI. Use the AL2023 option when your organization standardizes on AL2023 and already has a custom AMI build and patching process.

Node group	Architecture	FIPS node AMI options	Instance guidance	Kubernetes label
Server	x86_64	`BOTTLEROCKET_x86_64_FIPS` or custom FIPS-enabled AL2023 x86_64	x86 “d” series with local NVMe and a 1:4 vCPU-to-memory ratio; `m5d.16xlarge` is a compatible GovCloud example	`clickhouseGroup: server`
Keeper	x86_64	`BOTTLEROCKET_x86_64_FIPS` or custom FIPS-enabled AL2023 x86_64	`m8i.2xlarge` when available; `m7i.2xlarge`, `m6i.2xlarge`, or `m5.2xlarge` are compatible alternatives	`clickhouseGroup: keeper`
Operator	x86_64	`BOTTLEROCKET_x86_64_FIPS` or custom FIPS-enabled AL2023 x86_64	Minimum xlarge, using an x86-compatible instance type	No dedicated label required

Configure instance-store NVMe using the matching AL2023 or Bottlerocket tab in Step 5 of the standard AWS deployment. In the Helm command below, set SSD_CACHE_IS_ON_EMPTY_DIR=true for Bottlerocket, which binds instance-store NVMe to /var/lib/kubelet. Set it to false for AL2023, which mounts instance-store NVMe at /nvme/disk.

FIPS Version Tags

Use the following FIPS-specific image tags when copying artifacts:

Artifact	Tag
clickhouse-server	`26.4.1.2596-fips`
clickhouse-keeper	`26.2.1.258-fips`
clickhouse-operator	`main-1.22166.1-fips`
helm/clickhouse-operator-helm	`1.22166.1`
helm/onprem-clickhouse-cluster	`1.11.6`

S3 Bucket Naming Restriction

The S3 bucket name cannot contain any periods (.) when using FIPS. The S3 FIPS endpoints require virtual-host-style addressing, which does not work when the bucket name contains periods.

Step 1: Create FIPS Certificates

FIPS compliance requires TLS certificates generated using FIPS-approved algorithms in a FIPS-enabled environment. You need three sets of certificates:

Certificate	Private Key	Public Key	SAN Required
Server	`server.key`	`server.crt`	`*.c-${CLUSTER_NAME}-server-headless.ns-${CLUSTER_NAME}.svc.cluster.local`, `c-${CLUSTER_NAME}-server-any.ns-${CLUSTER_NAME}.svc.cluster.local`
Client	`client.key`	`client.crt`	(none)
Keeper	`keeper.key`	`keeper.crt`	`*.c-${CLUSTER_NAME}-keeper-headless.ns-${CLUSTER_NAME}.svc.cluster.local`

You also need a CA certificate (ca.crt and ca.key), either self-signed or from your organization’s certificate authority.

Requirements:

Minimum RSA 3072-bit key size
SHA-256 or higher signature algorithm
Generated in a FIPS-enabled environment
A unique set of certificates (excluding CA) must be created for each ClickHouse cluster

For the full certificate generation procedure using a Red Hat UBI8 container with FIPS crypto policy, see how-to/configure-fips-certificates.md.

Step 2: Install Kubernetes Prerequisites

Install VolumeSnapshot CRDs

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

Install StorageClass (Optional)

Only if you do not have a custom or existing StorageClass:

ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws
CHART_VERSION=1.11.6

helm install clickhouse-prerequisites \
    oci://$ECR_HOST/helm/onprem-clickhouse-cluster \
    --version=$CHART_VERSION \
    -n default \
    --set-json="storageClass.create=true" \
    --set-json="createCluster=false" \
    --set-json="serviceAccount.create=false"

Step 3: Install Operator

Note the FIPS ECR endpoint format: 0000000000.dkr-ecr-fips.us-west-2.on.aws (not the standard .amazonaws.com endpoint).

# update ECR_HOST as needed
ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws

# should use the version of the operator's helm chart, not of the operator itself (eg 1.22166.1, not main-1.22166.1)
OPERATOR_VERSION=1.22166.1

# image tag: the FIPS operator build (note the main- prefix and the -fips suffix).
# Always set this explicitly: the FIPS variant is selected by the tag, so an
# unset tag will not give you a FIPS build.
OPERATOR_IMAGE_TAG=main-1.22166.1-fips

# set AZs as determined by VPC subnets
AZ_LIST='["us-west-2a","us-west-2b","us-west-2c"]'

helm install clickhouse-operator \
   oci://$ECR_HOST/helm/clickhouse-operator-helm \
   --version=$OPERATOR_VERSION \
   --create-namespace \
   -n clickhouse-operator-system \
   --set-json="image.repository=\"$ECR_HOST/clickhouse-operator\"" \
   --set-json="image.tag=\"$OPERATOR_IMAGE_TAG\"" \
   --set-json='cilium.enabled=false' \
   --set-json='idleScalerEnabled=false' \
   --set-json='webhooks.enabled=false' \
   --set-json='operator.debug=true' \
   --set-json='operator.metricsScraper.enabled=false' \
   --set-json="operator.availabilityZones=$AZ_LIST" \
   --set-json='operator.featureFlags.backupOnASeparatePod=true' \
   --set-json='operator.featureFlags.serverCMEKEnabled=true'

Step 4: Create Certificate Secrets

Secrets must be installed in the same Kubernetes namespace where the ClickHouse cluster will run. If your cluster name is default-xx-01, the namespace will be ns-default-xx-01.

Ensure the namespace exists, then create the secrets. Do not change the secret names or key names.

CERT_DIR=.
CLUSTER_NAME=default-xx-01

# server
kubectl create secret generic -n ns-${CLUSTER_NAME} ${CLUSTER_NAME}-server-cert-secret \
  --from-file=ca.crt="$CERT_DIR/ca/ca.crt" \
  --from-file=server.crt="$CERT_DIR/server/server.crt" \
  --from-file=server.key="$CERT_DIR/server/server.key" \
  --from-file=client.crt="$CERT_DIR/client/client.crt" \
  --from-file=client.key="$CERT_DIR/client/client.key"

# keeper
kubectl create secret generic -n ns-${CLUSTER_NAME} ${CLUSTER_NAME}-keeper-cert-secret \
  --from-file=ca.crt="$CERT_DIR/ca/ca.crt" \
  --from-file=keeper.crt="$CERT_DIR/keeper/keeper.crt" \
  --from-file=keeper.key="$CERT_DIR/keeper/keeper.key"

Step 5: Deploy ClickHouse Cluster

Use the correct FIPS version of the endpoints for ECR and S3. FIPS endpoints can be found at https://aws.amazon.com/compliance/fips/.

Create the Admin Password Secret

The default user’s password is read from a Kubernetes Secret in the cluster’s namespace. Create it alongside the certificate secrets from Step 4, before installing the chart, because the operator reads it as soon as it reconciles the cluster.

CLUSTER_NAME=default-xx-01
PASSWORD_SECRET_NAME=clickhouse-admin-password

kubectl create secret generic $PASSWORD_SECRET_NAME -n ns-$CLUSTER_NAME --from-literal=password='p@$$w0rd'

Use the password itself, not a hash. Single-quote it so the shell does not expand characters such as $ in the value.

Deploy the Cluster

# update values below as needed,
# the ecr host should be the one in your account
# the s3 bucket should be the one created earlier
# the password secret is the one created above
CLUSTER_NAME=default-xx-01
PASSWORD_SECRET_NAME=clickhouse-admin-password
ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws
SA_IAM_ROLE_ARN=arn:aws:iam::0000000000:role/CH-S3-${CLUSTER_NAME}-uw2-00-Role

S3_BUCKET_REGION=us-gov-west-1
S3_BUCKET_URL=https://my-clickhouse-data.us-gov-west-1.amazonaws.com
S3_BUCKET_NAME=my-clickhouse-data

# s3 key prefix can use any UUID value, but must be unique for all clusters storing data in the bucket specified above
S3_KEY_PREFIX=ch-s3-$(uuidgen | tr '[:upper:]' '[:lower:]')

# these must match the x86_64 clickhouseGroup labels added to the node groups
SERVER_NG_GROUP=server
KEEPER_NG_GROUP=keeper

# these values should change depending on selected instance sizes
# be sure to take daemon set requirements into account when setting CPU and MEMORY values
SERVER_CPU=64
SERVER_MEMORY=256Gi
# Keeper: high-load profile; for moderate load 2-4 CPU / 8Gi also works (see the infrastructure reference)
KEEPER_CPU=4
KEEPER_MEMORY=16Gi

# Update the `verificationMode` field to control how TLS certificates are verified.
# Defaults to `none`. See Context.h in the ClickHouse source code for more details.
SERVER_VERIFICATION_MODE=none
KEEPER_VERIFICATION_MODE=none

# Currently, usage of `strict` is not recommended and you might encounter issues with the Operator.
# If you require mTLS cert auth, use `relaxed` (Teleport integration requires it)
# SERVER_VERIFICATION_MODE=relaxed
# KEEPER_VERIFICATION_MODE=relaxed

CHART_VERSION=1.11.6

# As a general rule, set cache size to 70-80% of the allocatable instance-store SSD capacity,
# accounting for DaemonSets and cloud-provider reserved disk space.
# The example below sizes the cache for the four 600 GB NVMe devices on m5d.16xlarge.
CACHE_DISK_SIZE_GI=1700Gi

# Select according to the Server node AMI:
# - Bottlerocket with instance-store NVMe bound to /var/lib/kubelet: true
# - AL2023 with instance-store NVMe mounted at /nvme/disk: false
SSD_CACHE_IS_ON_EMPTY_DIR=true

helm install $CLUSTER_NAME \
    oci://$ECR_HOST/helm/onprem-clickhouse-cluster \
    --version=$CHART_VERSION \
    -n ns-$CLUSTER_NAME \
    --create-namespace \
    --set-json="account.passwordSecretRef.name=\"$PASSWORD_SECRET_NAME\"" \
    --set-json="server.image.repository=\"$ECR_HOST/clickhouse-server\"" \
    --set-json="server.storage.s3.endpoint=\"$S3_BUCKET_URL\"" \
    --set-json="server.storage.s3.region=\"$S3_BUCKET_REGION\"" \
    --set-json="server.storage.s3.bucketName=\"$S3_BUCKET_NAME\"" \
    --set-json="server.storage.s3.keyPrefix=\"$S3_KEY_PREFIX\"" \
    --set-json='server.arm64=false' \
    --set-json="server.image.tag=\"26.4.1.2596-fips\"" \
    --set-json="server.podPolicy.nodeSelector.clickhouseGroup=\"$SERVER_NG_GROUP\"" \
    --set-json="server.openSSL.enabled=true" \
    --set-json="server.openSSL.required=true" \
    --set-json="server.openSSL.verificationMode=$SERVER_VERIFICATION_MODE" \
    --set-json="keeper.image.repository=\"$ECR_HOST/clickhouse-keeper\"" \
    --set-json='keeper.arm64=false' \
    --set-json="keeper.image.tag=\"26.2.1.258-fips\"" \
    --set-json="keeper.podPolicy.nodeSelector.clickhouseGroup=\"$KEEPER_NG_GROUP\"" \
    --set-json="keeper.openSSL.enabled=true" \
    --set-json="keeper.openSSL.required=true" \
    --set-json="keeper.openSSL.verificationMode=$KEEPER_VERIFICATION_MODE" \
    --set-json="serviceAccount.annotations={\"eks.amazonaws.com/role-arn\":\"$SA_IAM_ROLE_ARN\"}" \
    --set-json='server.tolerations=[{"effect":"NoSchedule","key":"clickhouse.com/do-not-schedule","operator":"Exists"}]' \
    --set-json='keeper.tolerations=[{"effect":"NoSchedule","key":"clickhouse.com/do-not-schedule","operator":"Exists"}]' \
    --set-json="server.ssdCacheConfiguration.cacheDiskSize=\"$CACHE_DISK_SIZE_GI\"" \
    --set-json="server.ssdCacheConfiguration.isOnEmptyDir=$SSD_CACHE_IS_ON_EMPTY_DIR" \
    --set-json="server.podPolicy.resources.limits.cpu=\"$SERVER_CPU\"" \
    --set-json="server.podPolicy.resources.limits.memory=\"$SERVER_MEMORY\"" \
    --set-json="server.podPolicy.resources.requests.cpu=\"$SERVER_CPU\"" \
    --set-json="server.podPolicy.resources.requests.memory=\"$SERVER_MEMORY\"" \
    --set-json="keeper.podPolicy.resources.limits.cpu=\"$KEEPER_CPU\"" \
    --set-json="keeper.podPolicy.resources.limits.memory=\"$KEEPER_MEMORY\"" \
    --set-json="keeper.podPolicy.resources.requests.cpu=\"$KEEPER_CPU\"" \
    --set-json="keeper.podPolicy.resources.requests.memory=\"$KEEPER_MEMORY\""

Key differences from the standard AWS deployment:

Setting	Standard	FIPS
Node architecture	ARM64 recommended on AWS	x86_64 required
Node AMI	Standard EKS-optimized AMI	`BOTTLEROCKET_x86_64_FIPS` or a custom FIPS-enabled x86_64 AMI
ECR endpoint	`0000000000.dkr.ecr.us-west-2.amazonaws.com`	`0000000000.dkr-ecr-fips.us-west-2.on.aws`
Server image tag	`26.4.1.2596`	`26.4.1.2596-fips`
Keeper image tag	`26.2.1.258`	`26.2.1.258-fips`
Operator image tag	`main-1.22166.1`	`main-1.22166.1-fips`
S3 endpoint	(default)	Explicit FIPS endpoint URL
`server.openSSL.enabled`	not set	`true`
`server.openSSL.required`	not set	`true`
`keeper.openSSL.enabled`	not set	`true`
`keeper.openSSL.required`	not set	`true`
Architecture flags	`server.arm64=true`, `keeper.arm64=true`	`server.arm64=false`, `keeper.arm64=false`
Cache configuration	`bytesPerGiRAM`	`cacheDiskSize` (explicit size); `isOnEmptyDir=true` for Bottlerocket or `false` for AL2023

Step 6: Verify Installation

Port-forward on TLS Port

With FIPS mode, ClickHouse uses TLS on port 9440 instead of the standard port 9000.

kubectl port-forward svc/c-default-xx-01-server-any 9440:9440 -n ns-default-xx-01

Create Client Configuration

Create a client-config.xml file to tell the ClickHouse client where to find the CA certificate. Update the path to your ca.crt file:

<config>
    <openSSL>
        <client>
            <caConfig>/path/to/certs/ca.crt</caConfig>
        </client>
    </openSSL>
</config>

Connect and Run a Query

clickhouse client --host localhost --port 9440 --user default --ask-password --secure --config=client-config.xml

Run a test query:

clickhouse-cloud :) select 1;

Expected output:

SELECT 1
Query id: 825591bf-a8e5-4995-ac9f-afb864854ba2
   ┌─1─┐
1. │ 1  │
   └───┘
1 row in set. Elapsed: 0.001 sec.

Important: Port 9000 (plaintext native) and port 8123 (plaintext HTTP) will refuse connections when openSSL.required=true. Use port 9440 (native TLS) or port 8443 (HTTPS) instead.

Next Steps

Teleport integration (optional): For secure database access via Teleport Enterprise, see Integrate Teleport with ClickHouse Private. This requires SERVER_VERIFICATION_MODE=relaxed and KEEPER_VERIFICATION_MODE=relaxed.