Deploy ClickHouse on Bare Metal

This tutorial walks you through deploying ClickHouse Private on your own bare-metal or on-premises Kubernetes cluster. By the end, you will have a running ClickHouse cluster with local NVMe SSD caching, MinIO AIStor-backed object storage, and the ClickHouse operator managing the deployment.

This guide does not assume a managed Kubernetes service or a specific Kubernetes distribution. It states the requirements your cluster must meet, then walks through the ClickHouse-specific setup. Any conformant Kubernetes distribution that satisfies Step 1, such as RKE2, kubeadm, or k3s, should work, though host-level factors outside Kubernetes (for example restrictive SELinux policies) can still affect a given environment.

For detailed infrastructure specifications, see reference/infrastructure-requirements.md. For an overview of how the operator works, see explanation/architecture.md.

Prerequisites

Before you begin, ensure you have the following tools installed:

• kubectl – compatible with your target Kubernetes version
• Helm v3.x
• skopeo – for copying container images between registries

You will also need:

• A Kubernetes cluster that meets the requirements in Step 1
• A private, OCI-compatible container registry reachable from the cluster, to hold the mirrored ClickHouse images and Helm charts
• Access to the ClickHouse Private ECR repository (<account-id-from-onboarding>.dkr.ecr.us-east-1.amazonaws.com). The source ECR account ID and pull credentials are provided by ClickHouse during onboarding.
• The component versions for the current release, also listed on Latest Release:
  • 26.2.1.525 – ClickHouse server image tag, to mirror into your registry
  • 26.2.1.258 – ClickHouse keeper image tag, to mirror into your registry
  • 1.20930.1 – Operator image and Helm chart tag
  • 1.8.7 – ClickHouse Cluster Helm Chart tag

Step 1: Kubernetes Cluster Requirements

ClickHouse Private runs on any conformant Kubernetes cluster that meets the requirements below. How you provision that cluster (bare-metal installer, kubeadm, RKE2, k3s, an existing on-premises platform, etc.) is up to you.

Cluster

Failure domains (availability zones)

ClickHouse Private uses three Keeper pods. For production HA, provide three independent failure domains and label the Keeper nodes with three distinct values of topology.kubernetes.io/zone. The operator spreads Keeper pods across the node label configured by operator.topologyKey, which defaults to topology.kubernetes.io/zone, with a maximum skew of one. Server replicas are likewise balanced across the zones you configure on the operator.

On bare metal, you assign these values yourself. Use separate power feeds, racks, or other real fault-isolation boundaries so that losing one domain does not take down the Keeper quorum.

• Use three distinct topology.kubernetes.io/zone values (for example zone-a, zone-b, zone-c).
• Place at least one Keeper node in each zone.
• Distribute Server nodes across the same three zones.

The zone label values you choose here are the ones you pass to the operator in Step 6 as operator.availabilityZones.

Node roles

Dedicate nodes to ClickHouse Server and Keeper, and keep at least one additional node available for the operator and cluster add-ons. Label and taint the ClickHouse nodes so the operator schedules pods correctly and nothing else lands on them.

Apply the labels and taints with kubectl label / kubectl taint, for example:

# Keeper node in zone-a
kubectl label node <keeper-node-a> clickhouseGroup=keeper topology.kubernetes.io/zone=zone-a --overwrite
kubectl taint node <keeper-node-a> clickhouse.com/do-not-schedule=true:NoSchedule --overwrite

# Server node in zone-a
kubectl label node <server-node-a> clickhouseGroup=server topology.kubernetes.io/zone=zone-a --overwrite
kubectl taint node <server-node-a> clickhouse.com/do-not-schedule=true:NoSchedule --overwrite

Repeat for zone-b and zone-c.

Step 2: Mirror Container Images

Copy the ClickHouse images and Helm charts from the ClickHouse ECR into your private registry using skopeo. The --all flag preserves all architectures (amd64, arm64).

The commands below refer to your registry as $REGISTRY_HOST (for example registry.internal:5000 or a project path on a hosted registry). Adjust the login step to match how your registry authenticates.

SOURCE_ECR_ACCOUNT_ID=<account-id-from-onboarding>
SOURCE_ECR_REPO=$SOURCE_ECR_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com
REGISTRY_HOST=registry.internal:5000

# log into the source ClickHouse ECR (requires the AWS credentials provided by ClickHouse)
aws ecr get-login-password --region us-east-1 | skopeo login --username AWS --password-stdin $SOURCE_ECR_REPO

# log into your private registry (authentication method may differ for your registry)
skopeo login $REGISTRY_HOST

# copy each image, be sure to include the --all flag
skopeo copy --all docker://$SOURCE_ECR_REPO/clickhouse-server:26.2.1.525 docker://$REGISTRY_HOST/clickhouse-server:26.2.1.525
skopeo copy --all docker://$SOURCE_ECR_REPO/clickhouse-keeper:26.2.1.258 docker://$REGISTRY_HOST/clickhouse-keeper:26.2.1.258
skopeo copy --all docker://$SOURCE_ECR_REPO/clickhouse-operator:main-1.20930.1 docker://$REGISTRY_HOST/clickhouse-operator:main-1.20930.1
skopeo copy --all docker://$SOURCE_ECR_REPO/helm/clickhouse-operator-helm:1.20930.1 docker://$REGISTRY_HOST/helm/clickhouse-operator-helm:1.20930.1
skopeo copy --all docker://$SOURCE_ECR_REPO/helm/onprem-clickhouse-cluster:1.8.7 docker://$REGISTRY_HOST/helm/onprem-clickhouse-cluster:1.8.7

The nodes pull these images directly from $REGISTRY_HOST, so the registry must be reachable from every ClickHouse node. If your registry requires authentication for pulls, create the operator namespace and its image pull secret now; the same secret is referenced by the operator install in Step 6, and the equivalent is created in the cluster namespace in Step 7.

kubectl create namespace clickhouse-operator-system

kubectl create secret docker-registry registry-credentials \
  --docker-server=$REGISTRY_HOST \
  --docker-username=<username> \
  --docker-password=<password> \
  -n clickhouse-operator-system

Step 3: Provision Local Storage

Two kinds of local storage are needed, and they are provisioned differently:

• Server filesystem cache – backed by local NVMe SSD on the Server nodes. This cache sits in front of object storage and is critical to query performance; see ClickHouse Filesystem Cache for Shared Merge Tree. Server pods are stateless by default (data lives in object storage), so the NVMe cache is the only local disk a Server node needs.
• Keeper data – a persistent volume for Keeper’s Raft log and snapshots, served by a StorageClass. Keeper fsyncs its Raft log before acknowledging committed writes, so storage latency (especially tail latency) directly affects coordination and DDL latency across the whole cluster. Use local NVMe for Keeper, as you would for the Server nodes. Capacity is not the constraint: 100 GiB per Keeper pod is ample.

Prepare local NVMe for the Server filesystem cache

On each Server node, mount the NVMe SSD at a consistent path – this guide uses /nvme/disk. If a node has more than one NVMe device, combine them into a single RAID-0 array for the best throughput. Format as ext4 and mount with noatime,nobarrier, then make the mount persistent in /etc/fstab.

For example, with two NVMe devices:

# combine the NVMe devices into a RAID-0 array
sudo mdadm --create --verbose /dev/md0 \
  --level=0 \
  --chunk=512 \
  --raid-devices=2 \
  /dev/nvme1n1 /dev/nvme2n1   # devices from `nvme list`

# format and mount at /nvme/disk
sudo mkfs.ext4 /dev/md0
sudo mkdir -p /nvme/disk
# add to /etc/fstab using the device UUID from `blkid`, then mount:
#   UUID=<device UUID>  /nvme/disk  ext4  defaults,noatime,nobarrier  0 0
sudo mount -a

For the full rationale (RAID chunk size, filesystem options, sizing) see Configure and operate the filesystem cache.

The ClickHouse operator mounts this NVMe into the Server pods itself – it supports two volume types for the cache:

• HostPath – maps the host’s /nvme/disk directly into the pod. This is the default on AWS and the simplest fit for bare metal, since you have already mounted the NVMe at a host path.
• EmptyDir – backed by the node’s local ephemeral storage; requires the kubelet’s ephemeral storage to live on the NVMe.

This guide uses HostPath at /nvme/disk; you configure it on the cluster in Step 7 via server.ssdCacheConfiguration. For a production setup we strongly recommend using EmptyDir mode, as Local Ephemeral Storage reduces the attack surface and long term is operationally simpler.

For the trade-offs between the two modes, see EmptyDir vs HostPath.

Provide a StorageClass for Keeper

Keeper stores its data on a persistent volume provisioned by a StorageClass. Pick any StorageClass whose PVs include a zone label (topology.kubernetes.io/zone, or a label matching topology.*zone / topology.*node) in their node affinity. The ClickHouse operator uses that zone information from Keeper PVs to enforce Keeper’s cross-zone topology spread; a StorageClass whose PVs only carry kubernetes.io/hostname affinity (as stock Rancher local-path-provisioner does) is not enough on its own.

Zone-aware options on bare metal include:

• A cloud CSI driver, if the cluster is running in a cloud (for example the AWS EBS CSI driver with a gp3 StorageClass)
• A zone-aware storage layer such as OpenEBS, Rook, or Longhorn

You reference this StorageClass by name for Keeper storage in Step 7.

Step 4: Deploy Object Storage

ClickHouse Private stores all of its data in S3-compatible object storage. On bare metal you provide this yourself. The recommended on-premises option is MinIO AIStor, MinIO’s commercial object store; any other S3-compatible object store (on-prem or external) works as well, as long as it is reachable from the cluster.

Deploy MinIO AIStor

Install AIStor following MinIO’s documentation. A few points specific to this deployment:

• Provision AIStor for performance. ClickHouse performance depends directly on the object store: if AIStor cannot sustain the load, neither can ClickHouse. Give it high network throughput and performant disks.
• Expose an S3 API endpoint reachable from the ClickHouse pods (in-cluster service DNS or an ingress/load balancer). If you terminate TLS on the endpoint, the CA must be trusted by the ClickHouse pods.
• Ensure it is configured for strict read-after-write consistency.
• Avoid object versioning as this can leave data on MinIO AIStor even if it is deleted in ClickHouse.
• Size it for your dataset plus growth; ClickHouse retains all primary data here.

If you already operate Minio AIStor, skip the install and use its endpoint.

Create a bucket and credentials

ClickHouse authenticates to the object store with a static access key / secret key pair. Provision the following and record the values for Step 7:

Step 5: Install Kubernetes Prerequisites

Install VolumeSnapshot CRDs

These CRDs are required by the ClickHouse operator. If your cluster does not already provide them, install them from the external-snapshotter project:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

In a disconnected environment, mirror these manifests alongside your other artifacts and apply them from your internal source.

Step 6: Install the Operator

Log into your registry from Helm if it requires authentication, then install the operator. Set the availability zones to the topology.kubernetes.io/zone labels you assigned in Step 1.

REGISTRY_HOST=registry.internal:5000

# operator Helm chart tag (eg 1.20930.1, not main-1.20930.1)
OPERATOR_VERSION=1.20930.1

# operator image tag -- the build itself (note the main- prefix)
OPERATOR_IMAGE_TAG=main-1.20930.1

# the zones you labeled nodes with in Step 1
ZONES='["zone-a","zone-b","zone-c"]'

# authenticate Helm to your registry (method may differ for your registry)
helm registry login $REGISTRY_HOST

helm install clickhouse-operator \
   oci://$REGISTRY_HOST/helm/clickhouse-operator-helm \
   --version=$OPERATOR_VERSION \
   --create-namespace \
   -n clickhouse-operator-system \
   --set-json="image.repository=\"$REGISTRY_HOST/clickhouse-operator\"" \
   --set-json="image.tag=\"$OPERATOR_IMAGE_TAG\"" \
   --set-json='cilium.enabled=false' \
   --set-json='idleScalerEnabled=false' \
   --set-json='webhooks.enabled=false' \
   --set-json="operator.availabilityZones=$ZONES" \
   --set-json='imagePullSecrets=[{"name":"registry-credentials"}]'

The final imagePullSecrets flag references the pull secret created in Step 2. Omit it if your registry allows unauthenticated pulls.

Step 7: Deploy a ClickHouse Cluster

Naming Your Cluster

Each ClickHouse cluster needs a unique name within the Kubernetes cluster. Use the convention $DESCRIPTOR-$LETTERS-$ORDINAL:

• $DESCRIPTOR – descriptive name using letters only
• $LETTERS – reserved, use xx for simplicity
• $ORDINAL – incrementing ordinal starting with 01
• Example: default-xx-01

Create the Namespace and Secrets

Create the cluster namespace and, inside it, the object storage credentials from Step 4 and the registry pull secret from Step 2:

CLUSTER_NAME=default-xx-01
kubectl create namespace ns-$CLUSTER_NAME

# object storage credentials, referenced by the server and keeper env vars below
kubectl create secret generic object-storage-credentials \
  -n ns-$CLUSTER_NAME \
  --from-literal=access-key-id=<access-key-id> \
  --from-literal=secret-access-key=<secret-access-key>

# registry pull secret (skip if your registry allows unauthenticated pulls)
kubectl create secret docker-registry registry-credentials \
  --docker-server=$REGISTRY_HOST \
  --docker-username=<username> \
  --docker-password=<password> \
  -n ns-$CLUSTER_NAME

Storing the credentials in a Secret keeps them out of your shell history and the Helm release values.

Generate Password Hash and Deploy

# this will be the `default` user's password
PASSWORD='My super secret p@$$w0rd'
if command -v sha256sum &> /dev/null; then
  HASHED_PASSWORD=$(echo -n "$PASSWORD" | sha256sum | awk '{printf $1}' | base64 | tr -d '\n')
else
  HASHED_PASSWORD=$(echo -n "$PASSWORD" | shasum -a 256 | awk '{printf $1}' | base64 | tr -d '\n')
fi

# update values below as needed
CLUSTER_NAME=default-xx-01
REGISTRY_HOST=registry.internal:5000

# object storage (from Step 4)
S3_ENDPOINT=https://minio.<your-domain>
S3_BUCKET_NAME=my-clickhouse-data

# s3 key prefix can use any UUID value, but must be unique for all clusters storing data in the bucket
S3_KEY_PREFIX=ch-s3-$(uuidgen | tr '[:upper:]' '[:lower:]')

# these should match the clickhouseGroup labels you applied in Step 1
SERVER_NG_GROUP=server
KEEPER_NG_GROUP=keeper

# these values should change depending on your hardware.
# be sure to take DaemonSet requirements into account when setting CPU and MEMORY values
SERVER_CPU=64
SERVER_MEMORY=128Gi
# Keeper: high-load profile; for moderate load 2-4 CPU / 8Gi also works (see the infrastructure reference)
KEEPER_CPU=4
KEEPER_MEMORY=16Gi

# size the cache at 70-80% of the usable NVMe capacity mounted at /nvme/disk (Step 3)
CACHE_DISK_SIZE=1400Gi

# StorageClass for Keeper's persistent volume (from Step 3). Must produce PVs with a zone label in their node affinity.
KEEPER_STORAGE_CLASS=<your-zone-aware-storageclass>
KEEPER_STORAGE_SIZE=100Gi

CHART_VERSION=1.8.7

# S3 credentials are read from the object-storage-credentials Secret, shared by
# the server and keeper pods so both authenticate to the object store.
S3_CREDS_ENV='[
  {"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"object-storage-credentials","key":"access-key-id"}}},
  {"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"object-storage-credentials","key":"secret-access-key"}}}
]'

helm install $CLUSTER_NAME \
    oci://$REGISTRY_HOST/helm/onprem-clickhouse-cluster \
    --version=$CHART_VERSION \
    -n ns-$CLUSTER_NAME \
    --set-json="account.hashedPassword=\"$HASHED_PASSWORD\"" \
    --set-json='isOnPremiseInstance=true' \
    --set-json='imagePullSecrets=[{"name":"registry-credentials"}]' \
    --set-json="server.image.repository=\"$REGISTRY_HOST/clickhouse-server\"" \
    --set-json='server.arm64=false' \
    --set-json="server.storage.s3.bucketName=\"$S3_BUCKET_NAME\"" \
    --set-json="server.storage.s3.endpoint=\"$S3_ENDPOINT\"" \
    --set-json="server.storage.s3.keyPrefix=\"$S3_KEY_PREFIX\"" \
    --set-json="server.ssdCacheConfiguration.cacheDiskSize=\"$CACHE_DISK_SIZE\"" \
    --set-json="server.podPolicy.nodeSelector.clickhouseGroup=\"$SERVER_NG_GROUP\"" \
    --set-json="server.podPolicy.resources.limits.cpu=\"$SERVER_CPU\"" \
    --set-json="server.podPolicy.resources.limits.memory=\"$SERVER_MEMORY\"" \
    --set-json="server.podPolicy.resources.requests.cpu=\"$SERVER_CPU\"" \
    --set-json="server.podPolicy.resources.requests.memory=\"$SERVER_MEMORY\"" \
    --set-json="server.additionalEnvVars=$S3_CREDS_ENV" \
    --set-json='server.tolerations=[{"effect":"NoSchedule","key":"clickhouse.com/do-not-schedule","operator":"Exists"}]' \
    --set-json="keeper.image.repository=\"$REGISTRY_HOST/clickhouse-keeper\"" \
    --set-json='keeper.arm64=false' \
    --set-json="keeper.storage.storageClassName=\"$KEEPER_STORAGE_CLASS\"" \
    --set-json="keeper.storage.resources.requests=\"$KEEPER_STORAGE_SIZE\"" \
    --set-json="keeper.podPolicy.nodeSelector.clickhouseGroup=\"$KEEPER_NG_GROUP\"" \
    --set-json="keeper.podPolicy.resources.limits.cpu=\"$KEEPER_CPU\"" \
    --set-json="keeper.podPolicy.resources.limits.memory=\"$KEEPER_MEMORY\"" \
    --set-json="keeper.podPolicy.resources.requests.cpu=\"$KEEPER_CPU\"" \
    --set-json="keeper.podPolicy.resources.requests.memory=\"$KEEPER_MEMORY\"" \
    --set-json="keeper.additionalEnvVars=$S3_CREDS_ENV" \
    --set-json='keeper.tolerations=[{"effect":"NoSchedule","key":"clickhouse.com/do-not-schedule","operator":"Exists"}]'

Monitor the rollout (the operator creates server pods after keepers are healthy):

kubectl get pods -n ns-$CLUSTER_NAME -w

Step 8: Verify Installation

Port-forward the ClickHouse Service

kubectl port-forward svc/c-default-xx-01-server-any 9000:9000 -n ns-default-xx-01

This forwards port 9000 to your local machine.

Connect and Run a Query

clickhouse client --host localhost --port 9000 --password "$PASSWORD"

Run a simple query:

SELECT 1;

Expected output:

   ┌─1─┐
1. │ 1 │
   └───┘
1 row in set. Elapsed: 0.001 sec.

Confirm data reaches object storage

SELECT 1 only proves the server is running. Because the object store is something you provide on bare metal, verify that ClickHouse can actually write to and read from it: create a table, insert data, then drop the local filesystem cache so the next read must come from object storage.

CREATE TABLE default.connectivity_check (id UInt64, v String)
ENGINE = SharedMergeTree
ORDER BY id;

INSERT INTO default.connectivity_check
SELECT number, toString(number) FROM numbers(1000000);

-- Force the next read to come from object storage, not the local NVMe cache.
SYSTEM DROP FILESYSTEM CACHE;
SELECT count(), min(id), max(id) FROM default.connectivity_check;

A correct count after dropping the cache confirms the full write-then-read path through object storage. You can also list the objects under your bucket and keyPrefix with your object store’s client or console to see the parts ClickHouse wrote.

Next Steps

• FIPS / government compliance: See tutorials/deploy-government.md to apply FIPS 140-3 certificates and TLS configuration on top of this infrastructure.
• Compute-Compute separation: See how-to/configure-compute-compute-separation.md to set up multiple compute groups with separate endpoints sharing a single dataset.
• Management API: See tutorials/install-api.md to install the optional Private API for backups and scaling operations.
• Monitoring and alerting: See how-to/configure-alerting.md to set up alerting for your deployment.
• Troubleshooting: See troubleshooting.md for common issues and solutions.