Migrate to Shared Catalog

Shared Catalog is the new approach to managing metadata in ClickHouse Private instances. Instead of databases using the Replicated engine, which uses a DDL queue in ClickHouse Keeper to ensure DDL statements are applied on all replicas, the Shared database engine stores all metadata in Keeper itself, removing the need to sync DDL statements across replicas.

Shared Catalog provides several benefits:

• Consistent state across all replicas
• Statelessness of compute nodes (enables stateless Server nodes)
• Atomic CREATE TABLE ... AS SELECT
• Support for UNDROP
• RENAME/move TABLE between databases
• Fast, reliable replica bootstrapping (decreases wake, start and provisioning times)

However, it does require a migration if your instance wasn’t set up with Shared Catalog on the first deployment. We recommend you test the migration first on a test environment before migrating production instances.

What to expect during the migration

The migration is performed on a live service. Queries continue to be served throughout: the Operator replaces the Server pods one at a time, and each Server drains its in-flight queries before exiting.

Plan for reduced capacity while the roll is in progress, since one replica is unavailable at a time, and make sure clients reconnect and retry — existing connections are closed when a replica is replaced.

Migration requirements

Before the migration can begin, verify the following prerequisites:

1. At least 2 Server replicas

The Operator performs a rolling upgrade to migrate databases to the new Shared database engine. Set the server.replicas helm value to 2 or greater. For instances configured through compute-compute separation this applies to the parent and to every child.

2. All databases use the Replicated engine

All databases (except system) must use the Replicated database engine. Verify with:

SELECT DISTINCT
    name,
    engine
FROM clusterAllReplicas('default', system.databases)
WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema'))
  AND (engine != 'Replicated')

The query should return no rows.

3. All tables use non-replicated table engines

No tables (except in system) should use replicated or legacy table engines. Verify with:

SELECT
    hostName(),
    name,
    engine,
    database
FROM clusterAllReplicas('default', system.tables)
WHERE ((engine LIKE '%Replicated%') OR (engine IN ('Ordinary', 'Atomic')))
  AND (database != 'system')

The query should return no rows.

4. No detached tables exist

SELECT hostName(), * FROM clusterAllReplicas('default', system.detached_tables)

The query should return no rows.

5. Metadata persistent volumes are still enabled

featureFlags.disableMetadataPersistentVolumes must be false. The migration reads the existing Replicated databases from the metadata volumes of the Server pods, so those volumes have to still be in place — setting this flag to true makes the Servers stateless and removes them.

From chart version 1.1.245 onwards this flag defaults to true, so instances migrating from Database Replicated must set it explicitly:

featureFlags:
  disableMetadataPersistentVolumes: false

Keep it false for the whole migration, and also while rolling back. Moving to stateless Servers is a separate change: apply it as its own Helm upgrade once the migration is verified complete.

Once all requirements are met the migration can be started. If any requirements are unmet, reach out to ClickHouse support for assistance.

Remove ON CLUSTER from DDL queries

Shared Catalog keeps all metadata in Keeper and every replica reads the same catalog, so a DDL statement already applies to the whole instance and the clause has nothing left to do. Some statements are rejected outright while it is present:

ALTER DATABASE analytics ON CLUSTER default MODIFY COMMENT 'Analytics tables'

ALTER DATABASE analytics MODIFY COMMENT 'Analytics tables'

If query logging is enabled, this query groups the statements that mentioned ON CLUSTER over the last seven days, and shows which users and clients sent them:

SELECT
    normalized_query_hash,
    any(query) AS example,
    groupUniqArray(user) AS users,
    groupUniqArray(client_name) AS clients,
    groupUniqArray(http_user_agent) AS user_agents,
    count() AS queries,
    max(event_time) AS last_seen
FROM clusterAllReplicas('default', system.query_log)
WHERE (type = 'QueryStart')
  AND (is_initial_query = 1)
  AND (event_time > (now() - toIntervalDay(7)))
  AND (query ILIKE '% ON CLUSTER %')
GROUP BY normalized_query_hash
ORDER BY last_seen DESC

It matches on the query text, so read the example column before acting on a row: statements that only mention ON CLUSTER in a comment or a string literal show up too. Results are also bound by how long the log is kept, so treat this as a starting point rather than an inventory. Clients that run DDL occasionally, such as schema migrations, backup tooling and ad hoc admin scripts, still need a manual review. On a parent instance, substitute all_groups.default for default so the audit covers its children as well.

Performing the migration

The steps below are performed on each instance. Choose one:

• A single instance with no children — follow steps 1 to 4 in order.
• An instance configured through compute-compute separation — the same steps apply to the parent and to every child, but the order across instances matters and the parent does not reach Running until every child has been migrated. Read Migrating instances with compute-compute separation before you start.

1. Enable Shared Catalog and add the migration markers

Set both feature flags in the same Helm upgrade:

featureFlags:
  enableSharedCatalog: true
  migrateToSharedCatalog: true
  disableMetadataPersistentVolumes: false

Together these two flags:

• Add the shared_database_catalog.migration_from_database_replicated setting to ClickHouse, telling it to expect existing Replicated databases while running in Shared Catalog mode
• Add the clickhouse.com/start-shared-catalog-migration: "true" annotation to the ClickHouseCluster Kubernetes resource, notifying the Operator to migrate this instance

Once deployed, the Operator replaces the Server pods one at a time and changes the type of all databases to Shared.

2. Wait for the instance to reach its expected state

kubectl get clickhousecluster -n <namespace> <instance-name>

3. Remove the migration markers

Once the status is Running, set featureFlags.migrateToSharedCatalog to false in your Helm values. As all databases have been migrated to type Shared instead of Replicated, the migration settings are no longer relevant.

4. Verify all databases are migrated

Confirm all non-system databases use the Shared engine:

SELECT DISTINCT
    name,
    engine
FROM clusterAllReplicas('default', system.databases)
WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema'))
  AND (engine != 'Shared')

The query should return no rows. At this point your instance is migrated to the Shared Catalog.

Migrating instances with compute-compute separation

An instance and its children share one Keeper ensemble and one catalog, so the migration is a single operation across the whole group rather than one migration per instance. The Operator only treats the migration as complete once every replica of the parent and of every child is running Shared Catalog.

Two consequences shape the procedure:

• The parent must be migrated first. A child cannot begin until the parent has stopped replicated DDL processing, which happens at the start of the parent’s migration.
• The parent reports Degraded from the moment its own replicas are migrated until the last child completes. This is the expected intermediate state, not a failure.

ClickHouse enforces this independently of the Operator: a replica refuses to migrate away from the Replicated engine unless replicated DDL processing has been stopped on every replica in the group, and fails with Replicated DDL queries are enabled on some replicas, cannot perform migration from Database Replicated. Migrating a child before its parent, or only part of the group, therefore fails server-side as well.

Work through the group in this order.

1. Migrate the parent

Apply Enable Shared Catalog and add the migration markers to the parent. Do not remove the markers.

The parent will move to Degraded with the reason failed to sync tables and databases:

kubectl get clickhousecluster -n <namespace> <parent-name> \
  -o jsonpath='{.status.state}{"\t"}{.status.reason}{"\n"}'
# Degraded	failed to sync tables and databases

Confirm the parent’s own replicas have migrated before moving on. Connected to the parent, every row must report Shared:

SELECT hostName(), engine
FROM clusterAllReplicas('default', system.databases)
WHERE name = 'default'
ORDER BY hostName()

2. Migrate every child

Apply Enable Shared Catalog and add the migration markers to each child instance, one at a time. Do not remove the markers.

Each child also moves to Degraded while it waits for the group to converge. Confirm its replicas have migrated with the same query, connected to the child.

3. Wait for the group to converge

Once the last child has migrated, the parent enables replicated DDL processing again and moves to Running. The children follow.

kubectl get clickhouseclusters -A

Wait until the parent and every child report Running. Every replica in the group should now report Shared:

SELECT hostName(), engine
FROM clusterAllReplicas('all_groups.default', system.databases)
WHERE name = 'default'
ORDER BY hostName()

4. Remove the migration markers

Only now, set featureFlags.migrateToSharedCatalog to false on the parent and on every child, then run Verify all databases are migrated against all_groups.default to confirm the whole group is migrated.

Enable stateless Servers and reclaim metadata PVCs

Once the migration is complete, you can make the Server nodes stateless by setting featureFlags.disableMetadataPersistentVolumes to true in your Helm values. Server pods will start without persistent metadata volumes and rely on the Shared Catalog and DatabaseDisk for metadata.

1. Apply the flag

Set the flag in your Helm values and upgrade the release:

featureFlags:
  enableSharedCatalog: true
  disableMetadataPersistentVolumes: true

At this point the existing Server StatefulSets still carry volumeClaimTemplates. The Operator logs an immutable-field warning and does not rewrite them.

2. Recreate each Server StatefulSet, one replica at a time

Each Server replica runs in its own StatefulSet (one Pod per StatefulSet). For each replica, delete the StatefulSet with --cascade=orphan so the running Pod keeps serving traffic while the Operator recreates the StatefulSet without volumeClaimTemplates. Once the new StatefulSet exists, its rolling update replaces the Pod so it starts without the metadata PVC.

List the Server StatefulSets for the cluster:

kubectl get statefulsets \
  -l app.kubernetes.io/name=clickhouse-server,app=<instance-name>-server

For each StatefulSet, delete it while keeping the Pod running:

kubectl delete statefulset <sts-name> --cascade=orphan

Wait for the Operator to recreate the StatefulSet and for its Pod to become Ready before moving to the next replica:

kubectl rollout status statefulset/<sts-name>

Confirm the recreated StatefulSet has no volumeClaimTemplates:

kubectl get statefulset <sts-name> \
  -o jsonpath='{.spec.volumeClaimTemplates}{"\n"}'
# []

Confirm the running Pod no longer references a metadata PVC:

kubectl get pod <pod-name> \
  -o jsonpath='{range .spec.volumes[?(@.persistentVolumeClaim)]}{.persistentVolumeClaim.claimName}{"\n"}{end}'
# (empty)

Repeat for every Server replica.

3. Delete the orphaned PVCs

Once every Server Pod is Ready without a PVC, delete the leftover PersistentVolumeClaims:

# List the orphaned metadata PVCs.
kubectl get pvc -l app=<instance-name>-server

# Delete them.
kubectl delete pvc -l app=<instance-name>-server

Rollback

If any issues are encountered with the Shared Catalog, you can roll back to the previous state:

1. Remove the migration markers

Set featureFlags.migrateToSharedCatalog to false in your Helm values.

2. Remove the Shared Catalog settings

Set featureFlags.enableSharedCatalog to false in your Helm values. This will trigger the Operator to revert the changes.

3. Remove the Catalog in Keeper

Port-forward a connection to Keeper and start a keeper-client session (using clickhouse keeper-client).

Once the session is started, execute the following commands:

ls '/clickhouse/catalog'
# Output: migration_from_database_replicated_completed names references replicas uuids

rmr '/clickhouse/catalog'
# You are going to recursively delete path /clickhouse/catalog Continue?
# [y/n] y

This will clean up all Shared Catalog state from Keeper.

4. Verify rollback

Confirm all non-system databases are using the Replicated engine again:

SELECT DISTINCT
    name,
    engine
FROM clusterAllReplicas('default', system.databases)
WHERE (name NOT IN ('INFORMATION_SCHEMA', 'system', 'information_schema'))
  AND (engine != 'Replicated')

The query should return no rows.