How to recover from a corrupt Keeper snapshot

Corrupt or bad ClickHouse Keeper snapshots can cause significant system instability, such as metadata inconsistencies, read-only states for tables, resource exhaustion, or failed backups. This article covers:

• What snapshots are and where to find them
• How the problem manifests
• Possible strategies for recovery and what each of them means

Overview of Keeper snapshots

What is a snapshot?

A snapshot is a serialized state of Keeper’s internal data (such as metadata about clusters, table coordination paths, and configurations) at a specific point in time. Snapshots are vital for resynchronizing Keeper nodes within a cluster, recovering metadata during failures, and start-up or restart processes that rely on a known-good Keeper state.

Where can I find snapshots?

Snapshots are stored as files on the local filesystem of Keeper nodes. By default, they are stored at /var/lib/clickhouse/coordination/snapshots/ or by the custom path specified by snapshot_storage_path in your keeper_server.xml file. Snapshots are named incrementally (e.g., snapshot.23), with newer ones having higher numbers.

For multi-node clusters, each Keeper node has its own snapshot directory.

Key symptoms and manifestations of corrupt Keeper snapshots

The table below details some common symptoms and manifestations of corrupt Keeper snapshots:

Log Indicators:

Before diagnosing snapshot corruption, check Keeper logs for specific error patterns:

Recovering from corrupt Keeper snapshots

Before touching any files, always:

1. Stop all Keeper nodes to prevent further corruption
2. Backup everything by copying the entire coordination directory to a safe location
3. Verify cluster quorum to ensure at least one node has good data

1. Restore from an existing backup

You should follow this process if:

• The Keeper metadata or snapshot corruption makes current data unsalvageable.
• A backup exists with a known-good Keeper state.

Follow the steps below to restore an existing backup:

1. Locate and validate the newest backup for metadata consistency.
2. Shut down the ClickHouse and Keeper services.
3. Replace the faulty snapshots and logs with those from the backup directory.
4. Restart the Keeper cluster and validate metadata synchronization.

2. Rollback to an older snapshot

You should follow this process when:

• Recent snapshots are corrupt, but older ones remain usable.
• Incremental logs are intact for consistent recovery.

Follow the steps below to roll back to an older snapshot:

1. Identify and select a valid older snapshot (e.g., snapshot.19) from the Keeper directory.
2. Remove newer snapshots and logs.
3. Restart Keeper so it replays logs to rebuild the metadata state.

3. Restore metadata using SYSTEM RESTORE REPLICA

You should follow this process when:

• Keeper metadata is lost or corrupted but table data still exists on disk
• Tables have switched to read-only mode due to missing ZooKeeper/Keeper metadata
• You need to recreate metadata in Keeper based on locally available data parts

Follow the steps below to restore metadata:

1. Verify that table data exists locally in your clickhouse-server data path, set by <path> in your config. (/var/lib/clickhouse/data/ by default)

2. For each affected table, execute:

SYSTEM RESTART REPLICA [db.]table_name;
SYSTEM RESTORE REPLICA [db.]table_name;

3. For database-level recovery (if using Replicated database engine):

SYSTEM RESTORE DATABASE REPLICA db_name;

4. Wait for synchronization to complete:

SYSTEM SYNC REPLICA [db.]table_name;

5. Verify recovery by checking system.replicas for is_readonly = 0 and monitoring system.detached_parts

4. Drop and recreate replica metadata in Keeper

You should follow this process when:

• The error occurs on a single replica of the cluster and has corrupt or inconsistent metadata in Keeper
• You encounter errors like “Part XXXXX intersects previous part YYYYY”
• You need to completely reset a replica’s Keeper metadata while preserving local data

Follow the steps below to drop and recreate metadata:

1. On the affected replica, detach the table:

DETACH TABLE [db.]table_name;

2. Remove the replica’s metadata from Keeper (execute on any replica):

SYSTEM DROP REPLICA 'replica_name' FROM ZKPATH '/clickhouse/tables/{shard}/table_name';

To find the correct ZooKeeper path:

SELECT zookeeper_path, replica_name FROM system.replicas WHERE table = 'table_name';

3. Reattach the table (it will be in read-only mode):

ATTACH TABLE [db.]table_name;

4. Restore the replica metadata:

SYSTEM RESTORE REPLICA [db.]table_name;

5. Synchronize with other replicas:

SYSTEM SYNC REPLICA [db.]table_name;

6. Check system.detached_parts on all replicas after recovery

Alternative: Using force_restore_data flag

For automatic recovery of all replicated tables at server startup:

1. Stop ClickHouse server
2. Create the recovery flag:

sudo -u clickhouse touch /var/lib/clickhouse/flags/force_restore_data

3. Start ClickHouse server
4. The server will automatically delete the flag and restore all replicated tables
5. Monitor logs for recovery progress

This approach is useful when multiple tables need recovery simultaneously.

5. Rebuild Keeper cluster

You should follow this process when:

• No valid snapshots, logs, or backups are available for recovery.
• You need to recreate the entire Keeper cluster and its metadata.

Follow the steps below to rebuild the Keeper cluster:

1. Fully stop the ClickHouse and Keeper clusters.
2. Reset each Keeper node by cleaning the snapshot and log directories.
3. Initialize one Keeper node as the leader and add other nodes incrementally.
4. Re-import metadata if available from external records.