system.events
The data in this system table is held locally on each node in ClickHouse Cloud. Obtaining a complete view of all data, therefore, requires the clusterAllReplicas function. See here for further details.
Contains information about the number of events that have occurred in the system. For example, in the table, you can find how many SELECT queries were processed since the ClickHouse server started.
Columns:
event(String) — Event name.value(UInt64) — Number of events occurred.description(String) — Event description.
Example
Event descriptions
ACMEAPIRequests
Number of ACME API requests issued.
ACMECertificateOrders
Number of ACME certificate orders issued.
AIORead
Number of reads with Linux or FreeBSD AIO interface
AIOReadBytes
Number of bytes read with Linux or FreeBSD AIO interface
AIOWrite
Number of writes with Linux or FreeBSD AIO interface
AIOWriteBytes
Number of bytes written with Linux or FreeBSD AIO interface
ASTFuzzerQueries
Number of fuzzed queries attempted by the server-side AST fuzzer.
AddressesDiscovered
Total count of new addresses in DNS resolve results for HTTP connections
AddressesExpired
Total count of expired addresses which is no longer presented in DNS resolve results for HTTP connections
AddressesMarkedAsFailed
Total count of addresses which have been marked as faulty due to connection errors for HTTP connections
AggregatingSortedMilliseconds
Total time spent while aggregating sorted columns
AggregationHashTablesInitializedAsTwoLevel
How many hash tables were inited as two-level for aggregation.
AggregationOptimizedEqualRangesOfKeys
For how many blocks optimization of equal ranges of keys was applied
AggregationPreallocatedElementsInHashTables
How many elements were preallocated in hash tables for aggregation.
AnalyzePatchRangesMicroseconds
Total time spent analyzing index of patch parts
ApplyPatchesMicroseconds
Total time spent applying patch parts to blocks
ArenaAllocBytes
Number of bytes allocated for memory Arena (used for GROUP BY and similar operations)
ArenaAllocChunks
Number of chunks allocated for memory Arena (used for GROUP BY and similar operations)
AsyncInsertBytes
Data size in bytes of asynchronous INSERT queries.
AsyncInsertCacheHits
Number of times a duplicate hash id has been found in asynchronous INSERT hash id cache.
AsyncInsertQuery
Same as InsertQuery, but only for asynchronous INSERT queries.
AsyncInsertRows
Number of rows inserted by asynchronous INSERT queries.
AsyncLoaderWaitMicroseconds
Total time a query was waiting for async loader jobs.
AsyncLoggingConsoleDroppedMessages
How many messages have been dropped from the console log due to the async log queue being full
AsyncLoggingConsoleTotalMessages
How many messages (accepted or dropped) have been sent to the async queue for the console log
AsyncLoggingErrorFileLogDroppedMessages
How many messages have been dropped from error file log due to the async log queue being full
AsyncLoggingErrorFileLogTotalMessages
How many messages (accepted or dropped) have been sent to the async queue for the error file log
AsyncLoggingFileLogDroppedMessages
How many messages have been dropped from the file log due to the async log queue being full
AsyncLoggingFileLogTotalMessages
How many messages (accepted or dropped) have been sent to the async queue for the file log
AsyncLoggingSyslogDroppedMessages
How many messages have been dropped from the syslog due to the async log queue being full
AsyncLoggingSyslogTotalMessages
How many messages (accepted or dropped) have been sent to the async queue for the syslog
AsyncLoggingTextLogDroppedMessages
How many messages have been dropped from text_log due to the async log queue being full
AsyncLoggingTextLogTotalMessages
How many messages (accepted or dropped) have been sent to the async queue for the text_log
AsynchronousReadWaitMicroseconds
Time spent in waiting for asynchronous reads in asynchronous local read.
AsynchronousReaderIgnoredBytes
Number of bytes ignored during asynchronous reading
AsynchronousRemoteReadWaitMicroseconds
Time spent in waiting for asynchronous remote reads.
AzureCommitBlockList
Number of Azure blob storage API CommitBlockList calls
AzureCopyObject
Number of Azure blob storage API CopyObject calls
AzureCreateContainer
Number of Azure blob storage API CreateContainer calls.
AzureDeleteObjects
Number of Azure blob storage API DeleteObject(s) calls.
AzureGetObject
Number of Azure API GetObject calls.
AzureGetProperties
Number of Azure blob storage API GetProperties calls.
AzureGetRequestThrottlerBlocked
Number of Azure GET requests blocked by throttler.
AzureGetRequestThrottlerCount
Number of Azure GET requests passed through throttler: blocked and not blocked.
AzureGetRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform Azure GET request throttling.
AzureListObjects
Number of Azure blob storage API ListObjects calls.
AzurePutRequestThrottlerBlocked
Number of Azure PUT requests blocked by throttler.
AzurePutRequestThrottlerCount
Number of Azure PUT requests passed through throttler: blocked and not blocked.
AzurePutRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform Azure PUT request throttling.
AzureReadMicroseconds
Total time spent waiting for Azure read requests.
AzureReadRequestsCount
Number of Azure read requests.
AzureReadRequestsErrors
Number of Azure read request errors.
AzureReadRequestsRedirects
Number of Azure read request redirects.
AzureReadRequestsThrottling
Number of Azure read requests throttled.
AzureStageBlock
Number of Azure blob storage API StageBlock calls
AzureUpload
Number of Azure blob storage API Upload calls
AzureWriteMicroseconds
Total time spent waiting for Azure read requests.
AzureWriteRequestsCount
Number of Azure write requests.
AzureWriteRequestsErrors
Number of Azure write request errors.
AzureWriteRequestsRedirects
Number of Azure write request redirects.
AzureWriteRequestsThrottling
Number of Azure write requests throttled.
BackgroundLoadingMarksTasks
Number of background tasks for loading marks
BackupEntriesCollectorForTablesDataMicroseconds
Time spent making backup entries for tables data
BackupEntriesCollectorMicroseconds
Time spent making backup entries
BackupEntriesCollectorRunPostTasksMicroseconds
Time spent running post tasks after making backup entries
BackupLockFileReads
How many times the '.lock' file was read while making backup
BackupPreparingFileInfosMicroseconds
Time spent preparing file infos for backup entries
BackupReadLocalBytesToCalculateChecksums
Total size of files read locally to calculate checksums for backup entries
BackupReadLocalFilesToCalculateChecksums
Number of files read locally to calculate checksums for backup entries
BackupReadMetadataMicroseconds
Time spent reading backup metadata from .backup file
BackupReadRemoteBytesToCalculateChecksums
Total size of files read from remote disks to calculate checksums for backup entries
BackupReadRemoteFilesToCalculateChecksums
Number of files read from remote disks to calculate checksums for backup entries
BackupThrottlerBytes
Bytes passed through 'max_backup_bandwidth_for_server' throttler.
BackupThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_backup_bandwidth_for_server' throttling.
BackupWriteMetadataMicroseconds
Time spent writing backup metadata to .backup file
BackupsOpenedForRead
Number of backups opened for reading
BackupsOpenedForUnlock
Number of backups opened for unlocking
BackupsOpenedForWrite
Number of backups opened for writing
BlobCopierThreadLockBlobsErrors
Number of blobs lock errors occurred during BlobCopier execution
BlobCopierThreadLockedBlobs
Number of blobs returned from metadata storage
BlobCopierThreadRecordBlobsErrors
Number of blobs recording errors occurred during BlobCopier execution
BlobCopierThreadRecordedBlobs
Number of blobs which replication by BlobCopier was recorded in metadata storage
BlobCopierThreadReplicateBlobsErrors
Number of blobs replication errors occurred during BlobCopier execution
BlobCopierThreadReplicatedBlobs
Number of blobs replicated by BlobCopier
BlobCopierThreadRuns
Number of BlobCopier thread executes
BlobKillerThreadLockBlobsErrors
Number of blobs lock errors occurred during BlobKiller execution
BlobKillerThreadLockedBlobs
Number of blobs returned from metadata storage
BlobKillerThreadRecordBlobsErrors
Number of blobs recording errors occurred during BlobKiller execution
BlobKillerThreadRecordedBlobs
Number of blobs which removal by BlobKiller was recorded in metadata storage
BlobKillerThreadRemoveBlobsErrors
Number of blobs removal errors occurred during BlobKiller execution
BlobKillerThreadRemoveTasks
Number of remove tasks created by BlobKiller
BlobKillerThreadRemovedBlobs
Number of blobs removed by BlobKiller
BlobKillerThreadRuns
Number of BlobKiller thread executes
BuildPatchesJoinMicroseconds
Total time spent building indexes and hash tables for applying patch parts with Join mode
BuildPatchesMergeMicroseconds
Total time spent building indexes for applying patch parts with Merge mode
CacheWarmerBytesDownloaded
Amount of data fetched into filesystem cache by dedicated background threads.
CacheWarmerDataPartsDownloaded
Number of data parts that were fully fetched by CacheWarmer.
CachedReadBufferCacheWriteBytes
Bytes written from source (remote fs, etc) to filesystem cache
CachedReadBufferCacheWriteMicroseconds
Time spent writing data into filesystem cache
CachedReadBufferCreateBufferMicroseconds
Prepare buffer time
CachedReadBufferPredownloadedBytes
Bytes read from filesystem cache source. Cache segments are read from left to right as a whole, it might be that we need to predownload some part of the segment irrelevant for the current task just to get to the needed data
CachedReadBufferPredownloadedFromSourceBytes
Bytes read from filesystem cache source for predownload (from remote fs, etc)
CachedReadBufferPredownloadedFromSourceMicroseconds
Time reading from filesystem cache source for predownload (from remote filesystem, etc)
CachedReadBufferReadFromCacheBytes
Bytes read from filesystem cache
CachedReadBufferReadFromCacheHits
Number of times the read from filesystem cache hit the cache.
CachedReadBufferReadFromCacheMicroseconds
Time reading from filesystem cache
CachedReadBufferReadFromCacheMisses
Number of times the read from filesystem cache miss the cache.
CachedReadBufferReadFromSourceBytes
Bytes read from filesystem cache source (from remote fs, etc)
CachedReadBufferReadFromSourceMicroseconds
Time reading from filesystem cache source (from remote filesystem, etc)
CachedReadBufferWaitReadBufferMicroseconds
Time spend waiting for internal read buffer (includes cache waiting)
CachedWriteBufferCacheWriteBytes
Bytes written from source (remote fs, etc) to filesystem cache
CachedWriteBufferCacheWriteMicroseconds
Time spent writing data into filesystem cache
CannotRemoveEphemeralNode
Number of times an error happened while trying to remove ephemeral node. This is not an issue, because our implementation of ZooKeeper library guarantee that the session will expire and the node will be removed.
CannotWriteToWriteBufferDiscard
Number of stack traces dropped by query profiler or signal handler because pipe is full or cannot write to pipe.
CoalescingSortedMilliseconds
Total time spent while coalescing sorted columns
CollapsingSortedMilliseconds
Total time spent while collapsing sorted columns
CommonBackgroundExecutorTaskCancelMicroseconds
Time spent in cancel() for Common executor tasks.
CommonBackgroundExecutorTaskExecuteStepMicroseconds
Time spent in executeStep() for Common executor tasks.
CommonBackgroundExecutorTaskResetMicroseconds
Time spent resetting task for Common executor.
CommonBackgroundExecutorWaitMicroseconds
Time spent waiting for completion in Common executor.
CompileExpressionsBytes
Number of bytes used for expressions compilation.
CompileExpressionsMicroseconds
Total time spent for compilation of expressions to LLVM code.
CompileFunction
Number of times a compilation of generated LLVM code (to create fused function for complex expressions) was initiated.
CompiledFunctionExecute
Number of times a compiled function was executed.
CompressedReadBufferBlocks
Number of compressed blocks (the blocks of data that are compressed independent of each other) read from compressed sources (files, network).
CompressedReadBufferBytes
Number of uncompressed bytes (the number of bytes after decompression) read from compressed sources (files, network).
CompressedReadBufferChecksumDoesntMatch
Number of times the compressed block checksum did not match.
CompressedReadBufferChecksumDoesntMatchMicroseconds
Total time spent detecting bit-flips due to compressed block checksum mismatches.
CompressedReadBufferChecksumDoesntMatchSingleBitMismatch
Number of times a compressed block checksum mismatch was caused by a single-bit difference.
ConcurrencyControlDownscales
Total number of CPU downscaling events
ConcurrencyControlPreemptedMicroseconds
Total time a query was waiting due to preemption of CPU slots.
ConcurrencyControlPreemptions
Total number of CPU preemptions
ConcurrencyControlQueriesDelayed
Total number of CPU slot allocations (queries) that were required to wait for slots to upscale
ConcurrencyControlSlotsAcquired
Total number of CPU slots acquired
ConcurrencyControlSlotsAcquiredNonCompeting
Total number of noncompeting CPU slot acquired
ConcurrencyControlSlotsDelayed
Number of CPU slot not granted initially and required to wait for a free CPU slot
ConcurrencyControlSlotsGranted
Number of CPU slot granted according to guarantee of 1 thread per query and for queries with setting 'use_concurrency_control' = 0
ConcurrencyControlUpscales
Total number of CPU upscaling events
ConcurrencyControlWaitMicroseconds
Total time a query was waiting on resource requests for CPU slots.
ConcurrentQuerySlotsAcquired
Total number of query slots acquired
ConcurrentQueryWaitMicroseconds
Total time a query was waiting for a query slots
ConnectionPoolIsFullMicroseconds
Total time spent waiting for a slot in connection pool.
ContextLock
Number of times the lock of Context was acquired or tried to acquire. This is global lock.
ContextLockWaitMicroseconds
Context lock wait time in microseconds
CoordinatedMergesMergeAssignmentRequest
Total number of merge assignment requests
CoordinatedMergesMergeAssignmentRequestMicroseconds
Total time spend in merge assignment client
CoordinatedMergesMergeAssignmentResponse
Total number of merge assignment requests
CoordinatedMergesMergeAssignmentResponseMicroseconds
Total time spend in merge assignment handler
CoordinatedMergesMergeCoordinatorFetchMetadataMicroseconds
Total time spend on fetching fresh metadata inside merge coordinator
CoordinatedMergesMergeCoordinatorFilterMicroseconds
Total time spend on filtering prepared merges inside merge coordinator
CoordinatedMergesMergeCoordinatorLockStateExclusivelyCount
Total number of exclusive captures of coordinator state lock
CoordinatedMergesMergeCoordinatorLockStateExclusivelyMicroseconds
Total time spend on locking coordinator state mutex exclusively
CoordinatedMergesMergeCoordinatorLockStateForShareCount
Total number of for share captures of coordinator state lock
CoordinatedMergesMergeCoordinatorLockStateForShareMicroseconds
Total time spend on locking coordinator state mutex for share
CoordinatedMergesMergeCoordinatorSelectMergesMicroseconds
Total time spend on finding merge using merge selectors inside merge coordinator
CoordinatedMergesMergeCoordinatorUpdateCount
Total number of merge coordinator updates
CoordinatedMergesMergeCoordinatorUpdateMicroseconds
Total time spend on updating merge coordinator state
CoordinatedMergesMergeWorkerUpdateCount
Total number merge worker updates
CoordinatedMergesMergeWorkerUpdateMicroseconds
Total time spend on updating local state of assigned merges on worker
CreatedLogEntryForMerge
Successfully created log entry to merge parts in ReplicatedMergeTree.
CreatedLogEntryForMutation
Successfully created log entry to mutate parts in ReplicatedMergeTree.
CreatedReadBufferDirectIO
Number of times a read buffer with O_DIRECT was created for reading data (while choosing among other read methods).
CreatedReadBufferDirectIOFailed
Number of times a read buffer with O_DIRECT was attempted to be created for reading data (while choosing among other read methods), but the OS did not allow it (due to lack of filesystem support or other reasons) and we fallen back to the ordinary reading method.
CreatedReadBufferMMap
Number of times a read buffer using 'mmap' was created for reading data (while choosing among other read methods).
CreatedReadBufferMMapFailed
Number of times a read buffer with 'mmap' was attempted to be created for reading data (while choosing among other read methods), but the OS did not allow it (due to lack of filesystem support or other reasons) and we fallen back to the ordinary reading method.
CreatedReadBufferOrdinary
Number of times ordinary read buffer was created for reading data (while choosing among other read methods).
DNSError
Total count of errors in DNS resolution
DataAfterMergeDiffersFromReplica
Number of times data after merge is not byte-identical to the data on another replicas. There could be several reasons:
- Using newer version of compression library after server update.
- Using another compression method.
- Non-deterministic compression algorithm (highly unlikely).
- Non-deterministic merge algorithm due to logical error in code.
- Data corruption in memory due to bug in code.
- Data corruption in memory due to hardware issue.
- Manual modification of source data after server startup.
- Manual modification of checksums stored in ZooKeeper.
- Part format related settings like 'enable_mixed_granularity_parts' are different on different replicas. The server successfully detected this situation and will download merged part from the replica to force the byte-identical result.
DataAfterMutationDiffersFromReplica
Number of times data after mutation is not byte-identical to the data on other replicas. In addition to the reasons described in 'DataAfterMergeDiffersFromReplica', it is also possible due to non-deterministic mutation.
DefaultImplementationForNullsRows
Number of rows processed by default implementation for nulls in function execution
DefaultImplementationForNullsRowsWithNulls
Number of rows which contain null values processed by default implementation for nulls in function execution
DelayedInserts
Number of times the INSERT of a block to a MergeTree table was throttled due to high number of active data parts for partition.
DelayedInsertsMilliseconds
Total number of milliseconds spent while the INSERT of a block to a MergeTree table was throttled due to high number of active data parts for partition.
DelayedMutations
Number of times the mutation of a MergeTree table was throttled due to high number of unfinished mutations for table.
DelayedMutationsMilliseconds
Total number of milliseconds spent while the mutation of a MergeTree table was throttled due to high number of unfinished mutations for table.
DeltaLakePartitionPrunedFiles
Number of skipped files during DeltaLake partition pruning
DeltaLakeScannedFiles
Number of files scanned during DeltaLake scan callbacks
DeltaLakeSnapshotInitializations
Number of times a DeltaLake table snapshot was initialized (loaded from object storage)
DictCacheKeysExpired
Number of keys looked up in the dictionaries of 'cache' types and found in the cache but they were obsolete.
DictCacheKeysHit
Number of keys looked up in the dictionaries of 'cache' types and found in the cache.
DictCacheKeysNotFound
Number of keys looked up in the dictionaries of 'cache' types and not found.
DictCacheKeysRequested
Number of keys requested from the data source for the dictionaries of 'cache' types.
DictCacheKeysRequestedFound
Number of keys requested from the data source for dictionaries of 'cache' types and found in the data source.
DictCacheKeysRequestedMiss
Number of keys requested from the data source for dictionaries of 'cache' types but not found in the data source.
DictCacheLockReadHoldNs
Number of nanoseconds spent holding read lock for the dictionaries of 'cache' types.
DictCacheLockReadNs
Number of nanoseconds spend in waiting for read lock to lookup the data for the dictionaries of 'cache' types.
DictCacheLockWriteHoldNs
Number of nanoseconds spent holding write lock for the dictionaries of 'cache' types.
DictCacheLockWriteNs
Number of nanoseconds spend in waiting for write lock to update the data for the dictionaries of 'cache' types.
DictCacheRequestTimeNs
Number of nanoseconds spend in querying the external data sources for the dictionaries of 'cache' types.
DictCacheRequests
Number of bulk requests to the external data sources for the dictionaries of 'cache' types.
DirectorySync
Number of times the F_FULLFSYNC/fsync/fdatasync function was called for directories.
DirectorySyncElapsedMicroseconds
Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for directories.
DiskAzureCommitBlockList
Number of Disk Azure blob storage API CommitBlockList calls
DiskAzureCopyObject
Number of Disk Azure blob storage API CopyObject calls
DiskAzureCreateContainer
Number of Disk Azure blob storage API CreateContainer calls.
DiskAzureDeleteObjects
Number of Azure blob storage API DeleteObject(s) calls.
DiskAzureGetObject
Number of Disk Azure API GetObject calls.
DiskAzureGetProperties
Number of Disk Azure blob storage API GetProperties calls.
DiskAzureGetRequestThrottlerBlocked
Number of Azure disk GET requests blocked by throttler.
DiskAzureGetRequestThrottlerCount
Number of Azure disk GET requests passed through throttler: blocked and not blocked.
DiskAzureGetRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform Azure disk GET request throttling.
DiskAzureListObjects
Number of Disk Azure blob storage API ListObjects calls.
DiskAzurePutRequestThrottlerBlocked
Number of Azure disk PUT requests blocked by throttler.
DiskAzurePutRequestThrottlerCount
Number of Azure disk PUT requests passed through throttler: blocked and not blocked.
DiskAzurePutRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform Azure disk PUT request throttling.
DiskAzureReadMicroseconds
Total time spent waiting for Azure disk read requests.
DiskAzureReadRequestsCount
Number of Azure disk read requests.
DiskAzureReadRequestsErrors
Number of Azure disk read request errors.
DiskAzureReadRequestsRedirects
Number of Azure disk read request redirects.
DiskAzureReadRequestsThrottling
Number of Azure disk read requests throttled.
DiskAzureStageBlock
Number of Disk Azure blob storage API StageBlock calls
DiskAzureUpload
Number of Disk Azure blob storage API Upload calls
DiskAzureWriteMicroseconds
Total time spent waiting for Azure disk write requests.
DiskAzureWriteRequestsCount
Number of Azure disk write requests.
DiskAzureWriteRequestsErrors
Number of Azure disk write request errors.
DiskAzureWriteRequestsRedirects
Number of Azure disk write request redirects.
DiskAzureWriteRequestsThrottling
Number of Azure disk write requests throttled.
DiskConnectionsCreated
Number of created connections for disk
DiskConnectionsElapsedMicroseconds
Total time spend on creating connections for disk
DiskConnectionsErrors
Number of cases when creation of a connection for disk is failed
DiskConnectionsExpired
Number of expired connections for disk
DiskConnectionsPreserved
Number of preserved connections for disk
DiskConnectionsReset
Number of reset connections for disk
DiskConnectionsReused
Number of reused connections for disk
DiskPlainRewritableAzureDirectoryCreated
Number of directories created by the 'plain_rewritable' metadata storage for AzureObjectStorage.
DiskPlainRewritableAzureDirectoryRemoved
Number of directories removed by the 'plain_rewritable' metadata storage for AzureObjectStorage.
DiskPlainRewritableLegacyLayoutDiskCount
Number of the 'plain_rewritable' disks with legacy layout.
DiskPlainRewritableLocalDirectoryCreated
Number of directories created by the 'plain_rewritable' metadata storage for LocalObjectStorage.
DiskPlainRewritableLocalDirectoryRemoved
Number of directories removed by the 'plain_rewritable' metadata storage for LocalObjectStorage.
DiskPlainRewritableS3DirectoryCreated
Number of directories created by the 'plain_rewritable' metadata storage for S3ObjectStorage.
DiskPlainRewritableS3DirectoryRemoved
Number of directories removed by the 'plain_rewritable' metadata storage for S3ObjectStorage.
DiskReadElapsedMicroseconds
Total time spent waiting for read syscall. This include reads from page cache.
DiskS3AbortMultipartUpload
Number of DiskS3 API AbortMultipartUpload calls.
DiskS3CompleteMultipartUpload
Number of DiskS3 API CompleteMultipartUpload calls.
DiskS3CopyObject
Number of DiskS3 API CopyObject calls.
DiskS3CreateMultipartUpload
Number of DiskS3 API CreateMultipartUpload calls.
DiskS3DeleteObjects
Number of DiskS3 API DeleteObject(s) calls.
DiskS3GetObject
Number of DiskS3 API GetObject calls.
DiskS3GetObjectTagging
Number of DiskS3 API GetObjectTagging calls.
DiskS3GetRequestThrottlerBlocked
Number of DiskS3 GET and SELECT requests blocked by throttler.
DiskS3GetRequestThrottlerCount
Number of DiskS3 GET and SELECT requests passed through throttler: blocked and not blocked.
DiskS3GetRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform DiskS3 GET and SELECT request throttling.
DiskS3HeadObject
Number of DiskS3 API HeadObject calls.
DiskS3ListObjects
Number of DiskS3 API ListObjects calls.
DiskS3PutObject
Number of DiskS3 API PutObject calls.
DiskS3PutRequestThrottlerBlocked
Number of DiskS3 PUT, COPY, POST and LIST requests blocked by throttler.
DiskS3PutRequestThrottlerCount
Number of DiskS3 PUT, COPY, POST and LIST requests passed through throttler: blocked and not blocked.
DiskS3PutRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform DiskS3 PUT, COPY, POST and LIST request throttling.
DiskS3ReadMicroseconds
Time of GET and HEAD requests to DiskS3 storage.
DiskS3ReadRequestAttempts
Number of attempts for GET and HEAD requests to DiskS3 storage, including the initial try and any retries, but excluding retries performed internally by the S3 retry strategy
DiskS3ReadRequestRetryableErrors
Number of retryable errors for GET and HEAD requests to DiskS3 storage, excluding retries performed internally by the S3 retry strategy
DiskS3ReadRequestsCount
Number of GET and HEAD requests to DiskS3 storage.
DiskS3ReadRequestsErrors
Number of non-throttling errors in GET and HEAD requests to DiskS3 storage.
DiskS3ReadRequestsRedirects
Number of redirects in GET and HEAD requests to DiskS3 storage.
DiskS3ReadRequestsThrottling
Number of 429 and 503 errors in GET and HEAD requests to DiskS3 storage.
DiskS3UploadPart
Number of DiskS3 API UploadPart calls.
DiskS3UploadPartCopy
Number of DiskS3 API UploadPartCopy calls.
DiskS3WriteMicroseconds
Time of POST, DELETE, PUT and PATCH requests to DiskS3 storage.
DiskS3WriteRequestAttempts
Number of attempts for POST, DELETE, PUT and PATCH requests to DiskS3 storage, including the initial try and any retries, but excluding retries performed internally by the retry strategy
DiskS3WriteRequestRetryableErrors
Number of retryable errors for POST, DELETE, PUT and PATCH requests to DiskS3 storage, excluding retries performed internally by the retry strategy
DiskS3WriteRequestsCount
Number of POST, DELETE, PUT and PATCH requests to DiskS3 storage.
DiskS3WriteRequestsErrors
Number of non-throttling errors in POST, DELETE, PUT and PATCH requests to DiskS3 storage.
DiskS3WriteRequestsRedirects
Number of redirects in POST, DELETE, PUT and PATCH requests to DiskS3 storage.
DiskS3WriteRequestsThrottling
Number of 429 and 503 errors in POST, DELETE, PUT and PATCH requests to DiskS3 storage.
DiskWriteElapsedMicroseconds
Total time spent waiting for write syscall. This include writes to page cache.
DistrCacheConnectAttempts
Distributed Cache connection event. The number of connection attempts to distributed cache
DistrCacheConnectMicroseconds
Distributed Cache connection event. The time spent to connect to distributed cache
DistrCacheFallbackReadMicroseconds
Distributed Cache read buffer event. Time spend reading from fallback buffer instead of distributed cache
DistrCacheGetClientMicroseconds
Distributed Cache connection event. Time spent getting client for distributed cache
DistrCacheGetResponseMicroseconds
Distributed Cache client event. Time spend to wait for response from distributed cache
DistrCacheHashRingRebuilds
Distributed Cache registry event. Number of distributed cache hash ring rebuilds
DistrCacheLockRegistryMicroseconds
Distributed Cache registry event. Time spent to take DistributedCacheRegistry lock
DistrCacheMakeRequestErrors
Distributed Cache client event. Number of distributed cache errors when making a request
DistrCacheNextImplMicroseconds
Distributed Cache read buffer event. Time spend in ReadBufferFromDistributedCache::nextImpl
DistrCacheOpenedConnections
Distributed Cache connection event. The number of open connections to distributed cache
DistrCacheOpenedConnectionsBypassingPool
Distributed Cache connection event. The number of open connections to distributed cache bypassing pool
DistrCachePrecomputeRangesMicroseconds
Distributed Cache read buffer event. Time spent to precompute read ranges
DistrCacheRangeChange
Distributed Cache read buffer event. Number of times we changed read range because of seek/last_position change
DistrCacheRangeResetBackward
Distributed Cache read buffer event. Number of times we reset read range because of seek/last_position change
DistrCacheRangeResetForward
Distributed Cache read buffer event. Number of times we reset read range because of seek/last_position change
DistrCacheReadBytesFromFallbackBuffer
Distributed Cache read buffer event. Bytes read from fallback buffer
DistrCacheReadErrors
Distributed Cache read buffer event. Number of distributed cache errors during read
DistrCacheReadMicroseconds
Distributed Cache read buffer event. Time spent reading from distributed cache
DistrCacheReadThrottlerBytes
Bytes passed through 'max_distributed_cache_read_bandwidth_for_server' throttler.
DistrCacheReadThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_distributed_cache_read_bandwidth_for_server' throttling.
DistrCacheReceiveResponseErrors
Distributed Cache client event. Number of distributed cache errors when receiving response a request
DistrCacheReceivedCredentialsRefreshPackets
Distributed Cache client event. Total number of received RefreshCredentials packets received from distributed cache
DistrCacheReceivedDataPackets
Distributed Cache client event. Total number of received data packets received from distributed cache
DistrCacheReceivedDataPacketsBytes
Distributed Cache client event. The number of bytes in Data packets received from distributed cache
DistrCacheReceivedErrorPackets
Distributed Cache client event. Total number of received Error packets received from distributed cache
DistrCacheReceivedOkPackets
Distributed Cache client event. Total number of received Ok packets received from distributed cache
DistrCacheReceivedStopPackets
Distributed Cache client event. Total number of received Stop packets received from distributed cache
DistrCacheReconnectsAfterTimeout
Distributed Cache read buffer event. The number of reconnects after timeout
DistrCacheRegistryUpdateMicroseconds
Distributed Cache registry event. Time spent updating distributed cache registry
DistrCacheRegistryUpdates
Distributed Cache registry event. Number of distributed cache registry updates
DistrCacheReusedConnections
Distributed Cache connection event. The number of reused connections to distributed cache
DistrCacheSentDataPackets
Distributed Cache client event. Total number of data packets sent to distributed cache
DistrCacheSentDataPacketsBytes
Distributed Cache client event. The number of bytes in Data packets sent to distributed cache
DistrCacheServerAckRequestPackets
Distributed Cache server event. Number of AckRequest packets in DistributedCacheServer
DistrCacheServerCachedReadBufferCacheHits
Distributed Cache server event. The number of times distributed cache hit the cache while reading from filesystem cache
DistrCacheServerCachedReadBufferCacheMisses
Distributed Cache server event. The number of times distributed cache missed the cache while reading from filesystem cache
DistrCacheServerCachedReadBufferCachePredownloadBytes
Distributed Cache server event. The number of bytes read from object storage for predownload in distributed cache while reading from filesystem cache
DistrCacheServerCachedReadBufferCacheReadBytes
Distributed Cache server event. The number of bytes read from cache in distributed cache while reading from filesystem cache
DistrCacheServerCachedReadBufferCacheWrittenBytes
Distributed Cache server event. The number of bytes written to cache in distributed cache while reading from filesystem cache
DistrCacheServerCachedReadBufferObjectStorageReadBytes
Distributed Cache server event. The number of bytes read from object storage in distributed cache while reading from filesystem cache
DistrCacheServerContinueRequestPackets
Distributed Cache server event. Number of ContinueRequest packets in DistributedCacheServer
DistrCacheServerCredentialsRefresh
Distributed Cache server event. The number of expired credentials were refreshed
DistrCacheServerEndRequestPackets
Distributed Cache server event. Number of EndRequest packets in DistributedCacheServer
DistrCacheServerNewS3CachedClients
Distributed Cache server event. The number of new cached s3 clients
DistrCacheServerProcessRequestMicroseconds
Distributed Cache server event. Time spent processing request on DistributedCache server side
DistrCacheServerReceivedCredentialsRefreshPackets
Distributed Cache server event. Number of RefreshCredentials client packets in DistributedCacheServer
DistrCacheServerReusedS3CachedClients
Distributed Cache server event. The number of reused cached s3 clients
DistrCacheServerSkipped
Distributed Cache server event. The number of times distributed cache server was skipped because of previous failed connection attempts
DistrCacheServerStartRequestPackets
Distributed Cache server event. Number of StartRequest packets in DistributedCacheServer
DistrCacheServerSwitches
Distributed Cache read buffer event. Number of server switches between distributed cache servers in read/write-through cache
DistrCacheServerUpdates
Distributed Cache read buffer event. The number of server updates because server is not longer registered in keeper
DistrCacheStartRangeMicroseconds
Distributed Cache read buffer event. Time spent to start a new read range with distributed cache
DistrCacheSuccessfulConnectAttempts
Distributed Cache connection event. The number of successful connection attempts to distributed cache
DistrCacheSuccessfulRegistryUpdates
Distributed Cache registry event. The number of successful server registry updates
DistrCacheTemporaryFilesBytesWritten
Distributed Cache connection event. Number of bytes written to temporary files created in distributed cache
DistrCacheTemporaryFilesCreated
Distributed Cache connection event. Number of temporary files created in distributed cache
DistrCacheUnsuccessfulConnectAttempts
Distributed Cache connection event. The number of unsuccessful connection attempts to distributed cache
DistrCacheUnsuccessfulRegistryUpdates
Distributed Cache registry event. The number of unsuccessful server registry updates
DistrCacheUnusedDataPacketsBytes
Distributed Cache client event. The number of bytes in Data packets which were ignored
DistrCacheUnusedPackets
Distributed Cache client event. Number of skipped unused packets from distributed cache
DistrCacheUnusedPacketsBufferAllocations
Distributed Cache client event. The number of extra buffer allocations in case we could not reuse existing buffer
DistrCacheWriteErrors
Distributed Cache write buffer event. Number of distributed cache errors during write
DistrCacheWriteThrottlerBytes
Bytes passed through 'max_distributed_cache_write_bandwidth_for_server' throttler.
DistrCacheWriteThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_distributed_cache_write_bandwidth_for_server' throttling.
DistributedAsyncInsertionFailures
Number of failures for asynchronous insertion into a Distributed table (with 'distributed_foreground_insert' = 0)
DistributedConnectionConnectCount
Number of connects to other servers done during distributed query execution. Happens when new connection is established instead of using existing from pool.
DistributedConnectionFailAtAll
Total count when distributed connection fails after all retries finished.
DistributedConnectionFailTry
Total count when distributed connection fails with retry.
DistributedConnectionMissingTable
Number of times we rejected a replica from a distributed query, because it did not contain a table needed for the query.
DistributedConnectionReconnectCount
Number of reconnects to other servers done during distributed query execution. It can happen when a stale connection has been acquired from connection pool
DistributedConnectionSkipReadOnlyReplica
Number of replicas skipped during INSERT into Distributed table due to replicas being read-only
DistributedConnectionStaleReplica
Number of times we rejected a replica from a distributed query, because some table needed for a query had replication lag higher than the configured threshold.
DistributedConnectionTries
Total count of distributed connection attempts.
DistributedConnectionUsable
Total count of successful distributed connections to a usable server (with required table, but maybe stale).
DistributedDelayedInserts
Number of times the INSERT of a block to a Distributed table was throttled due to high number of pending bytes.
DistributedDelayedInsertsMilliseconds
Total number of milliseconds spent while the INSERT of a block to a Distributed table was throttled due to high number of pending bytes.
DistributedIndexAnalysisMicroseconds
Total time spent during distributed index analysis
DistributedIndexAnalysisMissingParts
Number of missing parts during distributed index analysis that will be resolved locally
DistributedIndexAnalysisParts
Number of parts send for distributed index analysis
DistributedIndexAnalysisReplicaFallback
Number of times distributed index analysis failed on one of replicas with fallback to local replica
DistributedIndexAnalysisReplicaUnavailable
Number of times distributed index analysis failed on one of replicas without fallback (failed during connect)
DistributedIndexAnalysisScheduledReplicas
Number of replicas (local replica will be accounted once) to which distributed index analysis has been scheduled
DistributedRejectedInserts
Number of times the INSERT of a block to a Distributed table was rejected with 'Too many bytes' exception due to high number of pending bytes.
DistributedSyncInsertionTimeoutExceeded
A timeout has exceeded while waiting for shards during synchronous insertion into a Distributed table (with 'distributed_foreground_insert' = 1)
DuplicatedAsyncInserts
Number of async inserts in the INSERTed block to a ReplicatedMergeTree table was deduplicated.
DuplicatedInsertedBlocks
Number of the synchronous inserts to a *MergeTree table was deduplicated.
DuplicationElapsedMicroseconds
Total time spent checking for duplication of INSERTed blocks to *MergeTree tables.
EngineFileLikeReadFiles
Number of files read in table engines working with files (like File/S3/URL/HDFS).
ExecuteShellCommand
Number of shell command executions.
ExternalAggregationCompressedBytes
Number of bytes written to disk for aggregation in external memory.
ExternalAggregationMerge
Number of times temporary files were merged for aggregation in external memory.
ExternalAggregationUncompressedBytes
Amount of data (uncompressed, before compression) written to disk for aggregation in external memory.
ExternalAggregationWritePart
Number of times a temporary file was written to disk for aggregation in external memory.
ExternalDataSourceLocalCacheReadBytes
Bytes read from local cache buffer in RemoteReadBufferCache
ExternalJoinCompressedBytes
Number of compressed bytes written for JOIN in external memory.
ExternalJoinMerge
Number of times temporary files were merged for JOIN in external memory.
ExternalJoinUncompressedBytes
Amount of data (uncompressed, before compression) written for JOIN in external memory.
ExternalJoinWritePart
Number of times a temporary file was written to disk for JOIN in external memory.
ExternalProcessingCompressedBytesTotal
Number of compressed bytes written by external processing (sorting/aggragating/joining)
ExternalProcessingFilesTotal
Number of files used by external processing (sorting/aggragating/joining)
ExternalProcessingUncompressedBytesTotal
Amount of data (uncompressed, before compression) written by external processing (sorting/aggragating/joining)
ExternalSortCompressedBytes
Number of compressed bytes written for sorting in external memory.
ExternalSortMerge
Number of times temporary files were merged for sorting in external memory.
ExternalSortUncompressedBytes
Amount of data (uncompressed, before compression) written for sorting in external memory.
ExternalSortWritePart
Number of times a temporary file was written to disk for sorting in external memory.
FailedAsyncInsertQuery
Number of failed ASYNC INSERT queries.
FailedInitialQuery
Number of failed initial queries.
FailedInitialSelectQuery
Same as FailedInitialQuery, but only for SELECT queries.
FailedInsertQuery
Same as FailedQuery, but only for INSERT queries.
FailedInternalInsertQuery
Same as FailedInternalQuery, but only for INSERT queries.
FailedInternalQuery
Number of failed internal queries.
FailedInternalSelectQuery
Same as FailedInternalQuery, but only for SELECT queries.
FailedQuery
Number of total failed queries, both internal and user queries.
FailedSelectQuery
Same as FailedQuery, but only for SELECT queries.
FetchBackgroundExecutorTaskCancelMicroseconds
Time spent in cancel() for Fetch executor tasks.
FetchBackgroundExecutorTaskExecuteStepMicroseconds
Time spent in executeStep() for Fetch executor tasks.
FetchBackgroundExecutorTaskResetMicroseconds
Time spent resetting task for Fetch executor.
FetchBackgroundExecutorWaitMicroseconds
Time spent waiting for completion in Fetch executor.
FileOpen
Number of files opened.
FileSegmentCompleteMicroseconds
Duration of FileSegment::complete() in filesystem cache
FileSegmentFailToIncreasePriority
Number of times the priority was not increased due to a high contention on the cache lock
FileSegmentHolderCompleteMicroseconds
File segments holder complete() time
FileSegmentIncreasePriorityMicroseconds
File segment increase priority time
FileSegmentLockMicroseconds
Lock file segment time
FileSegmentRemoveMicroseconds
File segment remove() time
FileSegmentWaitMicroseconds
Wait on DOWNLOADING state
FileSegmentWriteMicroseconds
File segment write() time
FileSync
Number of times the F_FULLFSYNC/fsync/fdatasync function was called for files.
FileSyncElapsedMicroseconds
Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for files.
FilesystemCacheBackgroundDownloadQueuePush
Number of file segments sent for background download in filesystem cache
FilesystemCacheBackgroundEvictedBytes
Number of bytes evicted by background thread
FilesystemCacheBackgroundEvictedFileSegments
Number of file segments evicted by background thread
FilesystemCacheCheckCorrectness
Number of times FileCache::assertCacheCorrectness was called
FilesystemCacheCheckCorrectnessMicroseconds
How much time does FileCache::assertCacheCorrectness takes
FilesystemCacheCreatedKeyDirectories
Number of created key directories
FilesystemCacheEvictMicroseconds
Filesystem cache eviction time
FilesystemCacheEvictedBytes
Number of bytes evicted from filesystem cache
FilesystemCacheEvictedFileSegments
Number of file segments evicted from filesystem cache
FilesystemCacheEvictedFileSegmentsDuringPriorityIncrease
Number of file segments evicted from filesystem cache when increasing priority of file segments (Applies to SLRU cache policy)
FilesystemCacheEvictionReusedIterator
Number of filesystem cache iterator reusing
FilesystemCacheEvictionSkippedEvictingFileSegments
Number of file segments skipped for eviction because of being in evicting state
FilesystemCacheEvictionSkippedFileSegments
Number of file segments skipped for eviction because of being in unreleasable state
FilesystemCacheEvictionSkippedMovingFileSegments
Number of file segments skipped for eviction because of being in moving state
FilesystemCacheEvictionTries
Number of filesystem cache eviction attempts
FilesystemCacheFailToReserveSpaceBecauseOfCacheResize
Number of times space reservation was skipped due to the cache is being resized
FilesystemCacheFailToReserveSpaceBecauseOfLockContention
Number of times space reservation was skipped due to a high contention on the cache lock
FilesystemCacheFailedEvictionCandidates
Number of file segments which unexpectedly failed to be evicted during dynamic filesystem cache eviction
FilesystemCacheFreeSpaceKeepingThreadRun
Number of times background thread executed free space keeping job
FilesystemCacheFreeSpaceKeepingThreadWorkMilliseconds
Time for which background thread executed free space keeping job
FilesystemCacheGetMicroseconds
Filesystem cache get() time
FilesystemCacheGetOrSetMicroseconds
Filesystem cache getOrSet() time
FilesystemCacheHoldFileSegments
Filesystem cache file segments count, which were hold
FilesystemCacheLoadMetadataMicroseconds
Time spent loading filesystem cache metadata
FilesystemCacheLockKeyMicroseconds
Lock cache key time
FilesystemCacheLockMetadataMicroseconds
Lock filesystem cache metadata time
FilesystemCachePriorityReadLockMicroseconds
Lock filesystem cache time for read in priority queue
FilesystemCachePriorityWriteLockMicroseconds
Lock filesystem cache time for write to priority queue
FilesystemCacheReserveAttempts
Filesystem cache space reservation attempt
FilesystemCacheReserveMicroseconds
Filesystem cache space reservation time
FilesystemCacheStateLockMicroseconds
Lock filesystem cache time for state lock
FilesystemCacheUnusedHoldFileSegments
Filesystem cache file segments count, which were hold, but not used (because of seek or LIMIT n, etc)
FilterPartsByVirtualColumnsMicroseconds
Total time spent in filterPartsByVirtualColumns function.
FilterTransformPassedBytes
Number of bytes that passed the filter in the query
FilterTransformPassedRows
Number of rows that passed the filter in the query
FilteringMarksWithPrimaryKeyMicroseconds
Time spent filtering parts by PK.
FilteringMarksWithSecondaryKeysMicroseconds
Time spent filtering parts by skip indexes.
FunctionExecute
Number of SQL ordinary function calls (SQL functions are called on per-block basis, so this number represents the number of blocks).
GatheredColumns
Number of columns gathered during the vertical stage of merges.
GatheringColumnMilliseconds
Total time spent while gathering columns for vertical merge
GlobalThreadPoolExpansions
Counts the total number of times new threads have been added to the global thread pool. This metric indicates the frequency of expansions in the global thread pool to accommodate increased processing demands.
GlobalThreadPoolJobWaitTimeMicroseconds
Measures the elapsed time from when a job is scheduled in the thread pool to when it is picked up for execution by a worker thread. This metric helps identify delays in job processing, indicating the responsiveness of the thread pool to new tasks.
GlobalThreadPoolJobs
Counts the number of jobs that have been pushed to the global thread pool.
GlobalThreadPoolLockWaitMicroseconds
Total time threads have spent waiting for locks in the global thread pool.
GlobalThreadPoolShrinks
Counts the total number of times the global thread pool has shrunk by removing threads. This occurs when the number of idle threads exceeds max_thread_pool_free_size, indicating adjustments in the global thread pool size in response to decreased thread utilization.
GlobalThreadPoolThreadCreationMicroseconds
Total time spent waiting for new threads to start.
HTTPConnectionsCreated
Number of created client HTTP connections
HTTPConnectionsElapsedMicroseconds
Total time spend on creating client HTTP connections
HTTPConnectionsErrors
Number of cases when creation of a client HTTP connection failed
HTTPConnectionsExpired
Number of expired client HTTP connections
HTTPConnectionsPreserved
Number of preserved client HTTP connections
HTTPConnectionsReset
Number of reset client HTTP connections
HTTPConnectionsReused
Number of reused client HTTP connections
HTTPServerConnectionsClosed
Number of closed server HTTP connections. Keep alive has not been negotiated
HTTPServerConnectionsCreated
Number of created server HTTP connections
HTTPServerConnectionsExpired
Number of expired server HTTP connections.
HTTPServerConnectionsPreserved
Number of preserved server HTTP connections. Connection kept alive successfully
HTTPServerConnectionsReset
Number of reset server HTTP connections. Server closes connection
HTTPServerConnectionsReused
Number of reused server HTTP connections
HardPageFaults
The number of hard page faults in query execution threads. High values indicate either that you forgot to turn off swap on your server, or eviction of memory pages of the ClickHouse binary during very high memory pressure, or successful usage of the 'mmap' read method for the tables data.
HashJoinPreallocatedElementsInHashTables
How many elements were preallocated in hash tables for hash join.
HedgedRequestsChangeReplica
Total count when timeout for changing replica expired in hedged requests.
IOBufferAllocBytes
Number of bytes allocated for IO buffers (for ReadBuffer/WriteBuffer).
IOBufferAllocs
Number of allocations of IO buffers (for ReadBuffer/WriteBuffer).
IOUringCQEsCompleted
Total number of successfully completed io_uring CQEs
IOUringCQEsFailed
Total number of completed io_uring CQEs with failures
IOUringSQEsResubmitsAsync
Total number of asynchronous io_uring SQE resubmits performed
IOUringSQEsResubmitsSync
Total number of synchronous io_uring SQE resubmits performed
IOUringSQEsSubmitted
Total number of io_uring SQEs submitted
IcebergIteratorInitializationMicroseconds
Total time spent on synchronous initialization of iceberg data iterators.
IcebergMetadataFilesCacheHits
Number of times iceberg metadata files have been found in the cache.
IcebergMetadataFilesCacheMisses
Number of times iceberg metadata files have not been found in the iceberg metadata cache and had to be read from (remote) disk.
IcebergMetadataFilesCacheStaleMisses
Number of times iceberg metadata files have been found in the cache, but were considered stale and had to be read from (remote) disk.
IcebergMetadataFilesCacheWeightLost
Approximate number of bytes evicted from the iceberg metadata cache.
IcebergMetadataReadWaitTimeMicroseconds
Total time data readers spend waiting for iceberg metadata files to be read and parsed, summed across all reader threads.
IcebergMetadataReturnedObjectInfos
Total number of returned object infos from iceberg iterator.
IcebergMetadataUpdateMicroseconds
Total time spent on synchronous initialization of iceberg data iterators.
IcebergMinMaxIndexPrunedFiles
Number of skipped files by using MinMax index in Iceberg
IcebergMinMaxNonPrunedDeleteFiles
Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.
IcebergMinMaxPrunedDeleteFiles
Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.
IcebergPartitionPrunedFiles
Number of skipped files during Iceberg partition pruning
IcebergTrivialCountOptimizationApplied
Trivial count optimization applied while reading from Iceberg
IcebergVersionHintUsed
Number of times version-hint.text has been used.
IgnoredColdParts
See setting ignore_cold_parts_seconds. Number of times read queries ignored very new parts that weren't pulled into cache by CacheWarmer yet.
IndexAnalysisRounds
Number of times index analysis was performed within the query.
IndexBinarySearchAlgorithm
Number of times the binary search algorithm is used over the index marks
IndexGenericExclusionSearchAlgorithm
Number of times the generic exclusion search algorithm is used over the index marks
InitialQuery
Same as Query, but only counts initial queries (see is_initial_query).
InitialSelectQuery
Same as InitialQuery, but only for SELECT queries.
InsertQueriesWithSubqueries
Count INSERT queries with all subqueries
InsertQuery
Same as Query, but only for INSERT queries.
InsertQueryTimeMicroseconds
Total time of INSERT queries.
InsertedBytes
Number of bytes (uncompressed; for columns as they stored in memory) INSERTed to all tables.
InsertedCompactParts
Number of parts inserted in Compact format.
InsertedRows
Number of rows INSERTed to all tables.
InsertedWideParts
Number of parts inserted in Wide format.
InterfaceHTTPReceiveBytes
Number of bytes received through HTTP interfaces
InterfaceHTTPSendBytes
Number of bytes sent through HTTP interfaces
InterfaceInterserverReceiveBytes
Number of bytes received through interserver interfaces
InterfaceInterserverSendBytes
Number of bytes sent through interserver interfaces
InterfaceMySQLReceiveBytes
Number of bytes received through MySQL interfaces
InterfaceMySQLSendBytes
Number of bytes sent through MySQL interfaces
InterfaceNativeReceiveBytes
Number of bytes received through native interfaces
InterfaceNativeSendBytes
Number of bytes sent through native interfaces
InterfacePostgreSQLReceiveBytes
Number of bytes received through PostgreSQL interfaces
InterfacePostgreSQLSendBytes
Number of bytes sent through PostgreSQL interfaces
InterfacePrometheusReceiveBytes
Number of bytes received through Prometheus interfaces
InterfacePrometheusSendBytes
Number of bytes sent through Prometheus interfaces
JemallocFailedAllocationSampleTracking
Total number of times tracking of jemalloc allocation sample failed
JemallocFailedDeallocationSampleTracking
Total number of times tracking of jemalloc deallocation sample failed
JoinBuildTableRowCount
Total number of rows in the build table for a JOIN operation.
JoinOptimizeMicroseconds
Total time spent executing JOIN plan optimizations.
JoinProbeTableRowCount
Total number of rows in the probe table for a JOIN operation.
JoinReorderMicroseconds
Total time spent executing JOIN reordering algorithm.
JoinResultRowCount
Total number of rows in the result of a JOIN operation.
KafkaBackgroundReads
Number of background reads populating materialized views from Kafka since server start
KafkaCommitFailures
Number of failed commits of consumed offsets to Kafka (usually is a sign of some data duplication)
KafkaCommits
Number of successful commits of consumed offsets to Kafka (normally should be the same as KafkaBackgroundReads)
KafkaConsumerErrors
Number of errors reported by librdkafka during polls
KafkaDirectReads
Number of direct selects from Kafka tables since server start
KafkaMVNotReady
Number of failed attempts to stream data to a materialized view that is not ready
KafkaMessagesFailed
Number of Kafka messages ClickHouse failed to parse
KafkaMessagesPolled
Number of Kafka messages polled from librdkafka to ClickHouse
KafkaMessagesProduced
Number of messages produced to Kafka
KafkaMessagesRead
Number of Kafka messages already processed by ClickHouse
KafkaProducerErrors
Number of errors during producing the messages to Kafka
KafkaProducerFlushes
Number of explicit flushes to Kafka producer
KafkaRebalanceAssignments
Number of partition assignments (the final stage of consumer group rebalance)
KafkaRebalanceErrors
Number of failed consumer group rebalances
KafkaRebalanceRevocations
Number of partition revocations (the first stage of consumer group rebalance)
KafkaRowsRead
Number of rows parsed from Kafka messages
KafkaRowsRejected
Number of parsed rows which were later rejected (due to rebalances / errors or similar reasons). Those rows will be consumed again after the rebalance.
KafkaRowsWritten
Number of rows inserted into Kafka tables
KafkaWrites
Number of writes (inserts) to Kafka tables
KeeperAddWatchRequest
Number of add watches requests
KeeperBatchMaxCount
Number of times the size of batch was limited by the amount
KeeperBatchMaxTotalSize
Number of times the size of batch was limited by the total bytes size
KeeperChangelogFileSyncMicroseconds
Time spent in fsync for Keeper changelog (uncompressed logs only)
KeeperChangelogLockHoldMicroseconds
Time spent holding Keeper changelog lock
KeeperChangelogLockWaitMicroseconds
Time spent waiting for acquiring Keeper changelog lock
KeeperChangelogWrittenBytes
Number of bytes written to the changelog in Keeper
KeeperCheckRequest
Number of check requests
KeeperCheckWatchRequest
Number of remove watches requests
KeeperCommitWaitElapsedMicroseconds
Time spent waiting for certain log to be committed
KeeperCommits
Number of successful commits
KeeperCommitsFailed
Number of failed commits
KeeperCreateRequest
Number of create requests
KeeperExistsRequest
Number of exists requests
KeeperFinishedSessionsCacheFull
Number of times a finished session could not be tracked because the cache reached its size limit
KeeperGetRequest
Number of get requests
KeeperLatency
Keeper latency
KeeperListRequest
Number of list requests
KeeperLogsEntryReadFromCommitCache
Number of log entries in Keeper being read from commit logs cache
KeeperLogsEntryReadFromFile
Number of log entries in Keeper being read directly from the changelog file
KeeperLogsEntryReadFromLatestCache
Number of log entries in Keeper being read from latest logs cache
KeeperLogsPrefetchedEntries
Number of log entries in Keeper being prefetched from the changelog file
KeeperMultiReadRequest
Number of multi read requests
KeeperMultiRequest
Number of multi requests
KeeperPacketsReceived
Packets received by keeper server
KeeperPacketsSent
Packets sent by keeper server
KeeperPreprocessElapsedMicroseconds
Keeper preprocessing latency for a single request
KeeperProcessAndResponsesLockHoldMicroseconds
Time spent holding Keeper process and responses lock
KeeperProcessAndResponsesLockWaitMicroseconds
Time spent waiting for acquiring Keeper process and responses lock
KeeperProcessElapsedMicroseconds
Keeper commit latency for a single request
KeeperReadRequestQueueLockHoldMicroseconds
Time spent holding Keeper read request queue lock
KeeperReadRequestQueueLockWaitMicroseconds
Time spent waiting for acquiring Keeper read request queue lock
KeeperReadSnapshot
Number of snapshot read(serialization)
KeeperReconfigRequest
Number of reconfig requests
KeeperRemoveRequest
Number of remove requests
KeeperRemoveWatchRequest
Number of remove watches requests
KeeperRequestRejectedDueToSoftMemoryLimitCount
Number requests that have been rejected due to soft memory limit exceeded
KeeperRequestTotal
Total requests number on keeper server
KeeperRequestTotalWithSubrequests
Total requests number on keeper server, counting each subrequest within a multi request
KeeperSaveSnapshot
Number of snapshot save
KeeperServerWriteLockHoldMicroseconds
Time spent holding Keeper server write lock
KeeperServerWriteLockWaitMicroseconds
Time spent waiting for acquiring Keeper server write lock
KeeperSessionCallbackLockHoldMicroseconds
Time spent holding Keeper session callback lock
KeeperSessionCallbackLockWaitMicroseconds
Time spent waiting for acquiring Keeper session callback lock
KeeperSetRequest
Number of set requests
KeeperSetWatchesRequest
Number of set watches requests
KeeperSnapshotApplys
Number of snapshot applying
KeeperSnapshotApplysFailed
Number of failed snapshot applying
KeeperSnapshotCreations
Number of snapshots creations
KeeperSnapshotCreationsFailed
Number of failed snapshot creations
KeeperSnapshotFileSyncMicroseconds
Time spent in fsync for Keeper snapshot files
KeeperSnapshotWrittenBytes
Number of bytes written to snapshot files in Keeper
KeeperStaleRequestsSkipped
Number of Keeper requests skipped because the session was already finished
KeeperStorageLockHoldMicroseconds
Time spent holding Keeper storage lock
KeeperStorageLockWaitMicroseconds
Time spent waiting for acquiring Keeper storage lock
KeeperStorageSharedLockHoldMicroseconds
Time spent holding Keeper storage shared lock
KeeperStorageSharedLockWaitMicroseconds
Time spent waiting for acquiring Keeper storage shared lock
KeeperTotalElapsedMicroseconds
Keeper total latency for a single request
LoadedDataParts
Number of data parts loaded by MergeTree tables during initialization.
LoadedDataPartsMicroseconds
Microseconds spent by MergeTree tables for loading data parts during initialization.
LoadedMarksCount
Number of marks loaded (total across columns).
LoadedMarksFiles
Number of mark files loaded.
LoadedMarksMemoryBytes
Size of in-memory representations of loaded marks.
LoadedPrimaryIndexBytes
Number of rows of primary key loaded.
LoadedPrimaryIndexFiles
Number of primary index files loaded.
LoadedPrimaryIndexRows
Number of rows of primary key loaded.
LoadedStatisticsMicroseconds
Elapsed time of loading statistics from parts
LoadingMarksTasksCanceled
Number of times background tasks for loading marks were canceled
LocalReadThrottlerBytes
Bytes passed through 'max_local_read_bandwidth_for_server'/'max_local_read_bandwidth' throttler.
LocalReadThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_local_read_bandwidth_for_server'/'max_local_read_bandwidth' throttling.
LocalThreadPoolBusyMicroseconds
Total time threads have spent executing the actual work.
LocalThreadPoolExpansions
Counts the total number of times threads have been borrowed from the global thread pool to expand local thread pools.
LocalThreadPoolJobWaitTimeMicroseconds
Measures the elapsed time from when a job is scheduled in the thread pool to when it is picked up for execution by a worker thread. This metric helps identify delays in job processing, indicating the responsiveness of the thread pool to new tasks.
LocalThreadPoolJobs
Counts the number of jobs that have been pushed to the local thread pools.
LocalThreadPoolLockWaitMicroseconds
Total time threads have spent waiting for locks in the local thread pools.
LocalThreadPoolShrinks
Counts the total number of times threads have been returned to the global thread pool from local thread pools.
LocalThreadPoolThreadCreationMicroseconds
Total time local thread pools have spent waiting to borrow a thread from the global pool.
LocalWriteThrottlerBytes
Bytes passed through 'max_local_write_bandwidth_for_server'/'max_local_write_bandwidth' throttler.
LocalWriteThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_local_write_bandwidth_for_server'/'max_local_write_bandwidth' throttling.
LogDebug
Number of log messages with level Debug
LogError
Number of log messages with level Error
LogFatal
Number of log messages with level Fatal
LogInfo
Number of log messages with level Info
LogTest
Number of log messages with level Test
LogTrace
Number of log messages with level Trace
LogWarning
Number of log messages with level Warning
LoggerElapsedNanoseconds
Cumulative time spend in logging
MMappedFileCacheHits
Number of times a file has been found in the MMap cache (for the 'mmap' read_method), so we didn't have to mmap it again.
MMappedFileCacheMisses
Number of times a file has not been found in the MMap cache (for the 'mmap' read_method), so we had to mmap it again.
MainConfigLoads
Number of times the main configuration was reloaded.
MarkCacheEvictedBytes
Number of bytes evicted from the mark cache.
MarkCacheEvictedFiles
Number of mark files evicted from the mark cache.
MarkCacheEvictedMarks
Number of marks evicted from the mark cache.
MarkCacheHits
Number of times an entry has been found in the mark cache, so we didn't have to load a mark file.
MarkCacheMisses
Number of times an entry has not been found in the mark cache, so we had to load a mark file in memory, which is a costly operation, adding to query latency.
MarksTasksFromCache
Number of times marks were loaded synchronously because they were already present in the cache.
MemoryAllocatedWithoutCheck
Number of times memory has been allocated without checking for memory constraints.
MemoryAllocatedWithoutCheckBytes
Amount of bytes that has been allocated without checking for memory constraints.
MemoryAllocatorPurge
Total number of times memory allocator purge was requested
MemoryAllocatorPurgeTimeMicroseconds
Total time spent for memory allocator purge
MemoryOvercommitWaitTimeMicroseconds
Total time spent in waiting for memory to be freed in OvercommitTracker.
MemoryWorkerRun
Number of runs done by MemoryWorker in background
MemoryWorkerRunElapsedMicroseconds
Total time spent by MemoryWorker for background work
Merge
Number of launched background merges.
MergeExecuteMilliseconds
Total busy time spent for execution of background merges
MergeHorizontalStageExecuteMilliseconds
Total busy time spent for execution of horizontal stage of background merges
MergeHorizontalStageTotalMilliseconds
Total time spent for horizontal stage of background merges
MergeMutateBackgroundExecutorTaskCancelMicroseconds
Time spent in cancel() for MergeMutate executor tasks.
MergeMutateBackgroundExecutorTaskExecuteStepMicroseconds
Time spent in executeStep() for MergeMutate executor tasks.
MergeMutateBackgroundExecutorTaskResetMicroseconds
Time spent resetting task for MergeMutate executor.
MergeMutateBackgroundExecutorWaitMicroseconds
Time spent waiting for completion in MergeMutate executor.
MergePrewarmStageExecuteMilliseconds
Total busy time spent for execution of prewarm stage of background merges
MergePrewarmStageTotalMilliseconds
Total time spent for prewarm stage of background merges
MergeProjectionStageExecuteMilliseconds
Total busy time spent for execution of projection stage of background merges
MergeProjectionStageTotalMilliseconds
Total time spent for projection stage of background merges
MergeSourceParts
Number of source parts scheduled for merges.
MergeTextIndexStageExecuteMilliseconds
Total busy time spent for execution of text index stage of background merges
MergeTextIndexStageTotalMilliseconds
Total time spent for text index stage of background merges
MergeTotalMilliseconds
Total time spent for background merges
MergeTreeAllRangesAnnouncementsSent
The number of announcements sent from the remote server to the initiator server about the set of data parts (for MergeTree tables). Measured on the remote server side.
MergeTreeAllRangesAnnouncementsSentElapsedMicroseconds
Time spent in sending the announcement from the remote server to the initiator server about the set of data parts (for MergeTree tables). Measured on the remote server side.
MergeTreeDataProjectionWriterBlocks
Number of blocks INSERTed to MergeTree tables projection. Each block forms a data part of level zero.
MergeTreeDataProjectionWriterBlocksAlreadySorted
Number of blocks INSERTed to MergeTree tables projection that appeared to be already sorted.
MergeTreeDataProjectionWriterCompressedBytes
Bytes written to filesystem for data INSERTed to MergeTree tables projection.
MergeTreeDataProjectionWriterMergingBlocksMicroseconds
Time spent merging blocks
MergeTreeDataProjectionWriterRows
Number of rows INSERTed to MergeTree tables projection.
MergeTreeDataProjectionWriterSortingBlocksMicroseconds
Time spent sorting blocks (for projection it might be a key different from table's sorting key)
MergeTreeDataProjectionWriterUncompressedBytes
Uncompressed bytes (for columns as they stored in memory) INSERTed to MergeTree tables projection.
MergeTreeDataWriterBlocks
Number of blocks INSERTed to MergeTree tables. Each block forms a data part of level zero.
MergeTreeDataWriterBlocksAlreadySorted
Number of blocks INSERTed to MergeTree tables that appeared to be already sorted.
MergeTreeDataWriterCompressedBytes
Bytes written to filesystem for data INSERTed to MergeTree tables.
MergeTreeDataWriterMergingBlocksMicroseconds
Time spent merging input blocks (for special MergeTree engines)
MergeTreeDataWriterProjectionsCalculationMicroseconds
Time spent calculating projections
MergeTreeDataWriterRows
Number of rows INSERTed to MergeTree tables.
MergeTreeDataWriterSkipIndicesCalculationMicroseconds
Time spent calculating skip indices
MergeTreeDataWriterSortingBlocksMicroseconds
Time spent sorting blocks
MergeTreeDataWriterStatisticsCalculationMicroseconds
Time spent calculating statistics
MergeTreeDataWriterUncompressedBytes
Uncompressed bytes (for columns as they stored in memory) INSERTed to MergeTree tables.
MergeTreePrefetchedReadPoolInit
Time spent preparing tasks in MergeTreePrefetchedReadPool
MergeTreeReadTaskRequestsReceived
The number of callbacks requested from the remote server back to the initiator server to choose the read task (for MergeTree tables). Measured on the initiator server side.
MergeTreeReadTaskRequestsSent
The number of callbacks requested from the remote server back to the initiator server to choose the read task (for MergeTree tables). Measured on the remote server side.
MergeTreeReadTaskRequestsSentElapsedMicroseconds
Time spent in callbacks requested from the remote server back to the initiator server to choose the read task (for MergeTree tables). Measured on the remote server side.
MergeVerticalStageExecuteMilliseconds
Total busy time spent for execution of vertical stage of background merges
MergeVerticalStageTotalMilliseconds
Total time spent for vertical stage of background merges
MergeWrittenRows
Number of rows written during the merge.
MergedColumns
Number of columns merged during the horizontal stage of merges.
MergedIntoCompactParts
Number of parts merged into Compact format.
MergedIntoWideParts
Number of parts merged into Wide format.
MergedRows
Rows read for background merges. This is the number of rows before merge.
MergedUncompressedBytes
Uncompressed bytes (for columns as they stored in memory) that was read for background merges. This is the number before merge.
MergerMutatorPartsInRangesForMergeCount
Amount of candidate parts for merge
MergerMutatorPrepareRangesForMergeElapsedMicroseconds
Time spent to prepare parts ranges which can be merged according to merge predicate.
MergerMutatorRangesForMergeCount
Amount of candidate ranges for merge
MergerMutatorSelectPartsForMergeElapsedMicroseconds
Time spent to select parts from ranges which can be merged.
MergerMutatorSelectRangePartsCount
Amount of parts in selected range for merge
MergerMutatorsGetPartsForMergeElapsedMicroseconds
Time spent to take data parts snapshot to build ranges from them.
MergesRejectedByMemoryLimit
Number of background merges rejected due to memory limit
MergesThrottlerBytes
Bytes passed through 'max_merges_bandwidth_for_server' throttler.
MergesThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_merges_bandwidth_for_server' throttling.
MergingSortedMilliseconds
Total time spent while merging sorted columns
MetadataFromKeeperBackgroundCleanupBlobs
Number of times an empty blob layout part was cleaned up by background task
MetadataFromKeeperBackgroundCleanupErrors
Number of times an error was encountered in background cleanup task
MetadataFromKeeperBackgroundCleanupObjects
Number of times a old deleted object clean up was performed by background task
MetadataFromKeeperBackgroundCleanupTransactions
Number of times old transaction idempotency token was cleaned up by background task
MetadataFromKeeperCacheHit
Number of times an object storage metadata request was answered from cache without making request to Keeper
MetadataFromKeeperCacheMiss
Number of times an object storage metadata request had to be answered from Keeper
MetadataFromKeeperCacheTooManyInvalidated
Number of times filesystem cache returned too many invalidated entries
MetadataFromKeeperCacheUpdateMicroseconds
Total time spent in updating the cache including waiting for responses from Keeper
MetadataFromKeeperCleanupTransactionCommit
Number of times metadata transaction commit for deleted objects cleanup was attempted
MetadataFromKeeperCleanupTransactionCommitRetry
Number of times metadata transaction commit for deleted objects cleanup was retried
MetadataFromKeeperIndividualOperations
Number of paths read or written by single or multi requests to Keeper
MetadataFromKeeperIndividualOperationsMicroseconds
Time spend during single or multi requests to Keeper
MetadataFromKeeperOperations
Number of times a request was made to Keeper
MetadataFromKeeperReconnects
Number of times a reconnect to Keeper was done
MetadataFromKeeperTransactionCommit
Number of times metadata transaction commit was attempted
MetadataFromKeeperTransactionCommitRetry
Number of times metadata transaction commit was retried
MetadataFromKeeperUpdateCacheOneLevel
Number of times a cache update for one level of directory tree was done
MoveBackgroundExecutorTaskCancelMicroseconds
Time spent in cancel() for Move executor tasks.
MoveBackgroundExecutorTaskExecuteStepMicroseconds
Time spent in executeStep() for Move executor tasks.
MoveBackgroundExecutorTaskResetMicroseconds
Time spent resetting task for Move executor.
MoveBackgroundExecutorWaitMicroseconds
Time spent waiting for completion in Move executor.
MutateTaskProjectionsCalculationMicroseconds
Time spent calculating projections in mutations
MutatedRows
Rows read for mutations. This is the number of rows before mutation
MutatedUncompressedBytes
Uncompressed bytes (for columns as they stored in memory) that was read for mutations. This is the number before mutation.
MutationAffectedRowsUpperBound
The upper bound of number of rows that were affected by mutation (e.g. number of rows that satisfy the predicate of UPDATE or DELETE mutation). The actual number may be slightly less
MutationAllPartColumns
Number of times when task to mutate all columns in part was created
MutationCreatedEmptyParts
Number of total parts which were replaced to empty parts instead of running mutation
MutationExecuteMilliseconds
Total busy time spent for execution of mutations.
MutationSomePartColumns
Number of times when task to mutate some columns in part was created
MutationTotalMilliseconds
Total time spent for mutations.
MutationTotalParts
Number of total parts for which mutations tried to be applied
MutationUntouchedParts
Number of total parts for which mutations tried to be applied but which was completely skipped according to predicate
MutationsAppliedOnFlyInAllReadTasks
Total number of applied mutations on-fly among all read tasks
MutationsThrottlerBytes
Bytes passed through 'max_mutations_bandwidth_for_server' throttler.
MutationsThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_mutations_bandwidth_for_server' throttling.
NaiveBayesClassifierModelsAllocatedBytes
Number of bytes allocated for Naive Bayes Classifier models.
NaiveBayesClassifierModelsLoaded
Number of Naive Bayes Classifier models loaded.
NetworkReceiveBytes
Total number of bytes received from network. Only ClickHouse-related network interaction is included, not by 3rd party libraries.
NetworkReceiveElapsedMicroseconds
Total time spent waiting for data to receive or receiving data from network. Only ClickHouse-related network interaction is included, not by 3rd party libraries.
NetworkSendBytes
Total number of bytes send to network. Only ClickHouse-related network interaction is included, not by 3rd party libraries.
NetworkSendElapsedMicroseconds
Total time spent waiting for data to send to network or sending data to network. Only ClickHouse-related network interaction is included, not by 3rd party libraries.
NotCreatedLogEntryForMerge
Log entry to merge parts in ReplicatedMergeTree is not created due to concurrent log update by another replica.
NotCreatedLogEntryForMutation
Log entry to mutate parts in ReplicatedMergeTree is not created due to concurrent log update by another replica.
OSCPUVirtualTimeMicroseconds
CPU time spent seen by OS. Does not include involuntary waits due to virtualization.
OSCPUWaitMicroseconds
Total time a thread was ready for execution but waiting to be scheduled by OS, from the OS point of view.
OSIOWaitMicroseconds
Total time a thread spent waiting for a result of IO operation, from the OS point of view. This is real IO that doesn't include page cache.
OSReadBytes
Number of bytes read from disks or block devices. Doesn't include bytes read from page cache. May include excessive data due to block size, readahead, etc.
OSReadChars
Number of bytes read from filesystem, including page cache, as well as network and other files.
OSWriteBytes
Number of bytes written to disks or block devices. Doesn't include bytes that are in page cache dirty pages. May not include data that was written by OS asynchronously.
OSWriteChars
Number of bytes written to filesystem, including page cache, as well as network and other files.
ObjectStorageGlobFilteredObjects
Objects that did not match the glob or regex pattern and were skipped during listing.
ObjectStorageListedObjects
Total objects returned by object storage listing API before any filtering.
ObjectStoragePredicateFilteredObjects
Objects removed by virtual column predicate filtering on _path/_file.
ObjectStorageQueueCancelledFiles
Number cancelled files in StorageS3(Azure)Queue
ObjectStorageQueueCleanupMaxSetSizeOrTTLMicroseconds
Time spent to set file as failed
ObjectStorageQueueCommitRequests
Number of keeper requests to commit files as either failed or processed
ObjectStorageQueueExceptionsDuringInsert
Number of exceptions during insert in S3(Azure)Queue
ObjectStorageQueueExceptionsDuringRead
Number of exceptions during read in S3(Azure)Queue
ObjectStorageQueueFailedFiles
Number of files which failed to be processed
ObjectStorageQueueFailedToBatchSetProcessing
Number of times batched set processing request failed
ObjectStorageQueueFilteredFiles
Number of filtered files in StorageS3(Azure)Queue
ObjectStorageQueueInsertIterations
Number of insert iterations
ObjectStorageQueueListedFiles
Number of listed files in StorageS3(Azure)Queue
ObjectStorageQueueMovedObjects
Number of objects moved as part of after_processing = move
ObjectStorageQueueProcessedFiles
Number of files which were processed
ObjectStorageQueueProcessedRows
Number of processed rows in StorageS3(Azure)Queue
ObjectStorageQueuePullMicroseconds
Time spent to read file data
ObjectStorageQueueReadBytes
Number of read bytes (not equal to the number of actually inserted bytes)
ObjectStorageQueueReadFiles
Number of read files (not equal to the number of actually inserted files)
ObjectStorageQueueReadRows
Number of read rows (not equal to the number of actually inserted rows)
ObjectStorageQueueRemovedObjects
Number of objects removed as part of after_processing = delete
ObjectStorageQueueSuccessfulCommits
Number of successful keeper commits
ObjectStorageQueueTaggedObjects
Number of objects tagged as part of after_processing = tag
ObjectStorageQueueTrySetProcessingFailed
The number of times we unsuccessfully set file as processing
ObjectStorageQueueTrySetProcessingRequests
The number of times we tried to make set processing request
ObjectStorageQueueTrySetProcessingSucceeded
The number of times we successfully set file as processing
ObjectStorageQueueUnsuccessfulCommits
Number of unsuccessful keeper commits
ObjectStorageReadObjects
Objects actually opened for reading by the object storage source.
ObsoleteReplicatedParts
Number of times a data part was covered by another data part that has been fetched from a replica (so, we have marked a covered data part as obsolete and no longer needed).
OpenedFileCacheHits
Number of times a file has been found in the opened file cache, so we didn't have to open it again.
OpenedFileCacheMicroseconds
Amount of time spent executing OpenedFileCache methods.
OpenedFileCacheMisses
Number of times a file has been found in the opened file cache, so we had to open it again.
OtherQueryTimeMicroseconds
Total time of queries that are not SELECT or INSERT.
OverflowAny
Number of times approximate GROUP BY was in effect: when aggregation was performed only on top of first 'max_rows_to_group_by' unique keys and other keys were ignored due to 'group_by_overflow_mode' = 'any'.
OverflowBreak
Number of times, data processing was cancelled by query complexity limitation with setting '*_overflow_mode' = 'break' and the result is incomplete.
OverflowThrow
Number of times, data processing was cancelled by query complexity limitation with setting '*_overflow_mode' = 'throw' and exception was thrown.
PageCacheHits
Number of times a block of data has been found in the userspace page cache.
PageCacheMisses
Number of times a block of data has not been found in the userspace page cache.
PageCacheOvercommitResize
Number of times the userspace page cache was auto-resized to free memory during a memory allocation.
PageCacheReadBytes
Number of bytes read from userspace page cache.
PageCacheResized
Number of times the userspace page cache was auto-resized (typically happens a few times per second, controlled by memory_worker_period_ms).
PageCacheWeightLost
Number of bytes evicted from the userspace page cache
ParallelReplicasAnnouncementMicroseconds
Time spent to send an announcement
ParallelReplicasAvailableCount
Number of replicas available to execute a query with task-based parallel replicas
ParallelReplicasCollectingOwnedSegmentsMicroseconds
Time spent collecting segments meant by hash
ParallelReplicasDeniedRequests
Number of completely denied requests to the initiator
ParallelReplicasHandleAnnouncementMicroseconds
Time spent processing replicas announcements
ParallelReplicasHandleRequestMicroseconds
Time spent processing requests for marks from replicas
ParallelReplicasNumRequests
Number of requests to the initiator.
ParallelReplicasProcessingPartsMicroseconds
Time spent processing data parts
ParallelReplicasQueryCount
Number of (sub)queries executed using parallel replicas during a query execution
ParallelReplicasReadAssignedForStealingMarks
Sum across all replicas of how many of scheduled marks were assigned for stealing by consistent hash
ParallelReplicasReadAssignedMarks
Sum across all replicas of how many of scheduled marks were assigned by consistent hash
ParallelReplicasReadMarks
How many marks were read by the given replica
ParallelReplicasReadRequestMicroseconds
Time spent for read requests
ParallelReplicasReadUnassignedMarks
Sum across all replicas of how many unassigned marks were scheduled
ParallelReplicasStealingByHashMicroseconds
Time spent collecting segments meant for stealing by hash
ParallelReplicasStealingLeftoversMicroseconds
Time spent collecting orphaned segments
ParallelReplicasUnavailableCount
Number of replicas which was chosen, but found to be unavailable during query execution with task-based parallel replicas
ParallelReplicasUsedCount
Number of replicas used to execute a query with task-based parallel replicas
ParquetColumnsFilterExpression
The total number of columns that were passed through filter
ParquetDecodingTaskBatches
Task groups sent to a thread pool by parquet reader
ParquetDecodingTasks
Tasks issued by parquet reader
ParquetFetchWaitTimeMicroseconds
Time of waiting for parquet file reads from decoding threads (not prefetching threads)
ParquetMetadataCacheHits
Number of times parquet metadata has been found in the cache.
ParquetMetadataCacheMisses
Number of times parquet metadata has not been found in the cache and had to be read from disk.
ParquetMetadataCacheWeightLost
Approximate number of bytes evicted from the parquet metadata cache.
ParquetPrefetcherReadEntireFile
The total number of read with ReadMode::EntireFileIsInMemory by DB::Parquet::Prefetcher
ParquetPrefetcherReadRandomRead
The total number of reads with ReadMode::RandomRead by DB::Parquet::Prefetcher
ParquetPrefetcherReadSeekAndRead
The total number of reads with ReadMode::SeekAndRead by DB::Parquet::Prefetcher
ParquetPrunedRowGroups
The total number of row groups pruned from parquet data
ParquetReadRowGroups
The total number of row groups read from parquet data
ParquetRowsFilterExpression
The total number of rows that were passed through filter
PartsLockHoldMicroseconds
Total time spent holding data parts lock in MergeTree tables
PartsLockWaitMicroseconds
Total time spent waiting for data parts lock in MergeTree tables
PartsLocks
Number of times data parts lock has been acquired for MergeTree tables
PatchesAcquireLockMicroseconds
Total number of microseconds spent to acquire lock for executing lightweight updates
PatchesAcquireLockTries
Total number of tries to acquire lock for executing lightweight updates
PatchesAppliedInAllReadTasks
Total number of applied patch parts among all read tasks
PatchesJoinAppliedInAllReadTasks
Total number of applied patch parts with Join mode among all read tasks
PatchesJoinRowsAddedToHashTable
Total number of rows added to hash tables when applying patch parts with Join mode
PatchesMergeAppliedInAllReadTasks
Total number of applied patch parts with Merge mode among all read tasks
PatchesReadRows
Total number of rows read from patch parts
PatchesReadUncompressedBytes
Total number of uncompressed bytes read from patch parts
PerfAlignmentFaults
Number of alignment faults. These happen when unaligned memory accesses happen; the kernel can handle these but it reduces performance. This happens only on some architectures (never on x86).
PerfBranchInstructions
Retired branch instructions. Prior to Linux 2.6.35, this used the wrong event on AMD processors.
PerfBranchMisses
Mispredicted branch instructions.
PerfBusCycles
Bus cycles, which can be different from total cycles.
PerfCPUClock
The CPU clock, a high-resolution per-CPU timer
PerfCPUCycles
Total cycles. Be wary of what happens during CPU frequency scaling.
PerfCPUMigrations
Number of times the process has migrated to a new CPU
PerfCacheMisses
Cache misses. Usually this indicates Last Level Cache misses; this is intended to be used in conjunction with the PERFCOUNTHWCACHEREFERENCES event to calculate cache miss rates.
PerfCacheReferences
Cache accesses. Usually, this indicates Last Level Cache accesses, but this may vary depending on your CPU. This may include prefetches and coherency messages; again this depends on the design of your CPU.
PerfContextSwitches
Number of context switches
PerfDataTLBMisses
Data TLB misses
PerfDataTLBReferences
Data TLB references
PerfEmulationFaults
Number of emulation faults. The kernel sometimes traps on unimplemented instructions and emulates them for user space. This can negatively impact performance.
PerfInstructionTLBMisses
Instruction TLB misses
PerfInstructionTLBReferences
Instruction TLB references
PerfInstructions
Retired instructions. Be careful, these can be affected by various issues, most notably hardware interrupt counts.
PerfLocalMemoryMisses
Local NUMA node memory read misses
PerfLocalMemoryReferences
Local NUMA node memory reads
PerfMinEnabledRunningTime
Running time for event with minimum enabled time. Used to track the amount of event multiplexing
PerfMinEnabledTime
For all events, minimum time that an event was enabled. Used to track event multiplexing influence
PerfRefCPUCycles
Total cycles; not affected by CPU frequency scaling.
PerfStalledCyclesBackend
Stalled cycles during retirement.
PerfStalledCyclesFrontend
Stalled cycles during issue.
PerfTaskClock
A clock count specific to the task that is running
PolygonsAddedToPool
A polygon has been added to the cache (pool) for the 'pointInPolygon' function.
PolygonsInPoolAllocatedBytes
The number of bytes for polygons added to the cache (pool) for the 'pointInPolygon' function.
PreferredWarmedUnmergedParts
See setting prefer_warmed_unmerged_parts_seconds. Number of times read queries used outdated pre-merge parts that are in cache instead of merged part that wasn't pulled into cache by CacheWarmer yet.
PrimaryIndexCacheHits
Number of times an entry has been found in the primary index cache, so we didn't have to load a index file.
PrimaryIndexCacheMisses
Number of times an entry has not been found in the primary index cache, so we had to load a index file in memory, which is a costly operation, adding to query latency.
QueriesWithSubqueries
Count queries with all subqueries
Query
Number of queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.
QueryBackupThrottlerBytes
Bytes passed through 'max_backup_bandwidth' throttler.
QueryBackupThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_backup_bandwidth' throttling.
QueryCacheAgeSeconds
The sum of ages of found query cache entries in seconds. The value is set both for hits and misses.
QueryCacheHits
Number of times a query result has been found in the query cache (and query computation was avoided). Only updated for SELECT queries with SETTING use_query_cache = 1.
QueryCacheMisses
Number of times a query result has not been found in the query cache (and required query computation). Only updated for SELECT queries with SETTING use_query_cache = 1.
QueryCacheReadBytes
The number of (uncompressed) bytes read from the query cache.
QueryCacheReadRows
The number of rows read from the query cache.
QueryCacheWrittenBytes
The number of (uncompressed) bytes saved into the query cache
QueryCacheWrittenRows
The number of rows saved into the query cache.
QueryConditionCacheHits
Number of times an entry has been found in the query condition cache (and reading of marks can be skipped). Only updated for SELECT queries with SETTING use_query_condition_cache = 1.
QueryConditionCacheMisses
Number of times an entry has not been found in the query condition cache (and reading of mark cannot be skipped). Only updated for SELECT queries with SETTING use_query_condition_cache = 1.
QueryLocalReadThrottlerBytes
Bytes passed through 'max_local_read_bandwidth' throttler.
QueryLocalReadThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_local_read_bandwidth' throttling.
QueryLocalWriteThrottlerBytes
Bytes passed through 'max_local_write_bandwidth' throttler.
QueryLocalWriteThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_local_write_bandwidth' throttling.
QueryMaskingRulesMatch
Number of times query masking rules was successfully matched.
QueryMemoryLimitExceeded
Number of times when memory limit exceeded for query.
QueryPlanOptimizeMicroseconds
Total time spent executing query plan optimizations.
QueryPreempted
How many times tasks are paused and waiting due to 'priority' setting
QueryProfilerConcurrencyOverruns
Number of times we drop processing of a query profiler signal due to too many concurrent query profilers in other threads, which may indicate overload.
QueryProfilerErrors
Invalid memory accesses during asynchronous stack unwinding.
QueryProfilerRuns
Number of times QueryProfiler had been run.
QueryProfilerSignalOverruns
Number of times we drop processing of a query profiler signal due to overrun plus the number of signals that OS has not delivered due to overrun.
QueryRemoteReadThrottlerBytes
Bytes passed through 'max_remote_read_network_bandwidth' throttler.
QueryRemoteReadThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_remote_read_network_bandwidth' throttling.
QueryRemoteWriteThrottlerBytes
Bytes passed through 'max_remote_write_network_bandwidth' throttler.
QueryRemoteWriteThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_remote_write_network_bandwidth' throttling.
QueryTimeMicroseconds
Total time of all queries.
QuorumFailedInserts
Number of inserts failed due to quorum not reaching.
QuorumParts
Number of data parts written with quorum. It counts as one part for sync insert and maybe up to async inserts count for insert which flushes async inserts.
QuorumWaitMicroseconds
Total time spent waiting for quorum during inserts.
RWLockAcquiredReadLocks
Number of times a read lock was acquired (in a heavy RWLock).
RWLockAcquiredWriteLocks
Number of times a write lock was acquired (in a heavy RWLock).
RWLockReadersWaitMilliseconds
Total time spent waiting for a read lock to be acquired (in a heavy RWLock).
RWLockWritersWaitMilliseconds
Total time spent waiting for a write lock to be acquired (in a heavy RWLock).
ReadBackoff
Number of times the number of query processing threads was lowered due to slow reads.
ReadBufferFromAzureBytes
Bytes read from Azure.
ReadBufferFromAzureInitMicroseconds
Time spent initializing connection to Azure.
ReadBufferFromAzureMicroseconds
Time spent on reading from Azure.
ReadBufferFromAzureRequestsErrors
Number of exceptions while reading from Azure
ReadBufferFromFileDescriptorRead
Number of reads (read/pread) from a file descriptor. Does not include sockets.
ReadBufferFromFileDescriptorReadBytes
Number of bytes read from file descriptors. If the file is compressed, this will show the compressed data size.
ReadBufferFromFileDescriptorReadFailed
Number of times the read (read/pread) from a file descriptor have failed.
ReadBufferFromS3Bytes
Bytes read from S3.
ReadBufferFromS3InitMicroseconds
Time spent initializing connection to S3.
ReadBufferFromS3Microseconds
Time spent on reading from S3.
ReadBufferFromS3RequestsErrors
Number of exceptions while reading from S3.
ReadBufferSeekCancelConnection
Number of seeks which lead to new connection (s3, http)
ReadCompressedBytes
Number of bytes (the number of bytes before decompression) read from compressed sources (files, network).
ReadPatchesMicroseconds
Total time spent reading patch parts
ReadTaskRequestsReceived
The number of callbacks requested from the remote server back to the initiator server to choose the read task (for s3Cluster table function and similar). Measured on the initiator server side.
ReadTaskRequestsSent
The number of callbacks requested from the remote server back to the initiator server to choose the read task (for s3Cluster table function and similar). Measured on the remote server side.
ReadTaskRequestsSentElapsedMicroseconds
Time spent in callbacks requested from the remote server back to the initiator server to choose the read task (for s3Cluster table function and similar). Measured on the remote server side.
ReadTasksWithAppliedMutationsOnFly
Total number of read tasks for which there was any mutation applied on fly
ReadTasksWithAppliedPatches
Total number of read tasks for which there was any patch part applied
ReadWriteBufferFromHTTPBytes
Total size of payload bytes received and sent by ReadWriteBufferFromHTTP. Doesn't include HTTP headers.
ReadWriteBufferFromHTTPRequestsSent
Number of HTTP requests sent by ReadWriteBufferFromHTTP
RealTimeMicroseconds
Total (wall clock) time spent in processing (queries and other tasks) threads (note that this is a sum).
RefreshableViewLockTableRetry
How many times a SELECT from refreshable materialized view had to switch to a new table because the old table was dropped
RefreshableViewRefreshFailed
How many times refreshable materialized views failed to refresh
RefreshableViewRefreshSuccess
How many times refreshable materialized views refreshed
RefreshableViewSyncReplicaRetry
How many times a SELECT from refreshable materialized view failed and retried an implicit SYNC REPLICA
RefreshableViewSyncReplicaSuccess
How many times a SELECT from refreshable materialized view did an implicit SYNC REPLICA
RegexpLocalCacheHit
Number of times we fetched compiled regular expression from a local cache.
RegexpLocalCacheMiss
Number of times we failed to fetch compiled regular expression from a local cache.
RegexpWithMultipleNeedlesCreated
Regular expressions with multiple needles (VectorScan library) compiled.
RegexpWithMultipleNeedlesGlobalCacheHit
Number of times we fetched compiled regular expression with multiple needles (VectorScan library) from the global cache.
RegexpWithMultipleNeedlesGlobalCacheMiss
Number of times we failed to fetch compiled regular expression with multiple needles (VectorScan library) from the global cache.
RejectedInserts
Number of times the INSERT of a block to a MergeTree table was rejected with 'Too many parts' exception due to high number of active data parts for partition.
RejectedLightweightUpdates
Number of time the lightweight update was rejected due to too many uncompressed bytes in patches.
RejectedMutations
Number of times the mutation of a MergeTree table was rejected with 'Too many mutations' exception due to high number of unfinished mutations for table.
RemoteFSBuffers
Number of buffers created for asynchronous reading from remote filesystem
RemoteFSCancelledPrefetches
Number of cancelled prefecthes (because of seek)
RemoteFSLazySeeks
Number of lazy seeks
RemoteFSPrefetchedBytes
Number of bytes from prefecthed buffer
RemoteFSPrefetchedReads
Number of reads from prefecthed buffer
RemoteFSPrefetches
Number of prefetches made with asynchronous reading from remote filesystem
RemoteFSSeeks
Total number of seeks for async buffer
RemoteFSSeeksWithReset
Number of seeks which lead to a new connection
RemoteFSUnprefetchedBytes
Number of bytes from unprefetched buffer
RemoteFSUnprefetchedReads
Number of reads from unprefetched buffer
RemoteFSUnusedPrefetches
Number of prefetches pending at buffer destruction
RemoteReadThrottlerBytes
Bytes passed through 'max_remote_read_network_bandwidth_for_server'/'max_remote_read_network_bandwidth' throttler.
RemoteReadThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_remote_read_network_bandwidth_for_server'/'max_remote_read_network_bandwidth' throttling.
RemoteWriteThrottlerBytes
Bytes passed through 'max_remote_write_network_bandwidth_for_server'/'max_remote_write_network_bandwidth' throttler.
RemoteWriteThrottlerSleepMicroseconds
Total time a query was sleeping to conform 'max_remote_write_network_bandwidth_for_server'/'max_remote_write_network_bandwidth' throttling.
ReplacingSortedMilliseconds
Total time spent while replacing sorted columns
ReplicaPartialShutdown
How many times Replicated table has to deinitialize its state due to session expiration in ZooKeeper. The state is reinitialized every time when ZooKeeper is available again.
ReplicatedCoveredPartsInZooKeeperOnStart
For debugging purposes. Number of parts in ZooKeeper that have a covering part, but doesn't exist on disk. Checked on server start.
ReplicatedDataLoss
Number of times a data part that we wanted doesn't exist on any replica (even on replicas that are offline right now). That data parts are definitely lost. This is normal due to asynchronous replication (if quorum inserts were not enabled), when the replica on which the data part was written was failed and when it became online after fail it doesn't contain that data part.
ReplicatedPartChecks
Number of times we had to perform advanced search for a data part on replicas or to clarify the need of an existing data part.
ReplicatedPartChecksFailed
Number of times the advanced search for a data part on replicas did not give result or when unexpected part has been found and moved away.
ReplicatedPartFailedFetches
Number of times a data part was failed to download from replica of a ReplicatedMergeTree table.
ReplicatedPartFetches
Number of times a data part was downloaded from replica of a ReplicatedMergeTree table.
ReplicatedPartFetchesOfMerged
Number of times we prefer to download already merged part from replica of ReplicatedMergeTree table instead of performing a merge ourself (usually we prefer doing a merge ourself to save network traffic). This happens when we have not all source parts to perform a merge or when the data part is old enough.
ReplicatedPartMerges
Number of times data parts of ReplicatedMergeTree tables were successfully merged.
ReplicatedPartMutations
Number of times data parts of ReplicatedMergeTree tables were successfully mutated.
RestorePartsSkippedBytes
Total size of files skipped while restoring parts
RestorePartsSkippedFiles
Number of files skipped while restoring parts
RowsReadByMainReader
Number of rows read from MergeTree tables by the main reader (after PREWHERE step).
RowsReadByPrewhereReaders
Number of rows read from MergeTree tables (in total) by prewhere readers.
RuntimeDataflowStatisticsInputBytes
Collected statistics on the number of bytes replicas would read if the query was executed with parallel replicas
RuntimeDataflowStatisticsOutputBytes
Collected statistics on the number of bytes replicas would send to the initiator if the query was executed with parallel replicas
RuntimeFilterBlocksProcessed
Number of blocks processed by JOIN Runtime Filters
RuntimeFilterBlocksSkipped
Number of blocks skipped by JOIN Runtime Filters without processing due to filter being dynamically disabled because of poor filtering ratio
RuntimeFilterRowsChecked
Number of rows checked by JOIN Runtime Filters
RuntimeFilterRowsPassed
Number of rows that passed (not filtered out by) JOIN Runtime Filters
RuntimeFilterRowsSkipped
Number of rows in blocks that were skipped by JOIN Runtime Filters
RuntimeFiltersCreated
Number of distinct JOIN Runtime Filters created within a query
S3AbortMultipartUpload
Number of S3 API AbortMultipartUpload calls.
S3CachedCredentialsProvidersAdded
Total number of newly added credentials providers to the cache
S3CachedCredentialsProvidersReused
Total number of reused credentials provider from the cache
S3Clients
Number of created S3 clients.
S3CompleteMultipartUpload
Number of S3 API CompleteMultipartUpload calls.
S3CopyObject
Number of S3 API CopyObject calls.
S3CreateMultipartUpload
Number of S3 API CreateMultipartUpload calls.
S3DeleteObjects
Number of S3 API DeleteObject(s) calls.
S3GetObject
Number of S3 API GetObject calls.
S3GetObjectTagging
Number of S3 API GetObjectTagging calls.
S3GetRequestThrottlerBlocked
Number of S3 GET and SELECT requests blocked by throttler.
S3GetRequestThrottlerCount
Number of S3 GET and SELECT requests passed through throttler: blocked and not blocked.
S3GetRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform S3 GET and SELECT request throttling.
S3HeadObject
Number of S3 API HeadObject calls.
S3ListObjects
Number of S3 API ListObjects calls.
S3PutObject
Number of S3 API PutObject calls.
S3PutRequestThrottlerBlocked
Number of S3 PUT, COPY, POST and LIST requests blocked by throttler.
S3PutRequestThrottlerCount
Number of S3 PUT, COPY, POST and LIST requests passed through throttler: blocked and not blocked.
S3PutRequestThrottlerSleepMicroseconds
Total time a query was sleeping to conform S3 PUT, COPY, POST and LIST request throttling.
S3QueueSetFileFailedMicroseconds
Time spent to set file as failed
S3QueueSetFileProcessedMicroseconds
Time spent to set file as processed
S3QueueSetFileProcessingMicroseconds
Time spent to set file as processing
S3ReadMicroseconds
Time of GET and HEAD requests to S3 storage.
S3ReadRequestAttempts
Number of attempts for GET and HEAD requests, including the initial try and any retries, but excluding retries performed internally by the S3 retry strategy
S3ReadRequestRetryableErrors
Number of retryable errors for GET and HEAD requests, excluding retries performed internally by the S3 retry strategy
S3ReadRequestsCount
Number of GET and HEAD requests to S3 storage.
S3ReadRequestsErrors
Number of non-throttling errors in GET and HEAD requests to S3 storage.
S3ReadRequestsRedirects
Number of redirects in GET and HEAD requests to S3 storage.
S3ReadRequestsThrottling
Number of 429 and 503 errors in GET and HEAD requests to S3 storage.
S3UploadPart
Number of S3 API UploadPart calls.
S3UploadPartCopy
Number of S3 API UploadPartCopy calls.
S3WriteMicroseconds
Time of POST, DELETE, PUT and PATCH requests to S3 storage.
S3WriteRequestAttempts
Number of attempts for POST, DELETE, PUT and PATCH requests, including the initial try and any retries, but excluding retries performed internally by the retry strategy
S3WriteRequestRetryableErrors
Number of retryable errors for POST, DELETE, PUT and PATCH requests, excluding retries performed internally by the retry strategy
S3WriteRequestsCount
Number of POST, DELETE, PUT and PATCH requests to S3 storage.
S3WriteRequestsErrors
Number of non-throttling errors in POST, DELETE, PUT and PATCH requests to S3 storage.
S3WriteRequestsRedirects
Number of redirects in POST, DELETE, PUT and PATCH requests to S3 storage.
S3WriteRequestsThrottling
Number of 429 and 503 errors in POST, DELETE, PUT and PATCH requests to S3 storage.
ScalarSubqueriesCacheMiss
Number of times a read from a scalar subquery was not cached and had to be calculated completely
ScalarSubqueriesGlobalCacheHit
Number of times a read from a scalar subquery was done using the global cache
ScalarSubqueriesLocalCacheHit
Number of times a read from a scalar subquery was done using the local cache
SchedulerIOReadBytes
Bytes passed through scheduler for IO reads.
SchedulerIOReadRequests
Resource requests passed through scheduler for IO reads.
SchedulerIOReadWaitMicroseconds
Total time a query was waiting on resource requests for IO reads.
SchedulerIOWriteBytes
Bytes passed through scheduler for IO writes.
SchedulerIOWriteRequests
Resource requests passed through scheduler for IO writes.
SchedulerIOWriteWaitMicroseconds
Total time a query was waiting on resource requests for IO writes.
SchemaInferenceCacheEvictions
Number of times a schema from cache was evicted due to overflow
SchemaInferenceCacheHits
Number of times the requested source is found in schema cache
SchemaInferenceCacheInvalidations
Number of times a schema in cache became invalid due to changes in data
SchemaInferenceCacheMisses
Number of times the requested source is not in schema cache
SchemaInferenceCacheNumRowsHits
Number of times the number of rows is found in schema cache during count from files
SchemaInferenceCacheNumRowsMisses
Number of times the requested source is in cache but the number of rows is not in cache while count from files
SchemaInferenceCacheSchemaHits
Number of times the schema is found in schema cache during schema inference
SchemaInferenceCacheSchemaMisses
Number of times the requested source is in cache but the schema is not in cache during schema inference
Seek
Number of times the 'lseek' function was called.
SelectQueriesWithPrimaryKeyUsage
Count SELECT queries which use the primary key to evaluate the WHERE condition
SelectQueriesWithSubqueries
Count SELECT queries with all subqueries
SelectQuery
Same as Query, but only for SELECT queries.
SelectQueryTimeMicroseconds
Total time of SELECT queries.
SelectedBytes
Number of bytes (uncompressed; for columns as they stored in memory) SELECTed from all tables.
SelectedMarks
Number of marks (index granules) selected to read from a MergeTree table.
SelectedMarksTotal
Number of total marks (index granules) before selecting which ones to read from a MergeTree table.
SelectedParts
Number of data parts selected to read from a MergeTree table.
SelectedPartsTotal
Number of total data parts before selecting which ones to read from a MergeTree table.
SelectedRanges
Number of (non-adjacent) ranges in all data parts selected to read from a MergeTree table.
SelectedRows
Number of rows SELECTed from all tables.
SelfDuplicatedAsyncInserts
Number of async inserts in the INSERTed block to a ReplicatedMergeTree table was self deduplicated.
ServerStartupMilliseconds
Time elapsed from starting server to listening to sockets in milliseconds
SharedDatabaseCatalogFailedToApplyState
Number of failures to apply new state in SharedDatabaseCatalog
SharedDatabaseCatalogStateApplicationMicroseconds
Total time spend on application of new state in SharedDatabaseCatalog
SharedMergeTreeCondemnedPartsKillRequest
How many ZooKeeper requests were used to remove condemned parts
SharedMergeTreeCondemnedPartsLockConflict
How many times we failed to acquite lock because of conflict
SharedMergeTreeCondemnedPartsRemoved
How many condemned parts were removed
SharedMergeTreeDataPartsFetchAttempt
How many times we tried to fetch data parts
SharedMergeTreeDataPartsFetchFromPeer
How many times we fetch data parts from peer
SharedMergeTreeDataPartsFetchFromPeerMicroseconds
Data parts fetch from peer microseconds
SharedMergeTreeDataPartsFetchFromS3
How many times we fetch data parts from S3
SharedMergeTreeHandleBlockingParts
How many blocking parts to handle in scheduleDataProcessingJob
SharedMergeTreeHandleBlockingPartsMicroseconds
Time of handling blocking parts in scheduleDataProcessingJob
SharedMergeTreeHandleFetchPartsMicroseconds
Time of handling fetched parts in scheduleDataProcessingJob
SharedMergeTreeHandleOutdatedParts
How many outdated parts to handle in scheduleDataProcessingJob
SharedMergeTreeHandleOutdatedPartsMicroseconds
Time of handling outdated parts in scheduleDataProcessingJob
SharedMergeTreeLoadChecksumAndIndexesMicroseconds
Time of loadColumnsChecksumsIndexes only for SharedMergeTree
SharedMergeTreeMergeMutationAssignmentAttempt
How many times we tried to assign merge or mutation
SharedMergeTreeMergeMutationAssignmentFailedWithConflict
How many times we tried to assign merge or mutation and failed because of conflict in Keeper
SharedMergeTreeMergeMutationAssignmentFailedWithNothingToDo
How many times we tried to assign merge or mutation and failed because nothing to merge
SharedMergeTreeMergeMutationAssignmentSuccessful
How many times we tried to assign merge or mutation
SharedMergeTreeMergePartsMovedToCondemned
How many parts moved to condemned directory
SharedMergeTreeMergePartsMovedToOudated
How many parts moved to oudated directory
SharedMergeTreeMergeSelectingTaskMicroseconds
Merge selecting task microseconds for SMT
SharedMergeTreeMetadataCacheHintLoadedFromCache
Number of times metadata cache hint was found without going to Keeper
SharedMergeTreeOptimizeAsync
Asynchronous OPTIMIZE queries executed
SharedMergeTreeOptimizeSync
Synchronous OPTIMIZE queries executed
SharedMergeTreeOutdatedPartsConfirmationInvocations
How many invocations were made to confirm outdated parts
SharedMergeTreeOutdatedPartsConfirmationRequest
How many ZooKeeper requests were used to config outdated parts
SharedMergeTreeOutdatedPartsHTTPRequest
How many HTTP requests were send to confirm outdated parts
SharedMergeTreeOutdatedPartsHTTPResponse
How many HTTP responses were send to confirm outdated parts
SharedMergeTreePartsKillerMicroseconds
How much time does parts killer main thread takes
SharedMergeTreePartsKillerParts
How many parts has been scheduled by the killer
SharedMergeTreePartsKillerPartsMicroseconds
How many time does it take to remove parts (executed from multiple threads)
SharedMergeTreePartsKillerRuns
How many times parts killer has been running
SharedMergeTreeReplicaSetUpdatesFromZooKeeper
How many times we have update replica set from ZooKeeper
SharedMergeTreeReplicaSetUpdatesFromZooKeeperMicroseconds
How much time we spend to update replica set
SharedMergeTreeReplicaSetUpdatesFromZooKeeperRequests
How many total ZooKeeper requests we made to update replica set
SharedMergeTreeScheduleDataProcessingJob
How many times scheduleDataProcessingJob called/
SharedMergeTreeScheduleDataProcessingJobMicroseconds
scheduleDataProcessingJob execute time
SharedMergeTreeScheduleDataProcessingJobNothingToScheduled
How many times scheduleDataProcessingJob called but nothing to do
SharedMergeTreeSelectPartsForCoordinatedFetchMicroseconds
Time of selectPartsForCoordinatedFetch
SharedMergeTreeSelectPartsForCoordinatedFetchParts
Number of parts selected by selectPartsForCoordinatedFetch
SharedMergeTreeSelectPartsForFullFetchMicroseconds
Time of selectPartsForFullFetch
SharedMergeTreeSelectPartsForFullFetchParts
Number of parts selected by selectPartsForFullFetch
SharedMergeTreeSelectPartsForRendezvousFetchMicroseconds
Time of selectPartsForRendezvousFetch
SharedMergeTreeSelectPartsForRendezvousFetchParts
Number of parts selected by selectPartsForRendezvousFetch
SharedMergeTreeSnapshotPartsCleanRequest
How many times SnapshotCleanerThread decides to clean a part
SharedMergeTreeSnapshotPartsCleanerMicroseconds
How long time SnapshotCleanerThread has run
SharedMergeTreeSnapshotPartsCleanerParts
How long time SnapshotCleanerThread tries to clean a part
SharedMergeTreeSnapshotPartsCleanerPartsMicroseconds
How long time SnapshotCleanerThread takes to clean parts
SharedMergeTreeSnapshotPartsCleanerRuns
How many times SnapshotCleanerThread runs
SharedMergeTreeSnapshotPartsRemoved
How many times SnapshotCleanerThread successfully clean a part
SharedMergeTreeTryUpdateDiskMetadataCacheForPartMicroseconds
Time of tryUpdateDiskMetadataCacheForPart in scheduleDataProcessingJob
SharedMergeTreeVirtualPartsUpdateMicroseconds
Virtual parts update microseconds
SharedMergeTreeVirtualPartsUpdates
Virtual parts update count
SharedMergeTreeVirtualPartsUpdatesByLeader
Virtual parts updates by leader
SharedMergeTreeVirtualPartsUpdatesForMergesOrStatus
Virtual parts updates from non-default background job
SharedMergeTreeVirtualPartsUpdatesFromPeer
Virtual parts updates count from peer
SharedMergeTreeVirtualPartsUpdatesFromPeerMicroseconds
Virtual parts updates from peer microseconds
SharedMergeTreeVirtualPartsUpdatesFromZooKeeper
Virtual parts updates count from ZooKeeper
SharedMergeTreeVirtualPartsUpdatesFromZooKeeperMicroseconds
Virtual parts updates from ZooKeeper microseconds
SharedMergeTreeVirtualPartsUpdatesLeaderFailedElection
Virtual parts updates leader election failed
SharedMergeTreeVirtualPartsUpdatesLeaderSuccessfulElection
Virtual parts updates leader election successful
SharedMergeTreeVirtualPartsUpdatesPeerNotFound
Virtual updates from peer failed because no one found
SharedPartsLockHoldMicroseconds
Total time spent holding shared data parts lock in MergeTree tables
SharedPartsLockWaitMicroseconds
Total time spent waiting for shared data parts lock in MergeTree tables
SharedPartsLocks
Number of times shared data parts lock has been acquired for MergeTree tables
SleepFunctionCalls
Number of times a sleep function (sleep, sleepEachRow) has been called.
SleepFunctionElapsedMicroseconds
Time spent sleeping in a sleep function (sleep, sleepEachRow).
SleepFunctionMicroseconds
Time set to sleep in a sleep function (sleep, sleepEachRow).
SlowRead
Number of reads from a file that were slow. This indicate system overload. Thresholds are controlled by read_backoff_* settings.
SoftPageFaults
The number of soft page faults in query execution threads. Soft page fault usually means a miss in the memory allocator cache, which requires a new memory mapping from the OS and subsequent allocation of a page of physical memory.
StorageBufferErrorOnFlush
Number of times a buffer in the 'Buffer' table has not been able to flush due to error writing in the destination table.
StorageBufferFlush
Number of times a buffer in a 'Buffer' table was flushed.
StorageBufferLayerLockReadersWaitMilliseconds
Time for waiting for Buffer layer during reading.
StorageBufferLayerLockWritersWaitMilliseconds
Time for waiting free Buffer layer to write to (can be used to tune Buffer layers).
StorageBufferPassedAllMinThresholds
Number of times a criteria on min thresholds has been reached to flush a buffer in a 'Buffer' table.
StorageBufferPassedBytesFlushThreshold
Number of times background-only flush threshold on bytes has been reached to flush a buffer in a 'Buffer' table. This is expert-only metric. If you read this and you are not an expert, stop reading.
StorageBufferPassedBytesMaxThreshold
Number of times a criteria on max bytes threshold has been reached to flush a buffer in a 'Buffer' table.
StorageBufferPassedRowsFlushThreshold
Number of times background-only flush threshold on rows has been reached to flush a buffer in a 'Buffer' table. This is expert-only metric. If you read this and you are not an expert, stop reading.
StorageBufferPassedRowsMaxThreshold
Number of times a criteria on max rows threshold has been reached to flush a buffer in a 'Buffer' table.
StorageBufferPassedTimeFlushThreshold
Number of times background-only flush threshold on time has been reached to flush a buffer in a 'Buffer' table. This is expert-only metric. If you read this and you are not an expert, stop reading.
StorageBufferPassedTimeMaxThreshold
Number of times a criteria on max time threshold has been reached to flush a buffer in a 'Buffer' table.
StorageConnectionsCreated
Number of created connections for storages
StorageConnectionsElapsedMicroseconds
Total time spend on creating connections for storages
StorageConnectionsErrors
Number of cases when creation of a connection for storage is failed
StorageConnectionsExpired
Number of expired connections for storages
StorageConnectionsPreserved
Number of preserved connections for storages
StorageConnectionsReset
Number of reset connections for storages
StorageConnectionsReused
Number of reused connections for storages
SummingSortedMilliseconds
Total time spent while summing sorted columns
SuspendSendingQueryToShard
Total count when sending query to shard was suspended when async_query_sending_for_remote is enabled.
SynchronousReadWaitMicroseconds
Time spent in waiting for synchronous reads in asynchronous local read.
SynchronousRemoteReadWaitMicroseconds
Time spent in waiting for synchronous remote reads.
SystemLogErrorOnFlush
Number of times any of the system logs have failed to flush to the corresponding system table. Attempts to flush are repeated.
SystemTimeMicroseconds
Total time spent in processing (queries and other tasks) threads executing CPU instructions in OS kernel mode. This is time spent in syscalls, excluding waiting time during blocking syscalls.
TableFunctionExecute
Number of table function calls.
TextIndexDiscardHint
Number of index granules where a direct reading from the text index was added as hint and was discarded due to low selectivity.
TextIndexHeaderCacheHits
Number of times a header has been found in the cache.
TextIndexHeaderCacheMisses
Number of times a header has not been found in the cache.
TextIndexPostingsCacheHits
Number of times a text index posting list has been found in the cache.
TextIndexPostingsCacheMisses
Number of times a a text index posting list has not been found in the cache.
TextIndexReadDictionaryBlocks
Number of times a text index dictionary block has been read from disk.
TextIndexReadGranulesMicroseconds
Total time spent reading and analyzing granules of the text index.
TextIndexReadPostings
Number of times a posting list has been read from the text index.
TextIndexReadSparseIndexBlocks
Number of times a sparse index block has been read from the text index.
TextIndexReaderTotalMicroseconds
Total time spent reading the text index.
TextIndexTokensCacheHits
Number of times a text index token info has been found in the cache.
TextIndexTokensCacheMisses
Number of times a text index token info has not been found in the cache.
TextIndexUseHint
Number of index granules where a direct reading from the text index was added as hint and was used.
TextIndexUsedEmbeddedPostings
Number of times a posting list embedded in the dictionary has been used.
ThreadPoolReaderPageCacheHit
Number of times the read inside ThreadPoolReader was done from the page cache.
ThreadPoolReaderPageCacheHitBytes
Number of bytes read inside ThreadPoolReader when it was done from the page cache.
ThreadPoolReaderPageCacheHitElapsedMicroseconds
Time spent reading data from page cache in ThreadPoolReader.
ThreadPoolReaderPageCacheMiss
Number of times the read inside ThreadPoolReader was not done from page cache and was hand off to thread pool.
ThreadPoolReaderPageCacheMissBytes
Number of bytes read inside ThreadPoolReader when read was not done from page cache and was hand off to thread pool.
ThreadPoolReaderPageCacheMissElapsedMicroseconds
Time spent reading data inside the asynchronous job in ThreadPoolReader - when read was not done from the page cache.
ThreadpoolReaderPrepareMicroseconds
Time spent on preparation (e.g. call to reader seek() method)
ThreadpoolReaderReadBytes
Bytes read from a threadpool task in asynchronous reading
ThreadpoolReaderSubmit
Bytes read from a threadpool task in asynchronous reading
ThreadpoolReaderSubmitLookupInCacheMicroseconds
How much time we spent checking if content is cached
ThreadpoolReaderSubmitReadSynchronously
How many times we haven't scheduled a task on the thread pool and read synchronously instead
ThreadpoolReaderSubmitReadSynchronouslyBytes
How many bytes were read synchronously
ThreadpoolReaderSubmitReadSynchronouslyMicroseconds
How much time we spent reading synchronously
ThreadpoolReaderTaskMicroseconds
Time spent getting the data in asynchronous reading
ThrottlerSleepMicroseconds
Total time a query was sleeping to conform all throttling settings.
TinyS3Clients
Number of S3 clients copies which reuse an existing auth provider from another client.
USearchAddComputedDistances
Number of times distance was computed when adding vectors to usearch indexes.
USearchAddCount
Number of vectors added to usearch indexes.
USearchAddVisitedMembers
Number of nodes visited when adding vectors to usearch indexes.
USearchSearchComputedDistances
Number of times distance was computed when searching usearch indexes.
USearchSearchCount
Number of search operations performed in usearch indexes.
USearchSearchVisitedMembers
Number of nodes visited when searching in usearch indexes.
UncompressedCacheHits
Number of times a block of data has been found in the uncompressed cache (and decompression was avoided).
UncompressedCacheMisses
Number of times a block of data has not been found in the uncompressed cache (and required decompression).
UncompressedCacheWeightLost
Number of bytes evicted from the uncompressed cache.
UserTimeMicroseconds
Total time spent in processing (queries and other tasks) threads executing CPU instructions in user mode. This includes time CPU pipeline was stalled due to main memory access, cache misses, branch mispredictions, hyper-threading, etc.
VectorSimilarityIndexCacheHits
Number of times an index granule has been found in the vector index cache.
VectorSimilarityIndexCacheMisses
Number of times an index granule has not been found in the vector index cache and had to be read from disk.
VectorSimilarityIndexCacheWeightLost
Approximate number of bytes evicted from the vector index cache.
VersionedCollapsingSortedMilliseconds
Total time spent while version collapsing sorted columns
WaitMarksLoadMicroseconds
Time spent loading marks
WaitPrefetchTaskMicroseconds
Time spend waiting for prefetched reader
WasmDeserializationMicroseconds
Time spent executing WebAssembly code
WasmGuestExecuteMicroseconds
Time spent executing WebAssembly code
WasmMemoryAllocated
Total memory allocated for WebAssembly compartments
WasmModuleInstatiate
Number of WebAssembly compartments created
WasmSerializationMicroseconds
Time spent executing WebAssembly code
WasmTotalExecuteMicroseconds
Time spent executing WebAssembly code
WriteBufferFromFileDescriptorWrite
Number of writes (write/pwrite) to a file descriptor. Does not include sockets.
WriteBufferFromFileDescriptorWriteBytes
Number of bytes written to file descriptors. If the file is compressed, this will show compressed data size.
WriteBufferFromFileDescriptorWriteFailed
Number of times the write (write/pwrite) to a file descriptor have failed.
WriteBufferFromHTTPBytes
Total size of payload bytes received and sent by WriteBufferFromHTTP. Doesn't include HTTP headers.
WriteBufferFromHTTPRequestsSent
Number of HTTP requests sent by WriteBufferFromHTTP
WriteBufferFromS3Bytes
Bytes written to S3.
WriteBufferFromS3Microseconds
Time spent on writing to S3.
WriteBufferFromS3RequestsErrors
Number of exceptions while writing to S3.
WriteBufferFromS3WaitInflightLimitMicroseconds
Time spent on waiting while some of the current requests are done when its number reached the limit defined by s3_max_inflight_parts_for_one_file.
ZooKeeperBytesReceived
Number of bytes received over network while communicating with ZooKeeper.
ZooKeeperBytesSent
Number of bytes send over network while communicating with ZooKeeper.
ZooKeeperCheck
Number of 'check' requests to ZooKeeper. Usually they don't make sense in isolation, only as part of a complex transaction.
ZooKeeperClose
Number of times connection with ZooKeeper has been closed voluntary.
ZooKeeperCreate
Number of 'create' requests to ZooKeeper.
ZooKeeperExists
Number of 'exists' requests to ZooKeeper.
ZooKeeperGet
Number of 'get' requests to ZooKeeper.
ZooKeeperGetACL
Number of 'getACL' requests to ZooKeeper.
ZooKeeperHardwareExceptions
Number of exceptions while working with ZooKeeper related to network (connection loss or similar).
ZooKeeperInit
Number of times connection with ZooKeeper has been established.
ZooKeeperList
Number of 'list' (getChildren) requests to ZooKeeper.
ZooKeeperMulti
Number of 'multi' requests to ZooKeeper (compound transactions).
ZooKeeperMultiRead
Number of read 'multi' requests to ZooKeeper (compound transactions).
ZooKeeperMultiWrite
Number of write 'multi' requests to ZooKeeper (compound transactions).
ZooKeeperOtherExceptions
Number of exceptions while working with ZooKeeper other than ZooKeeperUserExceptions and ZooKeeperHardwareExceptions.
ZooKeeperReconfig
Number of 'reconfig' requests to ZooKeeper.
ZooKeeperRemove
Number of 'remove' requests to ZooKeeper.
ZooKeeperSet
Number of 'set' requests to ZooKeeper.
ZooKeeperSync
Number of 'sync' requests to ZooKeeper. These requests are rarely needed or usable.
ZooKeeperTransactions
Number of ZooKeeper operations, which include both read and write operations as well as multi-transactions.
ZooKeeperUserExceptions
Number of exceptions while working with ZooKeeper related to the data (no node, bad version or similar).
ZooKeeperWaitMicroseconds
Number of microseconds spent waiting for responses from ZooKeeper after creating a request, summed across all the requesting threads.
ZooKeeperWatchResponse
Number of times watch notification has been received from ZooKeeper.
See Also
- system.asynchronous_metrics — Contains periodically calculated metrics.
- system.metrics — Contains instantly calculated metrics.
- system.metric_log — Contains a history of metrics values from tables
system.metricsandsystem.events. - Monitoring — Base concepts of ClickHouse monitoring.