Prometheus protocols and PromQL

Expose ClickHouse server metrics

Configure a dedicated port when a Prometheus server needs to scrape ClickHouse’s own metrics:

<prometheus>
    <port>9363</port>
    <endpoint>/metrics</endpoint>
    <metrics>true</metrics>
    <asynchronous_metrics>true</asynchronous_metrics>
    <events>true</events>
    <errors>true</errors>
    <histograms>true</histograms>
    <dimensional_metrics>true</dimensional_metrics>
</prometheus>

Section <prometheus.handlers> can be used to make more extended handlers on the same port. This section is similar to <http_handlers> but works for prometheus protocols:

<prometheus>
    <port>9363</port>
    <handlers>
        <my_rule_1>
            <url>/metrics</url>
            <handler>
                <type>expose_metrics</type>
                <metrics>true</metrics>
                <asynchronous_metrics>true</asynchronous_metrics>
                <events>true</events>
                <errors>true</errors>
                <histograms>true</histograms>
                <dimensional_metrics>true</dimensional_metrics>
                <labels>
                    <environment>production</environment>
                    <shard from_env="SHARD_NAME"></shard>
                </labels>
            </handler>
        </my_rule_1>
    </handlers>
</prometheus>

Settings:

Check the endpoint:

curl http://127.0.0.1:9363/metrics

Prometheus HTTP API and PromQL

ClickHouse implements the Prometheus HTTP API over a TimeSeries table. One handler serves remote write, remote read, instant PromQL queries, and range PromQL queries.

Prerequisites

Enable the allow_experimental_time_series_table setting for the user that creates and accesses the table:

SET allow_experimental_time_series_table = 1;

Create a database and a TimeSeries table:

CREATE DATABASE prometheus;
CREATE TABLE prometheus.metrics ENGINE = TimeSeries;

For HTTP API requests, enable allow_experimental_time_series_table in the profile of the API user.

Configure the Prometheus API

Configure one prefix-routed handler on the main ClickHouse HTTP port:

<http_handlers>
    <defaults/>
    <rule>
        <url_prefix>/prometheus/api/v1</url_prefix>
        <handler>
            <type>prometheus_api_v1</type>
        </handler>
    </rule>
</http_handlers>

<defaults/> preserves the built-in handlers for endpoints such as /ping and for SQL requests. The prefix above exposes these endpoints through one handler:

The example omits database and table from the handler. Each request must provide the table query parameter (except for /format_query, which only parses the given PromQL expression and doesn’t need a table). It can also provide database, use a qualified table name such as prometheus.metrics, or omit the database to use default. This allows one handler to serve multiple TimeSeries tables.

To use one fixed table for every request, configure it in the handler:

<handler>
    <type>prometheus_api_v1</type>
    <database>prometheus</database>
    <table>metrics</table>
</handler>

A table configured in the handler cannot be overridden by request parameters.

Routing and handler settings:

Ingest metrics with remote write

ClickHouse supports the Prometheus remote-write protocol. Configure Prometheus to write to the handler:

remote_write:
  - url: https://clickhouse.example.com:8443/prometheus/api/v1/write?database=prometheus&table=metrics
    basic_auth:
      username: default
      password: <password>

Prometheus sends samples to the prometheus.metrics table.

To batch data from many concurrent remote-write requests into fewer parts, enable asynchronous inserts by adding the async_insert setting to the URL (or by enabling it in the user profile):

remote_write:
  - url: https://clickhouse.example.com:8443/prometheus/api/v1/write?database=prometheus&table=metrics&async_insert=1

ClickHouse acknowledges an asynchronous remote-write request only after the data is flushed to all inner tables of the TimeSeries table, regardless of the wait_for_async_insert setting: the remote-write protocol treats an acknowledged write as durable. If the flush fails, the request returns an error and Prometheus retries it.

Query with PromQL

Use the instant-query endpoint to evaluate a PromQL expression at one point in time:

curl --user default:<password> --get \
  "https://clickhouse.example.com:8443/prometheus/api/v1/query" \
  --data-urlencode "query=rate(http_requests_total[5m])" \
  --data-urlencode "database=prometheus" \
  --data-urlencode "table=metrics"

Use the range-query endpoint to evaluate an expression over a time range:

curl --user default:<password> --get \
  "https://clickhouse.example.com:8443/prometheus/api/v1/query_range" \
  --data-urlencode "query=rate(http_requests_total[5m])" \
  --data-urlencode "start=2026-08-15T12:00:00Z" \
  --data-urlencode "end=2026-08-15T13:00:00Z" \
  --data-urlencode "step=60s" \
  --data-urlencode "database=prometheus" \
  --data-urlencode "table=metrics"

Use the format-query endpoint to parse and format a PromQL expression without evaluating it:

curl --user default:<password> --get \
  "https://clickhouse.example.com:8443/prometheus/api/v1/format_query" \
  --data-urlencode "query=sum by(job)(http_requests_total{code=\"200\"})/2"

The expression is returned serialized from the parsed query, with the whitespace normalized, the comments removed, the redundant parentheses dropped, and the durations converted to numbers of seconds: sum by (job) (http_requests_total{code="200"}) / 2. This endpoint doesn’t evaluate the expression, so it doesn’t need the database and table parameters.

See the supported PromQL features for the function and aggregation operator list used by the HTTP API, the promql dialect, and the table functions.

Grafana

Configure a Prometheus data source with the base URL ending before /api/v1:

apiVersion: 1
datasources:
  - name: ClickHouse Prometheus
    type: prometheus
    access: proxy
    url: https://clickhouse.example.com:8443/prometheus
    basicAuth: true
    basicAuthUser: default
    jsonData:
      httpMethod: GET
      customQueryParameters: database=prometheus&table=metrics
    secureJsonData:
      basicAuthPassword: <password>

Grafana appends /api/v1/query or /api/v1/query_range to this base URL and adds customQueryParameters to each request.

SQL entry points

ClickHouse uses the same PromQL converter for the HTTP API, the promql dialect, and the prometheusQuery and prometheusQueryRange table functions.

Run PromQL directly with clickhouse-client:

clickhouse-client \
  --dialect promql \
  --promql_database prometheus \
  --promql_table metrics \
  --query 'rate(http_requests_total[5m])'

Use the table functions to embed PromQL in a SQL query:

SELECT *
FROM prometheusQuery(
    prometheus.metrics,
    'rate(http_requests_total[5m])',
    now()
);

Query metric metadata

The /prometheus/api/v1/metadata endpoint returns the metric metadata stored in the Metrics target table of the TimeSeries table: the type, help text, and unit of each metric family. It supports the following Prometheus parameters in the URL query string:

The default Metrics target table is a ReplacingMergeTree ordered by the metric family name: it keeps the most recently written metadata entry for each metric family. Several entries per family are returned only while the target table stores them — before its parts are merged, or when the table is defined with an engine that preserves them.

curl --user default:<password> --get \
  "https://clickhouse.example.com:8443/prometheus/api/v1/metadata" \
  --data-urlencode "metric=http_requests_total" \
  --data-urlencode "database=prometheus" \
  --data-urlencode "table=metrics"

Read metrics with remote read

ClickHouse supports the Prometheus remote-read protocol at /prometheus/api/v1/read.

Configure a Prometheus server to read from the same TimeSeries table:

remote_read:
  - url: https://clickhouse.example.com:8443/prometheus/api/v1/read?database=prometheus&table=metrics
    basic_auth:
      username: default
      password: <password>