ClickHouse monitoring should answer whether the analytical service can ingest fresh data, merge it, answer important queries, replicate it, and retain enough capacity to recover from spikes.
For production teams, process uptime alone is not enough. A ClickHouse server can be reachable while dashboards are stale, background merges are falling behind, replicas are unhealthy, or important queries are exhausting memory.
This guide owns ClickHouse monitoring and alert design. For physical query design, use ClickHouse Query Performance: ORDER BY, Keys & Memory Trade-Offs. For backup and HA, use ClickHouse Backup & Replication.
Monitor four layers instead of one dashboard
A useful model separates:
| Layer | Main question | Example signals |
|---|---|---|
| Query service | Can users get results within objectives? | latency, errors, memory, bytes/rows read |
| Ingestion | Is new data arriving and becoming queryable? | insert failures, freshness delay, rejected inserts |
| Background work | Can ClickHouse keep parts healthy? | active parts, merge backlog, mutations |
| Replication/storage | Can data survive and remain synchronized? | replica queues, lag, disk capacity, Keeper health |
The alerting rule is: page on service impact or imminent data risk; keep diagnostic metrics on dashboards unless they imply an actionable condition.
Query monitoring should focus on important workloads
ClickHouse exposes detailed query history through system tables such as system.query_log when configured.
For customer-facing or operational queries, track:
- p50/p95/p99 duration;
- failed queries;
- rows and bytes read;
- memory usage;
- result size;
- query frequency;
- exceptions;
- user/workload identity.
A query that becomes twice as expensive but is run once a week may be less urgent than a dashboard query whose p95 rises by 200 ms across thousands of executions.
Group queries by normalized pattern, application endpoint, dashboard, or workload instead of alerting on every raw SQL string.
Rows and bytes read explain inefficient query paths
Latency alone tells you a query is slow. Rows and bytes read help explain why.
A regression pattern can look like:
same result size + same traffic + rows read increase 20x → likely pruning / ORDER BY / filter regression
Correlate expensive queries with schema and deployment changes.
If a query scans much more data than expected, inspect the sorting key and filters before increasing compute.
Memory needs workload-level alerting
Large GROUP BY operations, joins, sorts, and broad scans can consume substantial memory.
Monitor:
- per-query memory;
- server memory pressure;
- out-of-memory exceptions;
- concurrent expensive queries;
- spill/external-processing behavior where configured;
- workload changes after deployments.
Avoid a universal “memory above 80%” pager. A short analytical burst can be normal. Page when sustained memory pressure is causing failed queries, OOM risk, or service degradation.
Parts and merges reveal ingestion health
MergeTree inserts create parts that ClickHouse merges in the background.
Too many small inserts can create excessive parts and merge pressure.
Monitor:
- number of active parts;
- part creation rate;
- ongoing merges;
- merge queue/backlog;
- merge duration;
- disk bandwidth during merges;
- rejected inserts related to part pressure.
A common chain is:
many tiny inserts → too many parts → merge backlog grows → disk/CPU pressure → queries slow → inserts may eventually be rejected
The fix is often better batching or ingestion design, not only larger servers.
Mutations need separate visibility
ALTER mutations and other background data changes can consume significant resources and remain in progress for long periods.
Track:
- pending mutations;
- age of oldest mutation;
- failures;
- parts remaining;
- query/merge pressure while mutations run.
Treat large production mutations as change events. Record when they started so a later CPU or disk spike has context.
Data freshness is a business-level metric
A ClickHouse cluster can be healthy while the ingestion pipeline is stale.
Define a freshness signal such as:
current time - newest expected event time
or compare source and ClickHouse ingestion checkpoints.
Alert when freshness crosses the application's actual objective.
For a customer analytics dashboard, 15 minutes of stale data may be an incident even if every ClickHouse process and replica is green.
This is why infrastructure monitoring must be paired with pipeline outcomes.
Replica monitoring should prove failover readiness
For replicated tables, monitor:
- replica delay;
- replication queue size;
- read-only state;
- parts to fetch;
- failed fetches;
- Keeper connectivity;
- replica exceptions;
- schema consistency.
A replica is not useful for recovery merely because it exists. Define a health condition that means it is current enough to receive traffic.
Use ClickHouse Backup & Replication for the recovery architecture behind these signals.
Disk monitoring needs growth and headroom, not only percentage used
ClickHouse continuously creates parts, merges them, performs mutations, builds backups, and may retain historical data through TTL policies.
Monitor:
- current free space;
- daily/weekly growth rate;
- bytes by table/partition;
- backup storage consumption;
- temporary/merge space pressure;
- TTL deletion progress.
Forecast time-to-full:
free bytes / average daily net growth = approximate days of headroom
Do not wait for a static 90% disk alert if growth shows the cluster will run out in two days.
Keeper health belongs in replicated-cluster monitoring
Self-managed replicated ClickHouse depends on ClickHouse Keeper for coordination.
Monitor:
- quorum/member availability;
- request latency;
- session/connectivity errors;
- storage health;
- leadership changes;
- resource pressure.
A healthy set of data replicas with an unhealthy coordination layer can still create write and metadata problems.
Keeper should have its own runbook and owner.
Alert severity should map to action
Use three broad classes:
| Class | Example |
|---|---|
| Page | important queries failing; disk exhaustion imminent; offline/unusable replicas threaten availability; ingestion freshness outside critical objective |
| Investigate/ticket | sustained merge backlog; rising query cost; capacity headroom shrinking; replica lag without immediate availability impact |
| Dashboard/context | CPU, network throughput, query volume, part counts within baseline |
Every page should name:
- condition;
- duration;
- user/data impact;
- owner;
- first diagnostic step.
An alert that says only “CPU > 80%” transfers diagnosis to the person being paged.
Baselines are better than borrowed universal thresholds
Collect normal ranges for:
- insert rate;
- query latency;
- rows/bytes read;
- parts per active partition;
- merge throughput;
- memory;
- replica delay;
- disk growth;
- data freshness.
Then alert on sustained deviations tied to impact.
A nightly ETL workload may legitimately use most CPU for 20 minutes. The same load at midday after a deployment may be abnormal. Context matters.
Managed ClickHouse reduces platform monitoring work, not application observability
Raff Managed ClickHouse provides managed database infrastructure and monitoring/query visibility so teams do not need to operate every host-level component themselves.
The data/application team still needs to define:
- which queries matter;
- acceptable dashboard latency;
- freshness objective;
- ingestion correctness;
- materialized-view validity;
- business-level recovery criteria.
At Raff, the monitoring rule is: alert on a broken analytical outcome first, then correlate queries, parts, merges, replicas, memory, and storage to explain it.
ClickHouse monitoring checklist
- Important query patterns have latency/error objectives.
- Rows/bytes read are visible for query regressions.
- Per-query and server memory pressure are monitored.
- Ingestion freshness has an application-level metric.
- Active parts and merge backlog are monitored.
- Long mutations are visible.
- Replica lag/queues/read-only state are monitored.
- Keeper health is monitored for self-managed HA.
- Disk growth is forecast, not only thresholded.
- Alerts distinguish page vs investigation vs dashboard context.
- Every page has an owner and first response step.
Conclusion
ClickHouse monitoring should prove that data is fresh, queries are usable, background work is keeping up, replicas are recoverable, and storage has enough headroom.
Start from customer or pipeline outcomes, then use ClickHouse-specific query, merge, part, replica, and Keeper signals to diagnose the cause. This produces fewer alerts and faster incident response than treating every host metric as equally important.