Kafka metrics are runtime measurements that show whether brokers, replicas, producers, and consumers are healthy enough to keep an event stream available and current. The useful question is not how many metrics Kafka exposes. It is which signals tell you that a production outcome is at risk.
For small teams, monitoring works best when it separates infrastructure health from application progress. See Kafka Architecture for Small Teams for the topic, partition, and operating-model decisions. The monitoring decisions here focus on metric selection, broker and replication health, signal correlation, and alerts.
A practical Kafka monitoring model uses two layers: broker/replica health and application stream progress. Treat consumer lag as a recovery-budget signal rather than a standalone instruction to add workers. A broker can be healthy while an application falls behind, and a lag spike can be harmless while replicas are temporarily moving.
Kafka metrics need infrastructure and stream context
A useful Kafka monitoring model answers two different questions at the same time:
- Can the cluster serve the stream safely? This is the broker and replica layer: partition availability, replication state, request handling, network processing, storage health, and controller state.
- Is the application keeping up with the stream? This is the producer and consumer layer: produce errors, fetch progress, consumer lag, event age, throttling, and application processing failures.
Apache Kafka 4.3 exposes broker, controller, replica, producer, consumer, and coordinator metrics through its monitoring interfaces, including JMX. The official monitoring reference lists metrics such as UnderReplicatedPartitions, UnderMinIsrPartitionCount, OfflinePartitionsCount, request-handler idle ratio, network-processor idle ratio, client request timings, and consumer lag.
The important design choice is to avoid turning that catalog into one undifferentiated dashboard. Some metrics describe a failure state. Others describe pressure. Others are useful only when correlated with an application symptom.
| Signal class | Examples | Main question |
|---|---|---|
| Availability | offline partitions, offline replicas, unavailable leaders | Can Kafka serve the required data? |
| Replication | under-replicated partitions, under-min-ISR partitions, ISR shrink/expand activity | Is redundancy degrading? |
| Broker pressure | request-handler idle ratio, network-processor idle ratio, request latency, queue pressure | Is broker capacity becoming constrained? |
| Stream progress | consumer lag, fetch rate, event age | Is the application keeping up? |
| Client health | produce/fetch errors, retries, throttling, request time | Are clients communicating and processing normally? |
| Storage health | log directory state, disk utilization, partition size | Can the broker continue storing and serving data? |
The operating rule is simple: monitor symptoms first, then use supporting metrics to explain the symptom. A dashboard is diagnostic context. An alert should represent a condition somebody can act on.
Broker metrics expose request and resource pressure
Broker metrics are most useful when they show either loss of service or sustained pressure that can become loss of service.
Apache Kafka's 4.3 monitoring reference identifies OfflinePartitionsCount as the number of non-internal topic partitions that are offline. That is materially different from a performance warning: an offline partition means some topic data cannot currently be served through an active leader.
Request handling also deserves attention. Kafka exposes RequestHandlerAvgIdlePercent and NetworkProcessorAvgIdlePercent as ratios between 0 and 1. Apache's documentation describes values above roughly 0.3 as desirable for those idle ratios. Treat that number as a reference from the Kafka project, not as a universal pager threshold for every workload.
A sustained fall in idle capacity becomes more meaningful when request latency, queue time, client timeouts, or throughput changes at the same time. One low idle sample during a burst is weaker evidence than a trend where multiple related signals move together.
Monitor broker health in groups:
- Request path: request rate, request latency, queue time, handler idle ratio.
- Network path: network processor idle ratio, connection behavior, bytes in/out.
- Storage path: disk utilization, log-directory status, partition size, I/O service time.
- Controller path: offline partition count, metadata processing errors, controller health.
Do not make raw CPU or memory percentage the only broker alarm. Resource utilization explains pressure, but Kafka-specific service signals show whether that pressure is affecting the broker's work.
A broker at high CPU with stable request latency and healthy replicas is a different incident from a broker at moderate CPU with offline partitions or failed storage. Start with service impact, then inspect host resources.
Replication metrics show whether partition redundancy is degrading
Replication metrics reveal whether Kafka still has the replica state required to tolerate failures and satisfy the configured durability model.
Apache Kafka documents UnderReplicatedPartitions as the count of partitions whose in-sync replica set is smaller than the full replica set, with an expected steady-state value of zero. It separately exposes UnderMinIsrPartitionCount for partitions whose in-sync replica count has fallen below min.insync.replicas.
Those two conditions should not be collapsed into one generic "replication warning."
| Replication signal | What it means | Operational interpretation |
|---|---|---|
| Under-replicated partitions | one or more replicas are not in the in-sync set | redundancy is degraded; investigate persistence and duration |
| Under-min-ISR partitions | ISR is below the configured minimum | write availability or durability expectations may be threatened, depending on producer settings |
| Offline partitions | no active leader is serving the partition | direct availability problem |
| ISR shrink activity | replicas are leaving the in-sync set | determine whether this is planned movement or instability |
| ISR expansion activity | replicas are returning to the in-sync set | recovery may be underway |
| Offline log directory | a broker log directory is unavailable | storage failure may affect replicas on that broker |
For partition count and replication-factor decisions, see Kafka Partitions: Keys, Replication & Scaling. Those design choices provide context when interpreting replication alerts.
Alerting should also account for planned operations. A brief under-replicated state during controlled partition reassignment is not equivalent to an unexplained, persistent loss of replica health. The metric is the signal; duration, scope, and change context determine urgency.
A stronger alert condition therefore combines at least two dimensions: severity of the state and persistence of the state. Add application impact when available.
Producer and consumer signals connect cluster health to application impact
Kafka can remain technically available while the product built on it becomes stale. Producer and consumer metrics close that gap.
For producers, watch error rates, retries, request latency, throughput, and throttling. A producer that is timing out or being throttled can reduce event freshness even when every partition is online.
For consumers, lag is important but should not stand alone. Apache Kafka's monitoring guidance recommends watching maximum consumer lag and fetch request rate. In application operations, event age can be even easier to map to business impact because it answers how old the data being processed actually is.
For detailed lag, rebalance, offset, and recovery decisions, see Kafka Consumer Groups: Lag, Rebalances & Recovery Planning. For monitoring, start with these questions:
- Is lag growing, stable, or draining?
- Is the problem isolated to one group or broad across groups?
- Did broker or replication health change at the same time?
- Did producer rate, consumer fetch rate, or throttling change?
- Is event age crossing the application's freshness objective?
- Did a deployment or planned operation explain the movement?
A useful correlation pattern is:
application symptom -> consumer or producer progress -> broker request health -> replica / partition health -> host and storage context
This ordering prevents a common failure mode: starting with dozens of infrastructure graphs before confirming which user-facing or data-processing outcome changed.
Consumer lag is one monitoring signal, not a complete diagnosis. Interpret it alongside event age, producer and consumer rates, rebalance history, offset progress, and downstream dependencies before escalating or adding workers.
The alert framework separates pages from diagnostic signals
Kafka alerting becomes manageable when every signal is assigned a response class before it fires.
Use three classes:
| Response class | Use when | Kafka examples |
|---|---|---|
| Page | service or data availability is at immediate risk and a person must act now | offline partitions; persistent under-min-ISR state tied to write risk; widespread client failures |
| Ticket / investigate | degradation is real but does not require immediate interruption | sustained under-replication without current impact; rising request latency with remaining headroom |
| Dashboard / context | metric explains incidents but is not actionable by itself | raw CPU, bytes in/out, partition counts, short ISR movement during planned work |
The exact threshold depends on workload and business tolerance. Avoid universal rules such as "page whenever lag is above 10,000" or "page whenever CPU exceeds 80%." Neither number says whether the stream is current, recoverable, or unavailable.
A better alert definition contains four parts:
- Condition: the specific Kafka or application state.
- Duration: how long it must persist before it is considered abnormal.
- Impact: what availability, freshness, or recovery objective is threatened.
- Owner: the team or service responsible for the next action.
For example, a useful consumer alert is not simply "lag > X." It is closer to "lag is increasing for the payments group for a sustained period, event age is outside its freshness objective, and catch-up rate is below incoming rate." The exact values come from that workload's baseline and recovery budget.
Similarly, under-replicated partitions should carry context about whether a reassignment or broker maintenance operation is in progress. Offline partitions deserve a higher response class because availability is already affected.
For engine-neutral monitoring and escalation principles, see Database Monitoring for Small Teams. Here, apply those principles to Kafka-specific availability, replication, and consumer-progress signals.
Raff Managed Kafka changes ownership, not observability
A managed Kafka service can reduce the broker operations a small team must perform, but it does not remove the need to observe the application that produces and consumes events.
For current plan sizes, first-month offers, and standard monthly prices, see Raff Managed Kafka. Compare the service's broker-operations scope with the monitoring and recovery work your application team must still perform.
The ownership boundary is more useful than a feature checklist:
| Layer | Managed-platform responsibility | Application-team responsibility |
|---|---|---|
| Broker platform | service lifecycle and underlying broker operations | understand service health and escalation state |
| Topics and partitions | platform provides Kafka capability | choose topic boundaries, keys, partition strategy |
| Replication | platform operates the managed service | understand application durability requirements |
| Producers | none of the application's business behavior is transferred | errors, retries, throughput, request behavior |
| Consumers | none of the consumer's business correctness is transferred | lag, event age, offsets, rebalances, retries, side effects |
| Business freshness | platform cannot define it | set freshness objectives and validate outcomes |
The monitoring decision is therefore ownership-first. Broker and replica signals tell you when the managed service layer needs attention; producer, consumer, and business-freshness signals tell you when the application team needs to act. The same incident can cross both boundaries, which is why correlation matters more than a single all-green dashboard.
Use Managed Kafka vs Self-Hosted Kafka when the unresolved decision is whether your team should operate the broker platform itself. Use this guide after that choice to decide what must remain visible either way.
