Kafka Metrics: Broker Health, Replication & Alerting

Kafka metrics are runtime measurements that show whether brokers, replicas, producers, and consumers are healthy enough to keep an event stream available and current. The useful question is not how many metrics Kafka exposes. It is which signals tell you that a production outcome is at risk.

For small teams, monitoring works best when it separates infrastructure health from application progress. Kafka Architecture for Small Teams owns the topic, partition, and ownership model; this guide owns metric selection, broker and replication health, signal correlation, and alert design.

Raff's Kafka metrics model uses 2 layers: broker/replica health and application stream progress. At Raff, we treat consumer lag as a recovery-budget signal rather than a standalone instruction to add workers. A broker can be healthy while an application falls behind, and a lag spike can be harmless while replicas are temporarily moving.

Kafka metrics need infrastructure and stream context

A useful Kafka monitoring model answers two different questions at the same time:

1. Can the cluster serve the stream safely? This is the broker and replica layer: partition availability, replication state, request handling, network processing, storage health, and controller state.
2. Is the application keeping up with the stream? This is the producer and consumer layer: produce errors, fetch progress, consumer lag, event age, throttling, and application processing failures.

Apache Kafka 4.3 exposes broker, controller, replica, producer, consumer, and coordinator metrics through its monitoring interfaces, including JMX. The official monitoring reference lists metrics such as UnderReplicatedPartitions, UnderMinIsrPartitionCount, OfflinePartitionsCount, request-handler idle ratio, network-processor idle ratio, client request timings, and consumer lag.

The important design choice is to avoid turning that catalog into one undifferentiated dashboard. Some metrics describe a failure state. Others describe pressure. Others are useful only when correlated with an application symptom.

The operating rule is simple: monitor symptoms first, then use supporting metrics to explain the symptom. A dashboard is diagnostic context. An alert should represent a condition somebody can act on.

Broker metrics expose request and resource pressure

Broker metrics are most useful when they show either loss of service or sustained pressure that can become loss of service.

Apache Kafka's 4.3 monitoring reference identifies OfflinePartitionsCount as the number of non-internal topic partitions that are offline. That is materially different from a performance warning: an offline partition means some topic data cannot currently be served through an active leader.

Request handling also deserves attention. Kafka exposes RequestHandlerAvgIdlePercent and NetworkProcessorAvgIdlePercent as ratios between 0 and 1. Apache's documentation describes values above roughly 0.3 as desirable for those idle ratios. Treat that number as a reference from the Kafka project, not as a universal pager threshold for every workload.

A sustained fall in idle capacity becomes more meaningful when request latency, queue time, client timeouts, or throughput changes at the same time. One low idle sample during a burst is weaker evidence than a trend where multiple related signals move together.

Monitor broker health in groups:

• Request path: request rate, request latency, queue time, handler idle ratio.
• Network path: network processor idle ratio, connection behavior, bytes in/out.
• Storage path: disk utilization, log-directory status, partition size, I/O service time.
• Controller path: offline partition count, metadata processing errors, controller health.

Do not make raw CPU or memory percentage the only broker alarm. Resource utilization explains pressure, but Kafka-specific service signals show whether that pressure is affecting the broker's work.

A broker at high CPU with stable request latency and healthy replicas is a different incident from a broker at moderate CPU with offline partitions or failed storage. Start with service impact, then inspect host resources.

Replication metrics show whether partition redundancy is degrading

Replication metrics reveal whether Kafka still has the replica state required to tolerate failures and satisfy the configured durability model.

Apache Kafka documents UnderReplicatedPartitions as the count of partitions whose in-sync replica set is smaller than the full replica set, with an expected steady-state value of zero. It separately exposes UnderMinIsrPartitionCount for partitions whose in-sync replica count has fallen below min.insync.replicas.

Those two conditions should not be collapsed into one generic "replication warning."

The Kafka Partitions guide owns replication-factor and partition-design choices. This guide uses those decisions as monitoring context rather than re-explaining them.

Alerting should also account for planned operations. A brief under-replicated state during controlled partition reassignment is not equivalent to an unexplained, persistent loss of replica health. The metric is the signal; duration, scope, and change context determine urgency.

A stronger alert condition therefore combines at least two dimensions: severity of the state and persistence of the state. Add application impact when available.

Producer and consumer signals connect cluster health to application impact

Kafka can remain technically available while the product built on it becomes stale. Producer and consumer metrics close that gap.

For producers, watch error rates, retries, request latency, throughput, and throttling. A producer that is timing out or being throttled can reduce event freshness even when every partition is online.

For consumers, lag is important but should not stand alone. Apache Kafka's monitoring guidance recommends watching maximum consumer lag and fetch request rate. In application operations, event age can be even easier to map to business impact because it answers how old the data being processed actually is.

The detailed lag, rebalance, offset, and recovery model belongs in Kafka Consumer Groups: Lag, Rebalances & Recovery Planning. Keep the monitoring layer narrower:

• Is lag growing, stable, or draining?
• Is the problem isolated to one group or broad across groups?
• Did broker or replication health change at the same time?
• Did producer rate, consumer fetch rate, or throttling change?
• Is event age crossing the application's freshness objective?
• Did a deployment or planned operation explain the movement?

A useful correlation pattern is:

application symptom
-> consumer or producer progress
-> broker request health
-> replica / partition health
-> host and storage context

This ordering prevents a common failure mode: starting with dozens of infrastructure graphs before confirming which user-facing or data-processing outcome changed.

Consumer lag also should not cannibalize the consumer-group decision page. This guide treats lag as one monitoring signal. The Consumer Groups guide owns why lag grows, how rebalances affect it, offset behavior, catch-up capacity, and recovery planning.

The alert framework separates pages from diagnostic signals

Kafka alerting becomes manageable when every signal is assigned a response class before it fires.

Use three classes:

The exact threshold depends on workload and business tolerance. Avoid universal rules such as "page whenever lag is above 10,000" or "page whenever CPU exceeds 80%." Neither number says whether the stream is current, recoverable, or unavailable.

A better alert definition contains four parts:

1. Condition: the specific Kafka or application state.
2. Duration: how long it must persist before it is considered abnormal.
3. Impact: what availability, freshness, or recovery objective is threatened.
4. Owner: the team or service responsible for the next action.

For example, a useful consumer alert is not simply "lag > X." It is closer to "lag is increasing for the payments group for a sustained period, event age is outside its freshness objective, and catch-up rate is below incoming rate." The exact values come from that workload's baseline and recovery budget.

Similarly, under-replicated partitions should carry context about whether a reassignment or broker maintenance operation is in progress. Offline partitions deserve a higher response class because availability is already affected.

For a broader, engine-neutral alert model, Database Monitoring for Small Teams owns the generic monitoring and escalation framework. This page applies the decision to Kafka-specific signals.

Raff Managed Kafka changes ownership, not observability

A managed Kafka service can reduce the broker operations a small team must perform, but it does not remove the need to observe the application that produces and consumes events.

Raff Managed Kafka is available through the managed Kafka product path. Raff Managed Kafka starts at $79.99/month for 2 vCPU, 4 GB RAM, and 80 GB storage. Product pricing and plan details should still be checked on the live product surface when making a deployment decision.

The ownership boundary is more useful than a feature checklist:

At Raff, the monitoring decision is therefore ownership-first. Broker and replica signals tell you when the managed service layer needs attention; producer, consumer, and business-freshness signals tell you when the application team needs to act. The same incident can cross both boundaries, which is why correlation matters more than a single all-green dashboard.

Use Managed Kafka vs Self-Hosted Kafka when the unresolved decision is whether your team should operate the broker platform itself. Use this guide after that choice to decide what must remain visible either way.

Conclusion

Kafka metrics are useful only when they reduce the time between a real problem and the correct response.

Monitor broker availability, request pressure, replication state, client errors, and stream progress as related layers. Treat offline partitions and persistent replica degradation differently from raw capacity signals. Treat consumer lag as one input to freshness and recovery, not as a universal scaling command.

For Kafka workloads, managed broker operations can reduce infrastructure ownership, but application teams still need clear signals for producer behavior, consumer progress, event freshness, and recovery.

Return to Kafka Architecture for Small Teams for the cluster-wide decision model, then use the Consumer Groups and Partitions guides when monitoring points to a design or recovery problem.

Sources

• Apache Kafka 4.3 Monitoring
• Apache Kafka 4.3 Operations
• Raff Managed Kafka
• Product and source paths reviewed: 2026-08-12.