Kubernetes monitoring is the collection and interpretation of cluster, workload, and application signals so teams can detect failures, understand behavior, and decide what to do next.
For small teams, the challenge is not collecting the largest possible telemetry set. It is knowing which signal answers which operational question. Metrics show that behavior changed. Kubernetes Events show what the control plane or another component reported while trying to reconcile state. Logs show what an application or system component emitted. Traces connect latency and work across an instrumented request path.
This guide owns monitoring and observability inside the Kubernetes Cluster Management for Small Teams operating model. It focuses on signal selection, correlation, alert design, and the boundary between platform visibility and application observability rather than on installing Prometheus, Grafana, Loki, or another monitoring stack.
At Raff, we start incidents with the narrowest signal that proves user impact, then move down the stack. A dashboard full of healthy infrastructure metrics is not evidence that an API request succeeded.
Kubernetes monitoring and observability are related but not identical
Monitoring usually asks whether known conditions are healthy: CPU pressure, unavailable replicas, restart counts, failed requests, queue depth, or latency thresholds.
Observability is broader. Kubernetes describes observability through the collection and analysis of metrics, logs, and traces to understand the internal state, performance, and health of the cluster. Those signals help operators investigate conditions they did not fully predict in advance.
A practical small-team model adds Kubernetes Events as a fourth diagnostic signal alongside the three primary observability pillars:
| Signal | Best question | Typical example |
|---|---|---|
| Metrics | What changed, by how much, and for how long? | CPU, memory, request rate, latency, error rate, replica availability |
| Events | What did Kubernetes report while reconciling the object? | Failed scheduling, image pull failure, eviction, volume mount issue |
| Logs | What did this component or application say happened? | Stack trace, timeout, authentication failure, controller message |
| Traces | Where did one instrumented request spend time? | API → worker → database call with span timing |
Kubernetes Events are deliberately separated from durable telemetry because the API documents them as limited-retention, best-effort, supplemental data. They are excellent incident clues, but they should not be treated as a permanent monitoring database.
Metrics tell you that behavior changed; Events tell you what Kubernetes tried to do; logs show what a component said; traces show how one request moved through instrumented systems.
Official references: Kubernetes Observability and Kubernetes Event API.
Metrics should separate resource health from service health
Kubernetes exposes several kinds of metrics, and the basic resource pipeline is only one of them.
The resource metrics pipeline uses Metrics Server to collect short-term CPU and memory usage from nodes and Pods and expose it through the metrics.k8s.io API. That data supports tools such as kubectl top and resource-based autoscaling. Kubernetes documentation explicitly describes this as a limited set of resource metrics.
That means a CPU graph is useful, but it cannot answer every production question.
A small-team monitoring model should separate at least three metric layers:
| Layer | Useful metrics | Operational question |
|---|---|---|
| Infrastructure | Node CPU, memory, disk pressure, node availability | Is worker capacity healthy? |
| Kubernetes workload | Ready replicas, restarts, Pending Pods, job failures | Is Kubernetes keeping the desired workload state? |
| Application/service | Request rate, error rate, latency, queue depth, business failures | Are users receiving the intended service? |
The third layer is the one most likely to be missed. A Deployment can have all replicas Ready while the application returns HTTP 500 responses. A node can have low CPU while a downstream database dependency is timing out. Kubernetes resource metrics do not know whether a checkout completed successfully.
Use resource metrics for capacity and workload behavior, then add application indicators that reflect actual service outcomes.
The richer monitoring pipeline can store time-series data and expose custom or external metrics. Prometheus is a common option, but this guide deliberately does not make one tool mandatory. The requirement is historical, queryable evidence for the signals your team needs to operate.
For the specific relationship between requests and CPU-based HPA signals, use Kubernetes Requests vs Limits and Kubernetes Autoscaling.
Logs need a lifecycle beyond the Pod that produced them
Kubernetes makes container output available through kubectl logs, but that should not be confused with durable cluster-level log storage.
The Kubernetes logging architecture expects applications to commonly write to stdout and stderr. The kubelet and container runtime make those streams accessible, while node-level agents or other logging architectures can forward data to a separate backend.
Kubernetes does not provide a native cluster-level log storage system. Its own documentation recommends that important logs have a storage and lifecycle independent of the node, Pod, or container that produced them.
That matters because Pods are disposable. During an incident, the failed replica may already have restarted or moved by the time someone investigates.
A useful logging policy answers:
- Which application and system logs are necessary for production diagnosis?
- Are logs structured enough to filter by service, environment, severity, and request identifier?
- Can the team find logs from a terminated or replaced Pod?
- How long are logs retained?
- Are secrets, tokens, passwords, or sensitive payloads excluded or redacted?
- Is there a way to correlate one log line with a deployment, Pod, or request?
Avoid treating log volume as observability maturity. High-volume debug logs can make important signals harder to find and increase storage cost.
A small production service usually benefits more from consistent structured fields and useful error context than from capturing every possible message forever.
Official reference: Kubernetes Logging Architecture.
Kubernetes Events explain reconciliation failures but are temporary
Events are especially useful when the failure is about what Kubernetes is trying to do rather than what the application code is doing.
Examples include:
- a Pod cannot be scheduled;
- an image cannot be pulled;
- a volume cannot be mounted;
- a container repeatedly fails startup;
- a node experiences pressure or eviction;
- a workload is waiting on a cluster-level resource.
This makes Events valuable for questions such as:
Why is this Pod Pending? Why did this container not start? Why did Kubernetes evict this workload? Why did this volume fail to attach?
They are less suitable for questions such as:
What was our p95 API latency last week? How many users received errors yesterday? Which request spent 800 ms in a downstream service?
The Event API explicitly warns that Events have limited retention, that reasons and messages can evolve, and that consumers should treat them as informative, best-effort supplemental data.
A Kubernetes Event is diagnostic evidence, not a durable monitoring database.
During an incident, capture relevant Events while they still exist, but rely on dedicated metrics, logs, traces, or audit data for historical analysis where the requirement demands durable records.
This boundary also prevents a common troubleshooting mistake. If a Pod is Pending, Events can quickly identify a scheduling or mount reason. If the Pod is Running and Ready but users receive errors, application metrics and logs are usually the stronger next signal.
Traces answer request-path questions that metrics and logs cannot
Distributed traces connect operations into one request path by propagating context across instrumented components.
A trace can show a request like:
Client request ↓ 18 ms API service ↓ 42 ms Internal service ↓ 620 ms Database call ↓ Response
That makes traces useful when aggregate metrics show a latency increase but do not explain where the time was spent.
Kubernetes now documents tracing for system components through OpenTelemetry Protocol (OTLP), and Kubernetes observability guidance also describes application spans flowing through an OpenTelemetry Collector or another compatible pipeline. Application tracing still requires the application and its dependencies to emit and propagate useful trace context; Kubernetes does not automatically create meaningful business-request traces for code that has not been instrumented.
Tracing is most valuable when:
- a request crosses multiple services;
- latency is distributed across dependencies;
- intermittent failures are difficult to reconstruct from aggregate metrics;
- the team needs to understand which downstream call contributed to a user-visible delay.
Tracing is less useful as the first telemetry project for a simple single-service application that still lacks basic error-rate and latency metrics.
Use the simplest signal that answers the operational question. Add trace depth when the architecture creates request paths worth tracing.
Official references: Kubernetes Observability and Traces for Kubernetes System Components.
The decision framework correlates signals instead of opening every dashboard
When something breaks, the team should not have to inspect every telemetry source in random order.
Use the symptom to choose the first signal, then correlate downward.
| Symptom | Start with | Correlate next |
|---|---|---|
| User error rate rises | Application/service metrics | Logs, then trace for affected requests |
| Latency rises across several services | Service metrics | Traces, dependency metrics, logs |
| Pod stays Pending | Kubernetes Events | Scheduler constraints, requests, node capacity |
| Pod restarts repeatedly | Restart/workload metrics | Events, previous container logs, memory/CPU evidence |
| Node pressure increases | Node/resource metrics | Workload requests, evictions, Events, system logs |
| Deployment caused regression | Service metric change around deploy time | Pod logs, workload state, traces |
| One dependency appears slow | Application metrics | Trace spans, dependency logs/metrics |
| Cluster looks healthy but users report failure | User/service indicators | Application logs and traces before infrastructure tuning |
A useful incident flow is:
Prove user or workload impact ↓ Identify the affected scope ↓ Check Kubernetes state and Events ↓ Inspect relevant logs ↓ Use traces when the request path is distributed ↓ Confirm recovery with the original impact signal
This prevents infrastructure dashboards from becoming the default answer to every application incident.
At Raff, the operating rule is that the recovery signal should match the failure signal. If an alert began because checkout errors crossed a threshold, successful node CPU graphs do not close the incident; the checkout error signal needs to recover.
Alerts should represent action, not telemetry volume
A monitoring system is useful when it changes an operational decision.
Small teams often create alert fatigue by converting every available metric into a notification. CPU above a threshold, one Pod restart, one failed job, one transient scheduling delay, and every warning log can quickly create more noise than signal.
Prioritize alerts in this order:
- User-visible service symptoms — sustained errors, latency, availability, failed critical transactions.
- Workload conditions that threaten service — unavailable replicas, crash loops, persistent Pending Pods, failed critical jobs.
- Capacity conditions with a clear action — sustained memory pressure, disk pressure, worker exhaustion, autoscaling ceiling reached.
- Platform warnings that require maintenance — certificate, storage, controller, or node conditions that will become incidents if ignored.
Use dashboards for investigation and trends. Use alerts for conditions that need a human or automated response within a defined time.
Every alert should have:
- an owner;
- a severity;
- a reason the condition matters;
- a first diagnostic action;
- a recovery condition;
- a way to avoid repeated notifications for the same unresolved condition.
If a signal does not change an operational decision, collecting more of it is not automatically better observability.
Retention and cardinality also need boundaries. Production may justify longer history than preview environments. High-cardinality labels and unbounded logs can make observability expensive and difficult to query. The broader spending policy remains in Kubernetes Cost Optimization.