Kafka Retention: Durability, Replication & Data-Loss Trade-Offs

Kafka retention determines how long records remain available in a topic before they become eligible for deletion or compaction. It is a storage-lifecycle policy, not a durability guarantee by itself.

For production teams, retention needs to answer three separate questions: how long consumers may need old events, how much storage that history will consume, and whether replication and producer acknowledgements provide the durability the business expects while the data is retained.

This guide owns Kafka retention and durability trade-offs. For partition and replication design, use Kafka Partitions: Keys, Replication & Scaling Trade-Offs. For consumer recovery, use Kafka Consumer Groups: Lag, Rebalances & Recovery Planning.

Kafka retention and replication solve different problems

A topic can retain seven days of data and still lose recent acknowledged records if durability settings are weak. A strongly replicated topic can also delete records after the configured retention window.

Time-based retention should follow replay needs

retention.ms defines how long Kafka retains log segments for topics using delete-based cleanup.

Choose the window from actual recovery and product needs:

• maximum expected consumer outage;
• time needed to deploy a fixed consumer;
• replay requirements after a bug;
• audit or downstream reconstruction needs;
• incident-detection delay;
• data-governance limits.

A useful starting formula is:

maximum consumer outage
+ repair / deployment time
+ replay safety margin
= minimum useful retention window

If a consumer can be down for 36 hours during a serious incident, a 24-hour retention window cannot support full replay afterward.

Size-based retention can shorten the effective history window

retention.bytes limits how much log data a partition can retain. Under high traffic, the size limit may be reached before the time limit.

This means a team cannot promise “seven days of replay” from retention.ms alone when a smaller byte cap may delete older segments first.

Estimate retained storage from observed write volume:

ingest bytes per second
× 86,400 seconds/day
× retention days
× replication factor
+ operational headroom

For example, a topic ingesting an average 2 MB/s produces about 172.8 GB of raw log data per day before replication overhead. Seven days is roughly 1.21 TB before multiplying for replicas and allowing for peaks, segment overhead, and other topics.

Use measured compressed/on-disk growth from your own cluster for final capacity planning.

Delete and compact cleanup policies serve different data models

Kafka supports delete-based retention, log compaction, or both depending on topic configuration.

Delete policy

Use cleanup.policy=delete when the topic is an event history and records can disappear after a defined time or size window.

Typical examples:

• application events;
• logs;
• transient integration events;
• telemetry;
• replayable processing streams.

Compact policy

Compaction preserves the latest value for each key while allowing older superseded records to be removed over time.

This fits state-like topics such as:

• latest customer configuration;
• entity state changes keyed by ID;
• changelog topics;
• materialized-state reconstruction.

Compaction is not the same as “keep everything forever.” It intentionally discards older values for the same key once compaction conditions are met.

Delete + compact

Some workloads use both policies so key history is compacted while old segments can still age out.

Choose this only when the team understands both lifecycle effects.

Retention should be longer than normal consumer lag

A lagging consumer needs the underlying records to remain present long enough to catch up.

Monitor:

• current maximum lag;
• event age at the consumer;
• catch-up throughput;
• incoming producer rate;
• oldest retained offset/time;
• expected repair window.

If the consumer is falling behind faster than it can recover, increasing retention only delays data loss. The real problem may be consumer throughput, partition count, downstream dependency latency, or application errors.

Retention is a recovery budget, not a substitute for fixing an unhealthy consumer.

Replication protects retained records from node failure

Replication factor determines how many broker copies exist for a partition. In-sync replicas identify the replicas considered caught up enough to participate in durability and leadership decisions.

For stronger write durability, production teams commonly combine:

replication factor > 1
+ appropriate min.insync.replicas
+ producer acks=all

The exact numbers depend on cluster size and availability goals. The principle is that retention controls how long records should exist, while replication controls how many healthy copies exist during that window.

Do not reduce replication simply because the topic has short retention. A 15-minute topic can still be business-critical during those 15 minutes.

min.insync.replicas creates a durability/availability trade-off

When producers use acks=all, min.insync.replicas can require a minimum number of in-sync replicas before writes are accepted.

A stricter configuration can reduce the chance of acknowledged data being lost after broker failures, but it can make writes unavailable when too few replicas are healthy.

That is an intentional trade-off:

The business should choose the failure behavior rather than inherit defaults accidentally.

Long retention increases more than disk cost

Longer history affects:

• broker storage capacity;
• recovery/rebalance time;
• backup or archive volume if used;
• partition movement duration;
• disk replacement and rebuild time;
• operational impact of reassignments;
• compliance and deletion obligations.

Do not set “infinite” retention as a default merely because storage is available today.

Define why the history is needed and who owns its lifecycle.

Kafka is not automatically a permanent archive

Kafka can hold long histories, but teams should decide whether the event log is the authoritative archive or an operational streaming layer.

For long-term retention, alternatives may include:

Kafka topic
→ consumer / sink
→ object storage or analytical store

This can allow Kafka retention to follow operational replay needs while another system owns multi-month or multi-year history.

The correct boundary depends on query access, restore requirements, cost, and governance.

Retention changes need a rollout plan

Reducing retention can delete older segments once they become eligible. Treat large changes as potentially destructive.

Before changing a production topic:

1. record current settings;
2. confirm consumers do not need older offsets;
3. estimate storage impact;
4. verify archive requirements;
5. change one topic group at a time;
6. monitor segment deletion and consumer health;
7. keep a rollback plan for settings, while recognizing already-deleted data cannot be recreated from configuration rollback alone.

Increasing retention is usually safer but can still create disk-pressure risk.

Managed Kafka reduces broker operations, not retention design

Raff Managed Kafka reduces the host and broker-platform work around Kafka. The application team still owns topic lifecycle, keys, retention requirements, consumer recovery expectations, and the business meaning of event loss.

A managed service cannot determine whether a payments event should live for 24 hours, seven days, or be archived indefinitely. That decision belongs to the workload owner.

At Raff, the useful retention rule is: keep Kafka history long enough to survive the realistic repair-and-replay window, then archive elsewhere when the business needs longer history than the streaming platform should carry operationally.

Kafka retention checklist

• Each topic has a documented replay window.
• retention.ms reflects outage and repair time.
• retention.bytes will not unexpectedly shorten that window.
• Delete vs compact policy matches the data model.
• Replication is sized independently from retention duration.
• Producer acknowledgements and ISR requirements match durability goals.
• Consumer lag remains well inside the retained window.
• Disk capacity includes replication and traffic peaks.
• Long-term archive requirements are separate from Kafka where appropriate.
• Retention reductions are treated as destructive changes.

Conclusion

Kafka retention is a balance between replay safety, storage growth, consumer recovery, and data lifecycle.

Set the retention window from the longest realistic repair-and-replay requirement, verify byte limits do not shorten it, and keep replication/durability settings as a separate design decision. Use compaction for latest-value state, delete retention for bounded event history, and external archives when business history should outlive Kafka's operational window.

Sources

• Apache Kafka Topic Configuration
• Apache Kafka Broker Configuration
• Apache Kafka Operations
• Raff Managed Kafka
• Sources reviewed: 2026-08-24