Kafka partitions are ordered log segments that divide a topic into independent units for storage, ordering, replication, and parallel processing.
The important design decision is not whether to use partitions; every Kafka topic is partitioned. It is how many partitions the topic should have, which record key should control placement, and how replication should protect those partitions without confusing fault tolerance with application parallelism.
Partition count should be treated as an application design decision before it is treated as a broker tuning parameter. A poor key can create hot partitions even in a large cluster, while an oversized partition count can add operational complexity without fixing the actual bottleneck.
For the broader event-streaming model, see Kafka Architecture for Small Teams. Here the key decisions concern partition keys, partition count, skew, replication factor, and how scaling affects record order.
Kafka partitions define the unit of order and parallel work
A Kafka topic can contain multiple partitions, and each partition is an ordered sequence of records. Kafka preserves record order within a partition rather than across the entire topic.
That makes a partition two things at once:
- an ordering boundary for related events; and
- a parallelism boundary for producers, brokers, and consumer work.
The architecture consequence is simple: adding partitions can create more independent work, but it also creates more places where ordering, replication, placement, recovery, and monitoring must be understood.
The current Apache Kafka introduction describes topics as partitioned across brokers for scalability and notes that records with the same key are written to the same partition. That same-key behavior is why partition design starts with event correctness rather than a target partition number.
The key question is what should determine the partition boundary for this workload?
Partition keys define the ordering domain
A partition key should represent the entity whose events need to remain together for ordered processing.
Examples can include:
- an account ID when account events must remain ordered;
- an order ID when each order has an independent event sequence;
- a device ID for telemetry that must be processed in device order;
- a tenant ID only when tenant-wide ordering is genuinely required.
The key should match the smallest business ordering domain that must remain consistent. Choosing a broader key than necessary reduces distribution. Choosing a key unrelated to the real ordering requirement can spread dependent events across partitions and move the ordering problem into the application.
Key cardinality also matters. A key with many reasonably distributed values can spread work across partitions. A key with only a few dominant values can concentrate traffic even when the topic has many partitions.
This creates a useful test:
| Key question | Good signal | Risk signal |
|---|---|---|
| Does the key match the required ordering boundary? | dependent events share a key | dependent events can land separately |
| Are there enough distinct keys? | many active values can distribute | very few values dominate traffic |
| Is traffic reasonably balanced across keys? | no single entity dominates | one tenant, account, or device dominates |
| Will the key remain stable? | key semantics persist over time | key definition may change with product logic |
At Raff, we treat the partition key as a correctness decision first, distribution decision second. More partitions cannot repair a key that funnels most of the workload into one ordering domain.
Partition count should follow concurrency, traffic, and growth
There is no universal correct partition count for a Kafka topic.
The count should be large enough for the required parallelism and expected growth, but not chosen from an arbitrary rule such as “more partitions are always better.”
Use four inputs:
- Required ordering. How many independent ordering domains can safely run in parallel?
- Consumer concurrency. How many consumers may need to process this topic concurrently within one consumer group?
- Traffic and storage distribution. Does the workload need more independent shards to spread sustained work?
- Growth headroom. How likely is the topic to require more parallelism later?
Partition count matters because one partition is assigned to at most one active consumer within a consumer group at a time. Ten consumers cannot create ten-way partition-level parallelism on a four-partition topic.
The reverse matters too. Creating hundreds of partitions for a workload that needs only a few independent workers adds metadata, replica placement, leadership, recovery, and monitoring surface without necessarily improving the application.
Do not use a fabricated “messages per second per partition” rule. Real capacity depends on record size, compression, acknowledgements, replication, storage, network, producer batching, consumer behavior, and broker resources.
A better planning question is:
required parallel workers + expected throughput growth + ordering constraints + operational headroom = partition-count decision
If traffic later exceeds the design, first identify whether the limit is partition parallelism, broker capacity, consumer processing time, or key skew. Those require different fixes.
Hot partitions are usually a distribution problem
A hot partition occurs when one partition receives substantially more work than its peers.
The cause is often the key distribution rather than the raw number of partitions.
Consider a SaaS event topic keyed by tenant ID. If one tenant produces 60% of all events and tenant-wide ordering is required, that tenant's records must remain together. Increasing the topic from 12 partitions to 24 does not split that one tenant's ordered stream across two partitions automatically.
The design options are then application decisions:
- keep strict tenant-wide ordering and provision for the hot key;
- narrow the ordering domain, such as tenant + resource, if the business logic permits it;
- separate unusually large workloads into a dedicated topic;
- redesign downstream processing so one entity does not serialize unrelated work.
This is why “add partitions” should not be the default answer to uneven throughput.
Use this diagnosis before changing the topic:
| Symptom | Likely first investigation |
|---|---|
| One partition much busier than peers | key distribution and hot entities |
| All partitions busy | aggregate topic capacity and broker resources |
| Consumers lag while brokers are comfortable | consumer processing and concurrency |
| More consumers do not increase throughput | partition count or a single hot partition |
| Storage pressure is concentrated | partition placement, key distribution, retention, and broker capacity |
Detailed lag, rebalance, and recovery behavior belongs in the upcoming Kafka consumer-groups guide. The partition page should stop at identifying whether partition design is constraining the consumer layer.
Replication factor changes fault tolerance, not application parallelism
Partition count and replication factor solve different problems.
A topic with 12 partitions and a replication factor of 3 still has 12 logical partitions for application ordering and consumer parallelism. The additional replicas provide redundant copies of those partitions; they do not turn the topic into 36 independent application partitions.
The Apache Kafka 4.3 operations guide describes replication factor separately from partition count because one controls replica copies while the other controls topic sharding.
| Setting | Primary purpose | Does it increase consumer-group parallelism? |
|---|---|---|
| Partition count | ordering domains and parallel work | Yes, when consumers can use the additional partitions |
| Replication factor | redundant copies and broker-failure tolerance | No |
| Broker count | physical capacity and replica placement | Not by itself |
| Consumer count | processing workers | Only up to available assigned partitions |
Kafka documentation commonly illustrates production replication with multiple copies; the exact factor should follow the failure tolerance, cluster size, durability expectations, and placement model of the workload.
A higher replication factor also consumes more storage and replication traffic. It should therefore be chosen as a reliability decision, not as a performance shortcut.
For the relationship between replication, acknowledgements, retention, and data-loss tolerance, see Kafka Retention: Durability, Replication & Data-Loss Trade-Offs.
Increasing partition count later can change routing assumptions
Partition count is not always permanent, but changing it later has consequences.
Kafka supports increasing the partition count of a topic. However, the current operations documentation warns that key-based partition assignment can change when the number of partitions changes. A producer that maps a key across N partitions may map that same key differently after the topic has N + x partitions.
That means teams should not assume that adding partitions preserves the historical location of every key.
Two effects matter:
- New records may route differently. A key that previously mapped to partition 2 may map to another partition after the count changes, depending on the producer's partitioning strategy.
- Existing records are not automatically reshuffled into the new partition layout. Historical data remains where it was written while new traffic can use the expanded set.
For applications that depend on uninterrupted per-key ordering across the change, this requires planning. The safe approach is to treat partition expansion as an architecture change with producer, consumer, and ordering implications rather than a routine capacity toggle.
Kafka's current operations guide also states that partition count cannot be reduced. That asymmetry is another reason not to create a very large count without a workload reason.
At Raff, the practical rule is to leave reasonable growth headroom but avoid speculative over-partitioning. A partition increase should have a documented reason such as consumer concurrency, sustained throughput, or a deliberate change in workload distribution.
The decision framework separates key, partition, replica, and broker choices
Partition planning becomes clearer when four decisions are kept separate.
| Requirement | Primary design lever | Question to answer |
|---|---|---|
| Preserve event order for an entity | Partition key | What is the smallest required ordering domain? |
| Increase independent processing | Partition count | How many parallel units does the workload need? |
| Avoid one overloaded shard | Key distribution | Are traffic-heavy keys skewing one partition? |
| Tolerate broker failure | Replication factor and placement | How many copies and failure domains are required? |
| Add physical capacity | Broker capacity/count | Is the cluster resource-bound rather than partition-bound? |
| Scale consumers | Consumer count + partitions | Are enough partitions available for useful workers? |
| Expand later | Partition-count change plan | Will key routing or ordering assumptions change? |
Use this sequence before changing infrastructure:
1. Define the ordering domain. 2. Choose a stable, well-distributed key where possible. 3. Estimate required consumer concurrency and throughput headroom. 4. Choose partition count from those requirements. 5. Choose replication from failure tolerance, not parallelism. 6. Verify broker capacity and replica placement. 7. Revisit the design when measurements show a specific bottleneck.
The same framework helps diagnose scaling actions:
| Observed constraint | Most likely action |
|---|---|
| One key dominates one partition | revisit key or topic boundary |
| All partitions saturated but brokers have room | evaluate more partitions |
| Brokers are resource-bound across many partitions | add/resize broker capacity or use managed scaling |
| Consumer count exceeds partitions | more consumers alone will not help |
| Failure tolerance is insufficient | review replication and placement |
| Lag grows with healthy brokers | inspect consumer processing before changing partition count |
This separation prevents a common architecture mistake: using broker capacity, partition count, and replication factor as interchangeable scaling controls.
Managed Kafka changes execution ownership, not partition design
A managed Kafka service can take ownership of more broker lifecycle and infrastructure work, but the application team still owns the partition model.
Your team still decides:
- topic boundaries;
- partition keys;
- required ordering;
- partition count and growth expectations;
- consumer concurrency;
- acceptable skew;
- retention requirements;
- application behavior when routing or topology changes.
The provider can execute more of the underlying broker operations, capacity work, maintenance, and supported service changes. That distinction is the same operating-model boundary described in Managed Kafka vs Self-Hosted Kafka.
Raff maintains a dedicated managed Kafka product path. Product availability, supported versions, plan sizes, and pricing can change independently of this architecture guide, so use the live product page for current service details.
For design work, the durable sequence remains:
key correctness -> partition distribution -> consumer parallelism -> replication requirements -> broker capacity.
