Kafka architecture is an event-streaming design that divides durable ordered logs into topics and partitions, then assigns producers, consumers, replication, retention, and operational ownership around those partitions.
For small teams, Kafka becomes easier to operate when four decisions are made early: what belongs in a topic, what must stay ordered, how much parallel consumption is required, and who owns broker-level reliability versus application-level correctness. Those choices determine partition count, key strategy, consumer-group behavior, retention, monitoring, and recovery expectations.
Raff Technologies offers Managed Kafka within Managed Databases, while this guide keeps Kafka architecture decisions separate from platform operations. For implementation detail, continue with Kafka partitions and consumer groups and lag.
Kafka scales through partitions, but every added partition also adds ordering, ownership, replication, and operational consequences.
Kafka architecture starts with topics and event boundaries
A Kafka topic should represent a durable stream with a clear business or system meaning. The useful boundary is usually an event family whose retention, access, ordering, and consumer behavior can be managed together.
Good topic boundaries make these questions answerable:
- Who produces the events?
- Which consumers depend on them?
- What ordering guarantee matters?
- How long should the data remain replayable?
- Which keys should route related events to the same partition?
- What happens when a consumer falls behind?
Overly broad topics couple unrelated workloads. Excessively narrow topics create operational sprawl. Small teams should prefer a topic model that is easy to explain and monitor before optimizing for theoretical flexibility.
Kafka retains records independently of whether consumers have read them, subject to the topic's retention configuration. That makes retention a product and recovery decision, not merely a storage setting.
Partitions are the unit of ordering and parallelism
A Kafka topic is divided into partitions. Within a partition, records have a defined order and offsets. Across different partitions, Kafka does not provide one global total order.
This creates the central architecture trade-off:
| More partitions can provide | More partitions also create |
|---|---|
| more producer and consumer parallelism | more partition metadata and replication work |
| higher aggregate throughput | more complex key and ordering decisions |
| more consumer instances doing useful work | more rebalance surface |
| wider distribution across brokers | more operational objects to monitor |
A partition key is therefore part of the data model. If all events for an account, order, device, or tenant must be processed in order, that entity often needs to map consistently to the same partition.
Do not choose partition count from traffic alone. Consider ordering domains, expected consumer parallelism, growth, replay behavior, and the cost of changing the design later.
Consumer groups define how stream work is distributed
Consumers using the same consumer group cooperate to process a topic. Kafka assigns partitions among active group members so that, within a group, one partition is handled by one consumer at a time.
That means useful parallelism for a consumer group is bounded by available partitions. Adding consumers beyond the number of assigned partitions does not create more partition-level work.
Consumer-group architecture should define:
- which application or worker owns each group;
- acceptable consumer lag;
- offset-commit behavior;
- retry and dead-letter handling outside the broker where needed;
- what happens during a rebalance;
- whether replay is a normal operating action or an exceptional recovery action.
Lag is especially important because a broker can be healthy while a consumer-dependent product feature is hours behind. Kafka monitoring therefore needs to cover both infrastructure and stream-processing health.
Replication and retention protect different failure cases
Replication keeps additional copies of partitions so the cluster can tolerate selected broker failures without losing committed records, subject to the configured replication and acknowledgement model.
Retention determines how long old records remain available for replay or catch-up. These controls solve different problems.
| Control | Main purpose |
|---|---|
| Replication | keep current partition data available through broker failure |
| Producer acknowledgement policy | define when a write is considered committed |
| Retention time/size | define how long historical records remain replayable |
| Consumer offsets | track how far each consumer group has progressed |
| Application idempotency | make retries or replays safe at the business layer |
A replicated stream can still lose usefulness if retention expires before a failed consumer catches up. Long retention can still be unsafe if application reprocessing creates duplicate side effects. Kafka durability must therefore be designed together with consumer correctness.
The operating-model decision separates broker ownership from application ownership
Managed and self-hosted Kafka differ mainly in who owns the operational substrate.
| Responsibility | Managed Kafka | Self-hosted Kafka |
|---|---|---|
| Broker lifecycle and host operations | more provider-owned | team-owned |
| Broker patching and service maintenance | more provider-owned | team-owned |
| Topic and partition design | team-owned | team-owned |
| Producer key strategy | team-owned | team-owned |
| Consumer groups and offsets | team-owned | team-owned |
| Event schema and compatibility | team-owned | team-owned |
| Application retries and idempotency | team-owned | team-owned |
| Business validation after replay | team-owned | team-owned |
Choose a managed operating model when the platform supports the required Kafka workload and the team does not gain product value from operating brokers.
Choose self-hosting when a documented requirement needs control that the managed service does not expose and the team can own upgrades, monitoring, capacity, replication, security, and incident response.
The managed-vs-self-hosted choice should be evaluated as an ownership decision rather than a simple server-price comparison.
The decision framework maps workload requirements to Kafka design
A small team can use a compact framework before creating topics or scaling consumers.
| Requirement | Architecture decision |
|---|---|
| Per-entity ordering | choose a stable key that keeps related records in one partition |
| Higher consumer parallelism | ensure enough partitions exist for the required active consumers |
| Replay after downstream failure | set retention beyond the realistic detection and recovery window |
| Low tolerance for broker failure | use an appropriate replication and acknowledgement model |
| Bursty producers | monitor broker capacity and consumer lag independently |
| Many independent subscribers | give each logical subscriber its own consumer group |
| Safe retry/replay | design consumers to be idempotent where side effects matter |
| Small operations team | prefer managed broker operations when product requirements fit |
At Raff, the recommended decision order is event boundary → ordering key → partition parallelism → consumer ownership → retention → broker operating model. This prevents infrastructure choices from hard-coding an event model the application later has to work around.
Monitoring should cover both cluster health and stream health
Kafka needs two layers of observability.
Cluster-level signals help answer whether the broker layer is healthy:
- broker availability;
- partition leadership and replication health;
- disk and storage pressure;
- request latency and throughput;
- under-replicated or unavailable partitions.
Stream-level signals help answer whether the application is keeping up:
- consumer lag by group and partition;
- rebalance frequency;
- failed processing and retry volume;
- producer error rates;
- event age at consumption;
- replay progress during recovery.
The operational mistake to avoid is monitoring only brokers. A green broker dashboard does not prove that payments, notifications, indexing, analytics, or background jobs are current.
Separate Kafka platform ownership from application correctness
Choose the operating model separately from the event-processing design. A managed service can reduce broker provisioning and maintenance work, while your team still owns partition keys, event schemas, consumer behavior, retries, and replay correctness. Compare the trade-offs in Managed Kafka vs Self-Hosted Kafka.
Before choosing managed or self-hosted Kafka, decide who responds to broker failures and who is responsible when event processing is delayed or incorrect. Check networking constraints, retention and replay needs, monitoring ownership, and recovery expectations.
