Pub/Sub and Event Streaming
In publish-subscribe, one event reaches every interested subscriber. Event logs such as Kafka keep events on disk so consumers can replay them. Learn topics, partitions, consumer groups, offsets, and when to use a queue or a log.
In a nutshell
- Queue: each message goes to one consumer. Pub/sub: each event goes to every subscriber.
- An event log (Kafka, Kinesis, Pulsar) stores events in order and keeps them; consumers track their own offset and can replay.
- Partitions split a topic for parallelism; order is kept within a partition, so key events by entity.
- A consumer group shares partitions among its members; different groups each get the full stream.
One event, many listeners
When an order is placed, email, invoicing, analytics and the warehouse all care. With a plain queue, the producer would have to send four messages and know about all four consumers. With pub/sub, it publishes one order.placed event to a topic, and each subscriber gets its own copy. New subscribers can be added without touching the producer.
Publish once, deliver to every subscriber
All steps Each subscriber group receives every event and processes it independently.
- The orders service publishes one event and does not know who is listening.
- Every subscriber receives its own copy of the event.
Diagram as text
- The orders service publishes one event and does not know who is listening.
- Every subscriber receives its own copy of the event.
Queues vs logs
Classic brokers (RabbitMQ, SQS, SNS) deliver a message and delete it once acknowledged. Log-based systems (Apache Kafka, Amazon Kinesis, Apache Pulsar) append events to an ordered, durable log and keep them for a set time (days, or forever). Consumers do not remove anything; each one remembers how far it has read: its offset.
| Message queue | Event log | |
|---|---|---|
| After consumption | Message deleted | Event kept until retention expires |
| Replay history | No | Yes: reset the offset |
| Ordering | Best effort, or per message group | Strict within a partition |
| Per-message routing, delays, priorities | Rich | Limited |
| Best for | Task distribution, jobs, commands | Event streams, many independent consumers, rebuilding state, analytics |
Partitions, consumer groups and offsets
A Kafka topic is split into partitions, each an ordered log. A producer chooses a partition by hashing a key (for example the order ID), so all events for one order land in one partition, in order. Within a consumer group, each partition is read by exactly one member, so a group with 3 members on a 6-partition topic gives each member 2 partitions. Another group reads the same topic independently, with its own offsets.
Partitions shared within a group, copied across groups
Diagram as text
- Partition 0 → Billing #1
- Partition 1 → Billing #1
- Partition 2 → Billing #2
- Partition 2 → Analytics: all partitions
A partitioned log with per-group offsets: groups read independently, and can replay
import java.nio.charset.StandardCharsets;
import java.util.*;
import java.util.zip.CRC32;
class PartitionedLog {
private final List<List<String>> partitions = new ArrayList<>();
private final Map<String, Integer> offsets = new HashMap<>(); // "group/partition" -> next index
PartitionedLog(int count) {
for (int i = 0; i < count; i++) {
partitions.add(new ArrayList<>());
}
}
synchronized int append(String key, String event) {
CRC32 crc = new CRC32();
crc.update(key.getBytes(StandardCharsets.UTF_8));
int p = (int) (crc.getValue() % partitions.size());
partitions.get(p).add(event);
return p;
}
// Returns events after the group's offset and moves the offset (commits) past them.
synchronized List<String> poll(String group, int partition, int max) {
String k = group + "/" + partition;
List<String> log = partitions.get(partition);
int from = offsets.getOrDefault(k, 0);
int to = Math.min(log.size(), from + max);
offsets.put(k, to);
return new ArrayList<>(log.subList(from, to));
}
synchronized void seek(String group, int partition, int offset) {
offsets.put(group + "/" + partition, offset);
}
}import threading
import zlib
class PartitionedLog:
def __init__(self, count):
self.partitions = [[] for _ in range(count)]
self.offsets = {} # (group, partition) -> next index
self.lock = threading.Lock()
def append(self, key, event):
with self.lock:
p = zlib.crc32(key.encode()) % len(self.partitions)
self.partitions[p].append(event)
return p
def poll(self, group, partition, max_events):
"""Returns events after the group's offset and moves the offset past them."""
with self.lock:
log = self.partitions[partition]
start = self.offsets.get((group, partition), 0)
end = min(len(log), start + max_events)
self.offsets[(group, partition)] = end
return log[start:end]
def seek(self, group, partition, offset):
with self.lock:
self.offsets[(group, partition)] = offsetWhat logs make possible
- Replay: a new service reads the topic from the beginning to build its own view of the data.
- Event sourcing: store every change as an event; current state is a fold over the events.
- Change data capture: stream every database change (Debezium) to caches, search indexes and warehouses.
- Stream processing: compute running totals, joins and alerts over events as they arrive (Kafka Streams, Flink).
Common mistakes
- Using a log where you need per-message retries, delays and priorities (a task queue fits better).
- Choosing too few partitions, capping how far consumers can scale.
- Expecting ordering across partitions; it only holds within one.
- Committing offsets before processing, losing events on a crash.
Interview questions
Try answering out loud before you open each one.
Kafka or RabbitMQ for this system?
For distributing jobs to workers with acknowledgements, retries, delays and priorities, a message broker like RabbitMQ or SQS fits. For a high-volume stream of events read by many independent consumers, needing ordering per key and the ability to replay, a log like Kafka fits. Many systems use both.
How do you keep events for one order in order?
Use the order ID as the partition key, so all its events go to the same partition, which is strictly ordered and read by one consumer in a group at a time. Do not rely on order across different keys.
Check yourself
Related topics
Further reading
- Introduction to Apache Kafka Apache Kafka
- Publisher-Subscriber pattern Microsoft Azure Architecture Center
- Kafka design: persistence, consumer position and delivery semantics Apache Kafka documentation
Your notes
Saved in this browser as you type.
Finished this topic?
Mark it read to track your progress, then carry on with the next one.