Ecommet
Topic 43 of 75

Pub/Sub and Event Streaming

In publish-subscribe, one event reaches every interested subscriber. Event logs such as Kafka keep events on disk so consumers can replay them. Learn topics, partitions, consumer groups, offsets, and when to use a queue or a log.

Advanced 7 min read 4 quiz questions

Jump to the quiz

In a nutshell

  • Queue: each message goes to one consumer. Pub/sub: each event goes to every subscriber.
  • An event log (Kafka, Kinesis, Pulsar) stores events in order and keeps them; consumers track their own offset and can replay.
  • Partitions split a topic for parallelism; order is kept within a partition, so key events by entity.
  • A consumer group shares partitions among its members; different groups each get the full stream.

One event, many listeners

When an order is placed, email, invoicing, analytics and the warehouse all care. With a plain queue, the producer would have to send four messages and know about all four consumers. With pub/sub, it publishes one order.placed event to a topic, and each subscriber gets its own copy. New subscribers can be added without touching the producer.

Publish once, deliver to every subscriber

Publish once, deliver to every subscriber Orders service → order.placed: publish order.placed → Email: copy order.placed → Analytics: copy order.placed → Warehouse: copy Orders serviceorder.placedtopicEmailAnalyticsWarehousepublishcopycopycopy

All steps Each subscriber group receives every event and processes it independently.

Diagram as text
  1. The orders service publishes one event and does not know who is listening.
  2. Every subscriber receives its own copy of the event.

Queues vs logs

Classic brokers (RabbitMQ, SQS, SNS) deliver a message and delete it once acknowledged. Log-based systems (Apache Kafka, Amazon Kinesis, Apache Pulsar) append events to an ordered, durable log and keep them for a set time (days, or forever). Consumers do not remove anything; each one remembers how far it has read: its offset.

Message queueEvent log
After consumptionMessage deletedEvent kept until retention expires
Replay historyNoYes: reset the offset
OrderingBest effort, or per message groupStrict within a partition
Per-message routing, delays, prioritiesRichLimited
Best forTask distribution, jobs, commandsEvent streams, many independent consumers, rebuilding state, analytics

Partitions, consumer groups and offsets

A Kafka topic is split into partitions, each an ordered log. A producer chooses a partition by hashing a key (for example the order ID), so all events for one order land in one partition, in order. Within a consumer group, each partition is read by exactly one member, so a group with 3 members on a 6-partition topic gives each member 2 partitions. Another group reads the same topic independently, with its own offsets.

Partitions shared within a group, copied across groups

Partitions shared within a group, copied across groups Partition 0 → Billing #1 Partition 1 → Billing #1 Partition 2 → Billing #2 Partition 2 → Analytics: all partitions Partition 0orders 1, 4, 7…Partition 1orders 2, 5, 8…Partition 2orders 3, 6, 9…Billing #1group: billingBilling #2group: billingAnalyticsgroup: analyticsall partitions
Billing has two members sharing the partitions. Analytics is a separate group and reads everything again, at its own pace.
Diagram as text
  • Partition 0 → Billing #1
  • Partition 1 → Billing #1
  • Partition 2 → Billing #2
  • Partition 2 → Analytics: all partitions

A partitioned log with per-group offsets: groups read independently, and can replay

Java
import java.nio.charset.StandardCharsets;
import java.util.*;
import java.util.zip.CRC32;

class PartitionedLog {
    private final List<List<String>> partitions = new ArrayList<>();
    private final Map<String, Integer> offsets = new HashMap<>(); // "group/partition" -> next index

    PartitionedLog(int count) {
        for (int i = 0; i < count; i++) {
            partitions.add(new ArrayList<>());
        }
    }

    synchronized int append(String key, String event) {
        CRC32 crc = new CRC32();
        crc.update(key.getBytes(StandardCharsets.UTF_8));
        int p = (int) (crc.getValue() % partitions.size());
        partitions.get(p).add(event);
        return p;
    }

    // Returns events after the group's offset and moves the offset (commits) past them.
    synchronized List<String> poll(String group, int partition, int max) {
        String k = group + "/" + partition;
        List<String> log = partitions.get(partition);
        int from = offsets.getOrDefault(k, 0);
        int to = Math.min(log.size(), from + max);
        offsets.put(k, to);
        return new ArrayList<>(log.subList(from, to));
    }

    synchronized void seek(String group, int partition, int offset) {
        offsets.put(group + "/" + partition, offset);
    }
}
Python
import threading
import zlib

class PartitionedLog:
    def __init__(self, count):
        self.partitions = [[] for _ in range(count)]
        self.offsets = {}  # (group, partition) -> next index
        self.lock = threading.Lock()

    def append(self, key, event):
        with self.lock:
            p = zlib.crc32(key.encode()) % len(self.partitions)
            self.partitions[p].append(event)
            return p

    def poll(self, group, partition, max_events):
        """Returns events after the group's offset and moves the offset past them."""
        with self.lock:
            log = self.partitions[partition]
            start = self.offsets.get((group, partition), 0)
            end = min(len(log), start + max_events)
            self.offsets[(group, partition)] = end
            return log[start:end]

    def seek(self, group, partition, offset):
        with self.lock:
            self.offsets[(group, partition)] = offset

What logs make possible

  • Replay: a new service reads the topic from the beginning to build its own view of the data.
  • Event sourcing: store every change as an event; current state is a fold over the events.
  • Change data capture: stream every database change (Debezium) to caches, search indexes and warehouses.
  • Stream processing: compute running totals, joins and alerts over events as they arrive (Kafka Streams, Flink).

Common mistakes

  • Using a log where you need per-message retries, delays and priorities (a task queue fits better).
  • Choosing too few partitions, capping how far consumers can scale.
  • Expecting ordering across partitions; it only holds within one.
  • Committing offsets before processing, losing events on a crash.

Interview questions

Try answering out loud before you open each one.

Kafka or RabbitMQ for this system?

For distributing jobs to workers with acknowledgements, retries, delays and priorities, a message broker like RabbitMQ or SQS fits. For a high-volume stream of events read by many independent consumers, needing ordering per key and the ability to replay, a log like Kafka fits. Many systems use both.

How do you keep events for one order in order?

Use the order ID as the partition key, so all its events go to the same partition, which is strictly ordered and read by one consumer in a group at a time. Do not rely on order across different keys.

Check yourself

1. In pub/sub, how many subscribers receive each event?
Show the answer

Answer: Every subscriber (or subscriber group). That is the difference from a work queue.

2. A topic has 4 partitions and a consumer group has 6 members. How many members are active?
Show the answer

Answer: 4. Each partition is read by one member of the group at a time.

3. What lets a new service process last week's events in Kafka?
Show the answer

Answer: Retention plus starting from an old offset. Logs keep events, and consumers choose where to read from.

4. Ordering in Kafka is guaranteed…
Show the answer

Answer: Within a partition. Key related events to the same partition.

Further reading

Your notes

Saved in this browser as you type.

Finished this topic?

Mark it read to track your progress, then carry on with the next one.

Next topic