• streamsocial-engagement-consumer: the first consumer, reading content-interactions (Day 3) for likes, comments, shares

• @KafkaListener with Spring’s JsonDeserializer<ContentInteractionEvent>, symmetric with Day 4’s producer-side serialization choices

• A second dashboard panel, live, alongside the first — nothing about Day 4’s panel changes

A correction worth making out loud

The error-handling class this lesson was originally scoped around, SeekToCurrentErrorHandler, doesn’t exist in current Spring Kafka. It was deprecated in version 2.8 and removed outright in 3.x. That’s the kind of thing that happens constantly in a fast-moving ecosystem, and pretending otherwise would mean shipping a lesson that doesn’t compile against anything you’d actually install today. The replacement, DefaultErrorHandler, does the same job the deprecated class did: on a listener failure, it seeks the failed record — and everything after it in that poll batch — back to be redelivered by the broker, retries against a configurable backoff, and only after retries are exhausted hands the record to a recoverer instead of retrying forever or crashing the consumer thread.

The poll loop, and what “consuming” actually means

A Kafka consumer doesn’t get pushed messages — it polls. poll() asks the broker for the next batch of records sitting after the consumer’s current offset on whatever partitions it’s assigned, gets a batch back, and the application processes that batch before calling poll() again. @KafkaListener hides this loop entirely: Spring’s listener container runs it on a background thread and calls your annotated method once per record. What you’re writing is the body of that loop, not the loop itself.

Deserialization is where the producer’s choices come back

Day 4’s producer configured JsonSerializer.ADD_TYPE_INFO_HEADERS = false — no type metadata riding along on every record, because every message on a given topic is one known type and saying so on every single message is waste. That decision has a mirror-image requirement on the consuming side: JsonDeserializer needs to be told the target type directly, new JsonDeserializer<>(ContentInteractionEvent.class, false), rather than expecting a header that was never sent. Producer and consumer configuration aren’t independent — a serialization decision made in one module is a contract the other module has to honor, and getting that pairing wrong is a silent, confusing failure mode (deserialization exceptions with no obvious cause) rather than a loud one. Today’s own EngagementTestDataGenerator follows the exact same wire format for this reason — it’s standing in for a real content-interactions producer this course hasn’t built yet, and doing so faithfully matters.

Validation didn’t stop mattering just because the JSON parsed

Successfully deserializing a record proves the bytes were well-formed JSON matching the expected shape. It proves nothing about whether the values make sense. This listener runs every deserialized ContentInteractionEvent through the same jakarta.validation.Validator Day 1 used — a blank contentId that somehow made it onto the topic is caught here, thrown as an exception, and handled by the same retry-then-recover path as any other processing failure. Bean Validation isn’t a REST-layer-only concern; it’s useful anywhere a payload crosses a trust boundary, and a Kafka topic populated by services you don’t fully control is exactly that.

What “graceful” actually means here

Today’s failure hook — a deliberately poisoned contentId of post-BOOM — exists to make the failure path observable, not because that’s a real production scenario. What matters is what happens when it fires: the record gets retried twice, one second apart, and if it still fails, RecoveryTracker logs it as STRUCTURED_ERROR and counts it — instead of the consumer thread dying (which would stop every other partition assigned to it from being processed too) or the record being silently dropped. “Graceful” here specifically means: bounded retries, then a loud, greppable record of what got given up on. It does not yet mean routing the record somewhere it can be reprocessed later — that’s a dead letter topic, and that’s a later lesson’s job, once poison-pill handling gets the dedicated treatment it deserves.

The dashboard gets its second panel

ContentInteractionsFeedListener follows the exact shape UserActionsFeedListener established Day 4 — its own consumer group (dashboard-live-feed-content-interactions, deliberately distinct from both engagement-consumer‘s group and the user-actions panel’s own group), broadcasting through the same EventStreamBroadcaster to a second SSE stream. Nothing about the user-actions panel changes; the two feeds run side by side, each backed by its own independent, observational consumer. Run EngagementTestDataGenerator and the same 14 events show up in two places at once — streamsocial-engagement-consumer‘s own log (processing them for real, with validation and error handling) and the dashboard’s browser view (just watching, no business logic at all) — which is itself a small, live demonstration of what a consumer group actually buys you: two entirely independent readers of the same topic, neither aware of the other, both correct.

Implementation Guide:

GitHub Link :

https://github.com/sysdr/streamsocial-java/tree/main/day05-v2-streamsocial-source/streamsocial

Build, Test, and Demo the Engagement Consumer

The idea, in pseudo-code

KafkaConsumerConfig:
    consumerFactory = DefaultKafkaConsumerFactory(
        bootstrap.servers, group.id="engagement-consumer",
        keyDeserializer=String, valueDeserializer=Json(ContentInteractionEvent))

    errorHandler = DefaultErrorHandler(
        recoverer = RecoveryTracker::recordRecovery,
        backOff = FixedBackOff(1000ms, retries=2))
    # replaces the removed SeekToCurrentErrorHandler - same seek-and-retry job

@KafkaListener("content-interactions"):
    onInteraction(record):
        event = record.value()
        if not valid(event): throw   # -> retried, then recovered
        if event.contentId == "post-BOOM": throw  # today's demo hook
        log STRUCTURED_EVENT ...
        processed.add(event)

# Dashboard, same shape as Day 4's panel, own group:
ContentInteractionsFeedListener:
    group.id = "dashboard-live-feed-content-interactions"
    on record: broadcast LiveFeedItem to SSE subscribers of "content-interactions"

Step 1 — Bring up the cluster and topics

cd streamsocial-infra && ./scripts/start.sh && cd ..
mvn org.codehaus.mojo:exec-maven-plugin:3.3.0:java -pl streamsocial-common \
  -Dexec.mainClass=com.streamsocial.common.demo.TopicBootstrapDemo \
  -Dstreamsocial.topics.user-actions-partitions=12 \
  -Dstreamsocial.topics.content-interactions-partitions=6

Step 2 — Start the dashboard and the consumer

mvn -pl streamsocial-dashboard -am spring-boot:run

Open http://localhost:8080 — two panels now, user-actions and content-interactions, both at 0.

In another terminal:

mvn -pl streamsocial-engagement-consumer -am spring-boot:run

Expected: Spring Boot startup log, ending quietly — no web server line, since this module has none.

Step 3 — Generate test traffic and watch both places at once

mvn org.codehaus.mojo:exec-maven-plugin:3.3.0:java -pl streamsocial-engagement-consumer \
  -Dexec.mainClass=com.streamsocial.consumer.demo.EngagementTestDataGenerator

Expected in the consumer’s terminal: 14 lines like:

STRUCTURED_EVENT event=engagement-processed interactionType=CONTENT_LIKED userId=user-0 contentId=post-0 partition=1 offset=0

Then, a couple seconds later:

STRUCTURED_ERROR event=engagement-processing-recovered topic=content-interactions partition=... offset=... key=post-BOOM reason=simulated processing failure for post-BOOM

Expected on the dashboard, at roughly the same time: the content-interactions panel’s counter climbs to 14 (not 15 — the poisoned event never succeeds, so it never gets broadcast either, the same way it never lands in the consumer’s processed list) and 14 rows appear.

Step 4 — Run the tests

mvn -pl streamsocial-engagement-consumer -am verify
mvn -pl streamsocial-dashboard -am verify

Expected: all green. EngagementEventListenerIT‘s poisonedEventIsRetriedThenRecoveredNotLostSilently takes noticeably longer than the other tests — it’s waiting out the real retry backoff, not asserting instantly. ContentInteractionsFeedBroadcastIT proves the new panel’s data path the same way Day 4’s UserActionsFeedBroadcastIT did for the first one.

Success criteria

EngagementEventListenerIT passes — both the normal-processing case and the poisoned-record recovery case, against a real broker, with the real listener bean. Run the test-data generator against a live consumer and dashboard together, and find STRUCTURED_EVENT/STRUCTURED_ERROR lines in the consumer’s log at the same moment matching rows appear on the dashboard’s second panel.

Working Demo Link :

Homework

Add a second @KafkaListener method to a new class, EngagementMetricsListener, that listens to content-interactions but with a different group.id. Have it maintain a simple in-memory count per InteractionType.

Implementation checklist:

1. New listener class, new @KafkaListener(topics = "content-interactions", groupId = "engagement-metrics") with its own ConsumerFactory/container factory bean, following KafkaConsumerConfig‘s existing pattern.

2. A ConcurrentHashMap<InteractionType, Long> (or similar) tallying counts, with a getter for tests.

3. A test that publishes a known mix of interaction types and asserts the final counts match.

4. Run both listeners at once and confirm both process every message — think about why a different group.id makes that true, tying back to what today’s dashboard panels already demonstrated.

Solution hints

• Two consumers in two different groups both get their own copy of every record — consumer groups are what make records get split across consumers, not consumers in different groups compete for the same records. The dashboard’s two independent panels reading the same topics as the business consumers are already a live example of exactly this.

• Your metrics listener doesn’t need the DefaultErrorHandler/retry setup this lesson’s main listener has — it’s fine for homework purposes to let a bad record just log and move on, since the goal here is understanding group isolation, not re-solving today’s error handling.

• If your counts come out lower than expected, double-check you created a new, distinctly-named container factory bean for the new group ID rather than accidentally reusing engagementListenerContainerFactory, which is already bound to group.id=engagement-consumer at the factory level.

streamsocial-producer-service — a Spring Boot app exposing POST /api/posts, backed by a pooled KafkaTemplate<String, UserActionEvent>

streamsocial-dashboard — the system’s first browser-visible surface: a live events/sec counter fed by a raw kafka-clients Consumer, pushed to the page over Server-Sent Events

The moment where Day 1’s event records and Day 3’s topics stop being things you can only prove with a CLI, and start being things you can watch happen

This is the first lesson where StreamSocial does something a browser can show you. Everything before today built the plumbing; today the plumbing carries real water.

Core concept: a producer is thread-safe, but “thread-safe” isn’t “fast enough”

A single KafkaProducer can be shared safely across every thread in your app — it’s built for that. But under real concurrent load, that one producer’s internal network client becomes a queue everyone waits behind. DefaultKafkaProducerFactory.setProducerPerThread(true) gives each calling thread its own producer instead, pulled from a ThreadLocal pool — that’s what “connection pooling” means for a Kafka producer, distinct from the JDBC sense of the term most engineers learn first.

The other half of today’s concept is what key you send with. PostProducerService keys every PostCreated event by userId, not randomly or round-robin — because Kafka guarantees ordering only within a partition, and every event for the same user has to land on the same partition to keep that user’s timeline in order. You’ll go deep on this on Day 14; today, just notice it’s already there in the code, on purpose.

Where this fits in StreamSocial

The producer service is the front door: it’s the only thing standing between “a user tapped Post” and a durable fact in Kafka. Everything downstream — Day 5’s engagement consumer, Day 41’s Streams topology, Day 44’s trending scores — depends on this door working correctly and staying open under load, because none of them can process an event that never arrived.

The dashboard, meanwhile, isn’t a side project — it’s the same kind of real-time visibility system StreamSocial’s own engineers would build to answer “is ingestion healthy right now?” without SSH-ing into a broker. It reads user-actions with a plain KafkaConsumer on a fresh, random group.id every time it starts, subscribed from latest — a monitoring consumer has no business replaying history, it only cares what’s happening now.

Architecture: request in, fact out, dashboard watching

A POST /api/posts becomes a PostCreated event, gets validated by the same Bean Validation constraints Day 1 defined, and is handed to the pooled KafkaTemplate. The send is asynchronous — the HTTP response doesn’t block on the broker round-trip, it completes when Kafka acknowledges the write and returns the real partition and offset the event landed on.

The dashboard’s browser connection has its own lifecycle, independent of any single event: it opens an EventSource, sits in a live state receiving a JSON snapshot every second from the server’s @Scheduled broadcaster, and silently reconnects if the connection drops — the browser’s EventSource API does that retry for you, no extra code required.

Implementation Guide :

GitHub Link :

https://github.com/sysdr/streamsocial-java/tree/main/streamsocial-day04/streamsocial

Fundamentals first

KafkaProducer batches records internally and manages its own background I/O thread — that’s why it’s thread-safe to share. The pooling question isn’t “is one producer safe to share,” it’s “does sharing it become a bottleneck before your network or the broker does.” At real scale, the answer is often yes, which is why Day 4’s pattern exists at all.

Today’s specific pattern: producer-per-thread + async send

Pseudo-code shape of the pooling config:

ProducerFactory<String, UserActionEvent> factory = new DefaultKafkaProducerFactory(props)
factory.setProducerPerThread(true)   // each thread gets its own pooled producer, not one shared instance
KafkaTemplate template = new KafkaTemplate(factory)

And the send path itself, keyed and async:

event = new PostCreated(eventId, now, userId, postId, content)
template.send(topic="user-actions", key=event.userId(), value=event)
    .thenApply(result -> respond 202 with result.partition(), result.offset())

Nothing here blocks the HTTP thread on the broker round-trip — the CompletableFuture completes when Kafka acknowledges the write, and the controller maps that into the HTTP response.

How it snaps into the system

streamsocial-producer-service imports PostCreated/UserActionEvent from streamsocial-common (Day 1) — it does not redefine them. It sends to user-actions, the exact topic Day 3’s TopicBootstrap created — it does not create its own topic. streamsocial-dashboard reads that same topic independently, with its own consumer group, so producer and dashboard have zero direct coupling to each other — only to the topic between them.

Build, test, and demo — step by step

1. Unit tests (no broker needed)

cd streamsocial
mvn -pl streamsocial-producer-service,streamsocial-dashboard -am test

Expected: CreatePostRequestValidationTest (3 tests) and ThroughputTrackerTest (2 tests) pass — these test validation and the throughput-counting logic in isolation.

2. Integration test (real broker via Testcontainers)

mvn -pl streamsocial-producer-service -am verify

Expected: PostProducerServiceIT spins up a real Kafka container, sends a real event, and reads it back with a real consumer. On a standard Docker host or CI runner this passes outright. If your sandbox’s Docker bridge doesn’t expose a Testcontainers-compatible API (some WSL2 + Docker Desktop setups don’t), start.sh detects that specific failure and falls back to manual verification against the already-running cluster instead of halting — check its output for which path actually ran.

3. Full lifecycle, with Docker

./start.sh

Expected: cluster up, topics bootstrapped (12/6 partitions locally — production default is 1000/500, see below), both services healthy, 5 real posts sent, and a live SSE snapshot printed showing totalEvents reflecting them.

4. Manual verification

curl -X POST localhost:8082/api/posts \
  -H "Content-Type: application/json" \
  -d '{"userId":"<any-uuid>","content":"hello StreamSocial"}'

Expected: 202 Accepted with a real partition and offset.

Open http://localhost:8080 and watch the events/sec number spike the instant you send another one.

5. No-Docker note

The producer and dashboard are plain Spring Boot apps — mvn -pl streamsocial-producer-service spring-boot:run works without Docker for compiling/starting the JVM process itself, but both need a real broker to do anything useful, so Docker (or a broker reachable some other way) is required for this lesson’s actual demo, unlike Day 1.

6. Shut down

./stop.sh

Expected: both apps and all three brokers stop cleanly; safe to re-run start.sh immediately after.

The local-scale vs. production-scale gap

The Day 4 challenge target is 5M posts/second. This lesson’s code doesn’t pretend to hit that on a laptop against a 3-broker local cluster — that number comes from partitioning (1000 partitions on user-actions, per Day 3) and running across a real horizontally-scaled cluster, not from any one producer’s configuration. What today’s code proves is the mechanism — pooling, async sends, correct keying — at a scale your machine can actually run in seconds.

Success criterion

Run ./start.sh. It will bring up the 3-broker cluster, bootstrap topics, start both new services, fire 5 real posts through the producer, and print a live snapshot from the dashboard’s own SSE stream — you should see totalEvents in that snapshot match what was just posted. Then open http://localhost:8080 yourself, curl -X POST localhost:8082/api/posts a few more times by hand, and watch the number move in real time. If you can explain why the dashboard’s consumer never commits an offset, you’ve got today’s concept.

Working Demo Link :

Homework assignment

1. Add a PostProducerThroughputTest (JUnit, real broker via Testcontainers) that sends 1,000 events in a tight loop across multiple threads and asserts it completes in under 5 seconds — a real, if modest, throughput floor.

2. Change the dashboard’s sparkline window from 30 seconds to 60, and confirm ThroughputTrackerTest‘s history-cap test still passes after you update its expected size.

3. Add a second dashboard panel: total events per partition (not just total), using the same ConsumerRecord.partition() you already have access to in the poll loop.

Solution hints

• For the throughput test, submit sends from an ExecutorService with several threads and CompletableFuture.allOf(...).get() to wait for all of them — this is exactly what exercises setProducerPerThread(true).

• ThroughputTracker‘s HISTORY_SIZE constant is the only thing that needs to change for the 60-second window; the deque logic already handles any size.

• Per-partition counts need a Map<Integer, AtomicLong> inside ThroughputTracker rather than a single counter — keyed by record.partition() from the consumer, broadcast the same way as the existing snapshot.

• user-actions and content-interactions, the first two topics on the Day 2 cluster

• TopicBootstrapper: an idempotent, AdminClient-based provisioning class in streamsocial-common — the first code in this repository that talks to Kafka at all

• The actual math behind why one topic gets 1000 partitions and the other gets 500, not a round number picked because it looked reasonable

A partition is the unit everything else scales around

A Kafka topic isn’t one log — it’s however many partitions you give it, each an independently ordered, independently replicated append-only log. That single fact is why partition count is a decision made once, carefully, rather than a config value bumped later without thinking: partition count sets the ceiling on consumer parallelism (a partition can only be read by one consumer within a group at a time), and it’s the unit min.insync.replicas and replication actually operate on.

Get it too low and you cap your own future scale-out. Get it too high relative to what a topic’s key space can actually spread across, and you pay real costs — more file handles per broker, longer leader-election windows during failover, more replication traffic — for parallelism nothing ever uses.

The math, not a guess

StreamSocial’s platform-wide target is 50M events/second at peak, across every topic. Today’s two topics carry an assumed split of that: 30M/sec on user-actions (posts, follows, profile updates), 20M/sec on content-interactions (likes, comments, shares).

Using a conservative sustained-throughput budget of roughly 30,000 events/sec per partition — small JSON events, replication factor 3, headroom left for consumer fan-out — the arithmetic is direct:

user-actions:          30,000,000 / 30,000  ≈ 1000 partitions
content-interactions:  20,000,000 / 30,000  ≈  667 partitions

user-actions lands on 1000 partitions, matching the math. content-interactions gets capped at 500 — deliberately below what the raw division suggests.

Why content-interactions is capped below the math

This is the part worth sitting with, because it’s the opposite of “more partitions is always safer.” user-actions is keyed by userId: with millions of distinct users, traffic spreads close to evenly across however many partitions you give it, so raw partition count converts almost directly into usable parallelism.

content-interactions is keyed by contentId, and content interaction volume follows a power law — a small number of posts go viral and absorb a wildly disproportionate share of likes, comments, and shares, while most content gets a handful of interactions total. Give that topic 667 or even 1000 partitions and the hot keys still concentrate on however many partitions their hashes land on; the rest of the extra partitions sit close to idle. You’ve added broker overhead — more replicated logs, more leader elections to manage on failover — without buying any real relief for the actual bottleneck. That bottleneck gets fixed in Day 15 with a smarter Partitioner, not with a bigger partition count today. Capping at 500 here is that engineering judgment made explicit, not a compromise.

Idempotent, not “run once and hope”

TopicBootstrapper follows a specific, proven shape: list what exists, diff against the desired catalog, only create what’s actually missing. Running it twice in a row against the same cluster produces the same end state and doesn’t throw on the second run just because the topics already exist. It’s extended one step further than the minimal version: topics that already exist get their partition count verified against the catalog, not just assumed correct — Kafka can’t shrink partitions at all, and growing them later changes which partition a given key hashes to for every message already flowing through the topic, quietly breaking per-key ordering guarantees downstream consumers depend on. A mismatch is reported loudly, not silently fixed; that’s a decision for a human.

This is also the first code in the repository with a kafka-clients dependency — via Admin/AdminClient only. Nothing here produces or consumes a message; that’s Day 4 and Day 5’s job.

Today’s demo runs at local-demo-scale — and says so

StreamSocialTopics carries 1000 and 500 as its real, documented production defaults — that’s what the code says, and what production would actually run. Standing up 1000 partitions on a single 3-broker laptop cluster works, but it’s needlessly slow for a five-minute check, so today’s automated demo overrides both counts via system property to 12 and 6 — the same 2:1 ratio the real math produced, at a fraction of the wait. The override is never quiet about itself: the demo prints *** LOCAL-DEMO-SCALE RUN *** the moment it’s active, specifically so a reader watching the output never mistakes today’s fast local numbers for the real ones.

A verification note, honestly

This lesson’s own generation happened in an environment with no Docker installed at all — not a Testcontainers-can’t-reach-Docker situation, no Docker binary present, period. TopicBootstrapperIT is written and included as the real, portable, correct verification — it’s exactly what would run on any standard machine with Docker — but it was not actually executed to produce this lesson. That gap is stated here rather than papered over; run it yourself, and mvn -pl streamsocial-common -am verify is where to start.

Implementation Guide:

GitHub Link :

https://github.com/sysdr/streamsocial-java/tree/main/day03-v2-streamsocial-source/streamsocial

Build, Test, and Demo Topic Provisioning

The idea, in pseudo-code

TopicSpec(name, partitions, replicationFactor, configs)

catalog = [
    TopicSpec("user-actions",         partitions=1000, rf=3),  # 12 at local-demo-scale
    TopicSpec("content-interactions", partitions=500,  rf=3),  #  6 at local-demo-scale
]

bootstrap(desired):                       # Appendix C's proven shape
    existing = adminClient.listTopics()
    missing  = desired where name not in existing
    if missing: adminClient.createTopics(missing)

    for spec in present (not missing):    # this course's extension
        actual = adminClient.describeTopic(spec.name).partitionCount
        report verified or mismatched

Two calls to the cluster — list, then either create or describe — and everything else is comparing what came back against the catalog.

Step 1 — Make sure the Day 2 cluster is running

cd streamsocial-infra
./scripts/start.sh

If it’s already up from Day 2, this is a no-op.

Step 2 — Run the bootstrap at local-demo-scale

cd ..
mvn org.codehaus.mojo:exec-maven-plugin:3.3.0:java -pl streamsocial-common \
  -Dexec.mainClass=com.streamsocial.common.demo.TopicBootstrapDemo \
  -Dstreamsocial.topics.user-actions-partitions=12 \
  -Dstreamsocial.topics.content-interactions-partitions=6

Expected output (first run):

*** LOCAL-DEMO-SCALE RUN ***
Partition counts below are overridden for a fast local check.
The real production defaults are 1000 (user-actions) / 500 (content-interactions).
...
Created:            [user-actions, content-interactions]
Already present ok:  []
Mismatched:          []

Run the exact same command again:

Expected output (second run):

Created:            []
Already present ok:  [user-actions, content-interactions]
Mismatched:          []

That difference between the two runs is the whole point — same command, different (correct) behavior depending on cluster state.

Step 3 — Run the real production numbers, once, to see the difference

mvn org.codehaus.mojo:exec-maven-plugin:3.3.0:java -pl streamsocial-common \
  -Dexec.mainClass=com.streamsocial.common.demo.TopicBootstrapDemo

This attempts to create user-actions at the real 1000 partitions — expect it to take noticeably longer than Step 2, and note that it will report a mismatch against the 12/6-partition topics Step 2 already created (Kafka can’t shrink or silently grow partitions). Use ./scripts/stop.sh --wipe and start fresh if you want to try this cleanly.

Step 4 — Verify independently from the CLI

cd streamsocial-infra
./scripts/list-topics.sh

Expected output: both topic names listed, with PartitionCount matching whichever run (local-demo-scale or production) actually created them. This uses kafka-topics directly against the broker, not the Java AdminClient code path — two independent tools agreeing is a real check.

Step 5 — Run the tests

mvn -pl streamsocial-common -am verify

StreamSocialTopicsTest runs under Surefire, no Docker needed. TopicBootstrapperIT — note the IT suffix, not Test — runs under Failsafe during the verify phase, spins up a real single-node broker via Testcontainers, and needs Docker running. If you only run mvn test, this class is silently skipped; that’s Failsafe’s convention, not a bug.

Expected: all green.

Success criteria

Run TopicBootstrapDemo against the Day 2 cluster and confirm both topics get created. Run it again immediately and confirm the second run creates nothing — both topics report as already present and verified. Then explain, in one sentence each, why user-actions got 1000 partitions and why content-interactions didn’t get the full 667 the raw math suggested.

Working Demo Link :

Homework

Add a third topic to the catalog: moderation-events, sized for an assumed peak of 2M events/sec using the same 30,000 events/sec-per-partition budget this lesson used for the other two — but don’t cap it below the raw math this time. Justify in a one-line comment why moderation-events doesn’t need the same treatment content-interactions got, and add a local-demo-scale override for it too, following the existing pattern.

Implementation checklist:

1. Add a MODERATION_EVENTS constant to StreamSocialTopics, computing the partition count from the stated peak and budget, with its own resolvePartitions(...) call and system property key.

2. Add it to StreamSocialTopics.ALL.

3. Add a test asserting the catalog grew to 3 entries and the new topic’s default partition count matches your math.

4. Re-run the bootstrap demo and list-topics.sh and confirm the new topic appears.

Solution hints

• 2,000,000 / 30,000 ≈ 67 — round to a clean number like 64 or 70, showing the arithmetic in a comment rather than picking a bare number.

• moderation-events doesn’t need capping because it’s a SystemEvent type (Day 1) keyed by subjectId — moderation flags aren’t subject to the same viral power-law skew as user-generated content interactions.

• Keep the local-demo-scale ratio consistent — if the real default is ~70, a local override around 2-3 keeps the same rough proportion to the other two topics’ 12/6.

A real 3-broker Kafka cluster running in KRaft mode — no Zookeeper anywhere in the stack

streamsocial-infra: Docker Compose, health checks, and scripts, sitting alongside streamsocial-common from Day 1

A controller quorum you can actually kill a node in and watch re-elect itself

Why KRaft, not Zookeeper

For most of Kafka’s life, the cluster’s metadata — which broker leads which partition, which topics exist, which consumer groups own what — lived in a separate Zookeeper ensemble. That meant running and operating two distributed systems to get one working Kafka cluster: Kafka itself, and Zookeeper underneath it holding the metadata Kafka depended on to function at all.

KRaft (Kafka Raft) removes that second system. Kafka’s own brokers — or a designated subset of them — now run a Raft consensus protocol directly to agree on cluster metadata. One system, one thing to operate, one thing to reason about when something goes wrong. This isn’t a minor internal refactor; it’s the reason a 3-node cluster today is three Kafka containers instead of three Kafka containers plus three (or five) Zookeeper containers watching over them.

Broker roles and the controller quorum

In KRaft mode, every node runs with a process.roles setting of broker, controller, or both. A broker serves client reads and writes — the thing producers and consumers actually talk to. A controller participates in the metadata Raft quorum — deciding partition leadership, tracking which brokers are alive, and persisting the cluster’s source of truth.

For a cluster this size, every node runs both roles at once — broker,controller — which is the standard shape for development and teaching clusters. Larger production deployments typically split dedicated controller-only nodes out from the brokers serving traffic.

What matters today is the controller quorum: the set of nodes voting on metadata changes via controller.quorum.voters. With three voters, any two form a majority — which is exactly what makes today’s challenge possible. Kill one voter, including the current leader, and the remaining two keep the cluster’s metadata consistent and elect a new active controller among themselves.

Two listener ports per broker, and why that split exists

Each broker in docker-compose.yml declares two listeners: an internal one on :19092 for inter-broker traffic, and an external PLAINTEXT_HOST listener on its own host-mapped port (9092, 9093, 9094). The internal listener is never exposed to the host at all — it exists purely so brokers can replicate data and coordinate with each other over the Docker network. Every host-side client this course builds — the AdminClient in Day 3, every producer and consumer after that, this lesson’s own verification scripts — talks to a broker’s external listener instead. Collapsing these into one listener is a common shortcut that works fine for a single-broker toy setup and breaks in subtle ways the moment replication and inter-broker traffic are real, which they are here from day one.

KAFKA_AUTO_CREATE_TOPICS_ENABLE is set to false on every broker, deliberately. Without it, the very first producer or consumer to reference a topic name that doesn’t exist yet would silently create one — with whatever default partition count the broker happens to be configured with, not the number Day 3’s math actually calls for. Turning this off forces every topic through the idempotent AdminClient bootstrap Day 3 builds, which is exactly where that decision belongs.

What actually happens when you kill the leader

Today’s challenge — verify controller failover — has three parts:

1. Find the active controller. kafka-metadata-quorum describe --status reports a LeaderId among the three voters. That’s the node currently deciding metadata changes.

2. Kill it. Not a graceful shutdown — docker kill, the same as pulling power on a physical machine. This is the failure mode worth testing, because graceful shutdowns are the easy case.

3. Watch the survivors. With node availability down to two out of three, the surviving voters still hold a majority. They detect the missing leader, run a new election, and one of them becomes the new LeaderId — typically within a few seconds.

The cluster’s client-facing brokers keep answering requests through all of this, because broker availability and controller leadership are separate concerns — losing the active controller doesn’t take the cluster offline, it just changes who’s writing the metadata log.

Implementation Guide:

GitHub Link :

https://github.com/sysdr/streamsocial-java/tree/main/day02-v2-streamsocial-source/streamsocial

Build, Verify, and Demo the Kafka Cluster

The idea, in pseudo-code

cluster = 3 nodes, each: process.roles = broker + controller
quorum_voters = {1@kafka-1:19093, 2@kafka-2:19093, 3@kafka-3:19093}

start():
    for each node: format storage with shared CLUSTER_ID, join quorum
    wait until all nodes report healthy

describe_status():
    ask any broker: "who is the current controller leader?"
    -> LeaderId, CurrentVoters, LeaderEpoch

failover_demo():
    leader = describe_status().LeaderId
    kill(container_for(leader))
    poll a surviving node's describe_status() until LeaderId changes
    -> failover confirmed
    restart(container_for(leader))  # rejoins as follower

Three nodes, one shared quorum, one script that proves the quorum survives losing a member.

Step 1 — Unpack and start everything

cd streamsocial
./start.sh

Expected output: Maven dependency resolution, BUILD SUCCESS on streamsocial-common‘s tests, then all three streamsocial-kafka-N containers reporting healthy, ending with a cluster verification block. First run pulls the confluentinc/cp-kafka:7.7.1 image, so expect a longer wait the very first time.

Step 2 — Verify brokers and quorum directly

cd streamsocial-infra
./scripts/verify-cluster.sh

Expected output: an API version response from each broker, then a describe --status block. Confirm:

• CurrentVoters lists node IDs 1, 2, 3

• Exactly one LeaderId is reported

If any broker fails its health check, check its logs:

docker compose -f "$(pwd)/docker-compose.yml" logs kafka-2

A common first-run cause is insufficient Docker memory — bump Docker Desktop’s allocation to at least 4GB and retry.

Step 3 — Demo controller failover (today’s challenge)

./scripts/demo-failover.sh

Expected output: the current quorum status, a line identifying and killing the active controller’s container, then a second describe --status block with a different LeaderId than the first. The script restarts the killed container automatically at the end.

Run verify-cluster.sh again afterward — all three nodes should show healthy and back in the voter set.

Step 4 — Stop everything

cd ..
./stop.sh

Data volumes persist by default, so a later ./start.sh picks the cluster back up with the same metadata. Use ./stop.sh --wipe to delete the volumes and force a clean re-format on next start.

Success criteria

./scripts/verify-cluster.sh shows all three brokers healthy and all three node IDs as controller quorum voters. ./scripts/demo-failover.sh shows a LeaderId change after the active controller’s container is killed, and the killed node rejoining as a follower once restarted. This is a pure infrastructure lesson with no application-level behavior yet, so today’s proof lives entirely in these two scripts’ terminal output — the live dashboard doesn’t arrive until Day 4, once there’s a running service worth watching.

Working Demo Link :

Homework

Modify docker-compose.yml to run a 5-broker cluster instead of 3, updating KAFKA_CONTROLLER_QUORUM_VOTERS and adding kafka-4 and kafka-5 service definitions with their own reserved external ports.

Implementation checklist:

1. Extend the x-kafka-common anchor’s KAFKA_CONTROLLER_QUORUM_VOTERS to list all 5 node IDs on their internal :19093 controller listener.

2. Add kafka-4 and kafka-5 service blocks following the existing three as a template — new KAFKA_NODE_ID, new external host port (9095, 9096 — check these aren’t already claimed before using them elsewhere), new named volume.

3. Update scripts/start.sh‘s health-check loop to include the two new services.

4. Run the failover demo against the 5-node cluster and kill two nodes in a row — with 5 voters, a majority is 3, so the cluster should survive losing 2.

Solution hints

• The CLUSTER_ID stays the same across all 5 nodes — it identifies the cluster, not a specific node.

• Each new broker needs a unique KAFKA_NODE_ID, a unique external port, and its own volume — reusing a port or volume name collides with an existing container.

• Killing 2 out of 5 should still leave a working quorum (3 remaining voters, still a majority); killing 3 out of 5 should not — try that too and observe describe --status stall until a voter comes back, which is the concrete meaning of “quorum lost.”

• If a new port you pick for kafka-4/kafka-5 turns out to already be reserved for something later in the course, pick a different one and note the change — the reserved-ports list only helps if it’s kept current.

A shared DomainEvent sealed interface — the single root every fact about StreamSocial will ever hang off of

Three event categories under it: UserActionEvent, ContentInteractionEvent, SystemEvent

Eight concrete, immutable Java records (three, three, and two respectively) with Jakarta Bean Validation constraints baked in

The streamsocial-common Maven module — the one dependency every other module in this 60-day build will import from

No Kafka broker today. No Spring Boot app today. Today is entirely about the shape of the data before a single byte of it touches a topic — because the taxonomy you commit to on Day 1 is the one 53 more lessons of producers, consumers, and Streams topologies will all key off of.

Where this fits in StreamSocial

streamsocial-common is the only module every other module in the system depends on — the producer service (Day 4) serializes these records onto topics, the engagement consumer (Day 5) deserializes them back, and the Kafka Streams topology (Day 41) pattern-matches over them to compute trending scores. Get the taxonomy wrong here and you’re fighting schema evolution on Day 29; get it right, and Days 2 through 60 just import it.

That’s also why the Bean Validation annotations live directly on the records instead of in some separate “validation service” — the constraint that a post’s content can’t be blank is a property of what a PostCreated event is, not a rule some other layer enforces on it later.

Core concept: an event taxonomy is a contract, not a formality

Request-response systems ask “what do you want right now?” Event-driven systems ask “what just happened?” — and every downstream consumer, from Day 5’s engagement consumer to Day 44’s trending-hashtag topology, only knows what happened because the shape of that event told it precisely, with no ambiguity.

Java’s sealed keyword turns that shape into something the compiler enforces, not just something a wiki page describes. DomainEvent permits UserActionEvent, ContentInteractionEvent, SystemEvent means there is no fourth category anyone can quietly add without touching this file — and every record component (eventId, occurredAt, postId, content) is final by construction, because Java records give you immutability for free.

Architecture: from action to accepted (or rejected) fact

A user action becomes a DomainEvent in three steps: something happens, a record gets constructed with that action’s specific fields, and Jakarta Bean Validation checks every @NotNull/@NotBlank/@Size constraint before the event is allowed to be considered “real.” Nothing here talks to Kafka yet — that constructed, validated record is exactly what Day 4’s producer will hand to KafkaTemplate.send(...).

The validation step isn’t a one-shot check — it’s a small state machine every event instance passes through: Constructed → Validating → Valid or Constructed → Validating → Invalid. Both are terminal today; there’s no retry loop, because a malformed event isn’t a transient failure, it’s a bug in the caller.

Implementation Guide :

GitHub Link :

https://github.com/sysdr/streamsocial-java/tree/main/streamsocial-day01/streamsocial

Fundamentals first

An event is a fact, not a command — PostCreated says a post was created, it doesn’t ask anyone to create one. That distinction is why event-driven systems scale differently than request-response ones: nobody has to be listening at the moment the fact happens for the fact to be true.

Java gives you two tools to make that fact durable and unambiguous: records (all fields final, equals/hashCode/toString generated, no setters — a fact can’t un-happen) and sealed interfaces (a fixed, compiler-known list of what a “fact” can be).

Today’s specific pattern: sealed hierarchy + Bean Validation

Pseudo-code shape of what you’re building — not the real file, just the skeleton:

sealed interface DomainEvent permits UserActionEvent, ContentInteractionEvent, SystemEvent
    eventId(): UUID
    occurredAt(): Instant

sealed interface UserActionEvent extends DomainEvent permits PostCreated, PostDeleted, ProfileUpdated

record PostCreated(eventId, occurredAt, userId, postId, @NotBlank content) implements UserActionEvent

The validation constraints (@NotNull, @NotBlank, @Size) sit directly on the record’s constructor parameters. Jakarta’s Validator.validate(event) walks those annotations at runtime and hands back a Set<ConstraintViolation<T>> — empty means valid.

How it snaps into the system

Nothing yet — that’s the point of Day 1. streamsocial-common has zero dependencies on any other StreamSocial module; every module from Day 4 onward depends on it.

Build, test, and demo — step by step

1. Build the reactor and run the tests

cd streamsocial
mvn -pl streamsocial-common -am verify

Expected output (tail of it):

Tests run: 5, Failures: 0, Errors: 0 -- in ...DomainEventValidationTest
Tests run: 3, Failures: 0, Errors: 0 -- in ...DomainEventDescriberTest
Tests run: 8, Failures: 0, Errors: 0
BUILD SUCCESS

2. Run the one-command lifecycle script

./start.sh

Expected: prerequisite checks pass, the reactor builds, and since neither Docker infra (Day 2) nor the dashboard (Day 4) exist yet in this snapshot, both are skipped with a message rather than an error — then the terminal demo runs and prints all 8 event types plus one rejected validation.

3. Run just the demo, without rebuilding

mvn -q org.codehaus.mojo:exec-maven-plugin:3.3.0:java \
    -pl streamsocial-common \
    -Dexec.mainClass=com.streamsocial.common.event.EventTaxonomyDemo \
    -Dexec.classpathScope=test

Expected: the same 8-line taxonomy printout plus REJECTED: content must not be blank.

4. Docker note

There is no Docker step today — streamsocial-infra doesn’t exist until Day 2. start.sh already knows to skip it; you don’t need to do anything differently with or without Docker installed for this lesson.

5. Shut down cleanly

./stop.sh

Expected: Maven build output cleaned, no errors, safe to run again even if nothing was running.

Success criterion

Run ./start.sh from the repository root. You should see a build that compiles cleanly, 8 passing JUnit 5 tests, and a terminal demo that prints all eight event types by name plus one deliberately-blank PostCreated getting rejected with content must not be blank. If you can explain out loud why DomainEvent is sealed instead of just a plain interface — and what breaks on Day 44 if it weren’t — you’ve got today’s concept.

Working Demo Link :

Homework assignment

Extend the taxonomy to the full 10 event types the Day 1 challenge calls for, without breaking anything that exists:

1. Add two new leaf records: PostEdited (under UserActionEvent — a user changed a post’s content after publishing) and PostReported (under ContentInteractionEvent — a user flagged someone else’s post).

2. Give each at least one meaningful @NotNull/@NotBlank/@Size constraint beyond the shared eventId/occurredAt.

3. Update the permits clause on the relevant category interface — the compiler will now force you to handle the new type anywhere a switch or instanceof chain claims to be exhaustive.

4. Add a validation test and a DomainEventDescriber branch for each new type; run mvn -pl streamsocial-common -am verify and confirm the test count goes from 8 to 10+.

Solution hints

• PostEdited needs its own postId and the new content, following the exact same field pattern as PostCreated — copy its constraint style, don’t invent a new one.

• PostReported needs reporterId, postId, and a @NotBlank reason — resist adding an enum for report categories today, that’s more taxonomy than Day 1 needs.

• If DomainEventDescriber.describe(...) doesn’t get a new else if branch for each new type, its final throw new IllegalStateException(...) is your safety net — it should fire in a test if you forget one, not silently return a wrong description.

• Don’t touch DomainEvent, UserActionEvent‘s or ContentInteractionEvent‘s existing permits entries — only add to them.

Event-driven systems are one of the most deceptively difficult problems in backend engineering. Every non-trivial platform eventually needs to move data between services in real time, fan events out to dozens of downstream consumers, or guarantee that a payment event is processed exactly once — and the moment a system moves past a single-broker tutorial setup, the simple producer-and-consumer pattern that worked in a five-minute quickstart starts causing duplicate processing, out-of-order writes, and silent data loss in production.

This course was built to address that exact gap. It takes learners from a three-broker Kafka cluster and a first producer all the way to a secured, observable, production-hardened event-driven platform — the kind of system that underlies real-time feeds, activity streams, and event backbones at companies operating at genuine scale. Rather than treating Kafka as a message queue with extra configuration, this program treats it as a complete systems-design subject, covering producer and consumer internals, schema management, Kafka Connect, Kafka Streams, observability, and security along the way.

The material is organized as a 60-day publishing plan, with each day representing a focused lesson that builds directly on the one before it, inside a single continuously growing codebase. Java 17 and Spring Boot are the implementation language throughout — the JVM-native, production-standard way real teams actually run Kafka clients — but Kafka itself is the subject of every lesson. By the end, learners will have designed, coded, and operated a distributed, event-driven social platform called StreamSocial, comparable in shape to the real-time backbones behind large-scale social and activity-feed systems, built from first principles, one architectural decision at a time.

Why This Course Matters :

Event-driven architecture sits at the center of modern backend systems. Activity feeds, engagement and analytics pipelines, fraud and trust signals, recommendation inputs, and cross-service communication all depend on a messaging backbone that can move enormous volumes of data without losing ordering, duplicating writes, or falling over during a rebalance. As companies move from request-response APIs to event-driven architectures, a naive producer-consumer setup becomes a liability rather than a convenience, and engineers are expected to understand why.

Common challenges learners face:

Most Java developers know how to call send() on a KafkaTemplate, but far fewer understand what happens when that producer retries under network failure, or when a consumer group rebalances mid-batch. Typical pain points include:

• Duplicate or lost messages when producers retry after a transient failure

• No clear strategy for partition count, keys, or ordering guarantees at scale

• Consumers that silently drop records during a rebalance or crash mid-commit

• Schemas that break downstream consumers the moment a field changes

• Lack of production readiness — no security, no observability, no plan for connecting Kafka to the rest of the data platform

Skills addressed by the course :

The curriculum directly targets these gaps by teaching producer reliability (idempotence, transactions, acknowledgment strategies, custom partitioning), consumer resilience (manual commits, rebalance listeners, dead letter queues), schema management (JSON Schema, Avro, Protobuf, and Schema Registry compatibility modes), data integration (Kafka Connect, custom connectors, Debezium-based CDC), stream processing (the Kafka Streams DSL and the low-level Processor API), and full production operations (observability, SASL/TLS/ACL security, and a real microservices split).

Who Should Enroll

This course is designed for a range of backend-focused learners, including:

• Java and Spring Boot developers who are comfortable with the basics but have never operated Kafka in production

• Backend engineers and architects responsible for messaging infrastructure, event pipelines, or real-time data at their organization

• Students and career changers aiming to build a portfolio project that demonstrates real distributed-systems capability, not just CRUD skills

• Freelancers and consultants who need to design reliable event-driven systems for client projects

• Engineering managers and technical leads who want a structured reference for evaluating or reviewing Kafka architecture decisions

• Anyone preparing for backend or systems-design interviews, where partitioning strategy, delivery guarantees, and exactly-once semantics are frequent topics

• No prior experience with Kafka is assumed, but comfort with core Java and basic Spring Boot concepts (dependency injection, REST controllers, configuration) is expected going in.

Learning Outcomes

By the end of the course, learners will be able to:

• Explain why toy, single-broker Kafka setups fail in production and design around the failure modes

• Implement idempotent and transactional producers for exactly-once write guarantees across topics

• Design partition count, key, and custom partitioning strategy for ordering and throughput at a defined scale target

• Build resilient consumers using manual offset commits, cooperative rebalance listeners, and dead letter topics for poison-pill handling

• Apply Avro and Protobuf serialization with Confluent Schema Registry, and manage schema evolution under forward, backward, and full compatibility

• Build custom Kafka Connect source and sink connectors, chain Single Message Transformations, and stream database changes in real time with Debezium

• Build Kafka Streams topologies — stateless transformations, windowed aggregations, KTable materialized views, stream-table and table-table joins, interactive queries, and the low-level Processor API

• Instrument a Kafka-based system with Micrometer, Prometheus, and Grafana, and ship structured logs across every service

• Secure a cluster end-to-end with SASL/SCRAM authentication, least-privilege ACLs, and cluster-wide TLS

• Split a single growing codebase into independently deployable microservices that communicate purely through Kafka topics

Course Curriculum

Module 1: Foundation & Core Concepts (Days 1–10)

• Day 1: Event-Driven Architecture Fundamentals

• Day 2: Kafka Cluster Setup

• Day 3: Topics & Partitions Strategy

• Day 4: High-Volume Producer Implementation

• Day 5: Engagement Consumer Development

• Day 6: Consumer Groups & Scalability

• Day 7: Offset Management Strategies

• Day 8: Commit Strategies & Reliability

• Day 9: Dynamic Consumer Rebalancing

• Day 10: Delivery Guarantees

Module 2: Producer Reliability & Performance (Days 11–20)

• Day 11: Producer Acknowledgment Strategies

• Day 12: Retry Logic & Failure Handling

• Day 13: Idempotent Producers

• Day 14: Message Ordering & Keys

• Day 15: Custom Partitioning Logic

• Day 16: Batching & Throughput Optimization

• Day 17: Compression Strategies

• Day 18: Transactional Producers

• Day 19: Asynchronous Operations & Callbacks

• Day 20: Replication & ISR Management

Module 3: Advanced Consumer Patterns (Days 21–30)

• Day 21: Manual Partition Assignment

• Day 22: Low-Latency Consumer Optimization

• Day 23: Graceful Shutdown Patterns

• Day 24: Message Headers & Metadata

• Day 25: Error Handling & Poison Pills

• Day 26: Schema Definition with JSON Schema

• Day 27: Avro Serialization & Schema Registry

• Day 28: Protocol Buffers Integration

• Day 29: Schema Evolution Strategies

• Day 30: Log Compaction

Module 4: Data Integration & Connect (Days 31–40)

• Day 31: Kafka Connect Architecture

• Day 32: Source Connector Implementation

• Day 33: Sink Connector Development

• Day 34: Single Message Transformations

• Day 35: Distributed Connect Deployment

• Day 36: Connect Monitoring & Observability

• Day 37: Custom Connector Development

• Day 38: Change Data Capture with Debezium

• Day 39: Database Integration Patterns

• Day 40: Error Handling in Connect

Module 5: Stream Processing with Kafka Streams (Days 41–50)

• Day 41: Kafka Streams Fundamentals

• Day 42: KStream Processing

• Day 43: Stateless Transformations

• Day 44: Stateful Aggregations

• Day 45: KTable Operations

• Day 46: Stream-Table Joins

• Day 47: Table-Table Joins

• Day 48: Interactive Queries

• Day 49: Fault Tolerance & Recovery

• Day 50: Processor API

Module 6: Production Operations & Security (Days 51–60)

• Day 51: Broker Monitoring & Metrics

• Day 52: Client Metrics & Observability

• Day 53: Centralized Logging Strategy

• Day 54: Authentication with SASL

• Day 55: Authorization with ACLs

• Day 56: Encryption & TLS

• Day 57: Microservices Event Architecture

• Day 58: Schema Governance

• Day 59: Advanced Ecosystem Tools

• Day 60: System Integration & Production Readiness

Practical Applications

The concepts in this course map directly onto real engineering work. Producer and consumer reliability patterns apply to any system moving data through a message broker — not just Kafka, but any queue-backed pipeline that needs to avoid duplication and data loss. The schema-management arc in Module 3 is directly transferable to any service boundary where data contracts need to evolve without breaking consumers. The Connect and CDC patterns in Module 4 are the same techniques used to build real-time data lakes, analytics pipelines, and cross-database synchronization in production systems. The stream-processing work in Module 5 mirrors what real-time analytics and fraud-detection teams build, and the observability and security work in Module 6 is standard practice for operating any Kafka-backed platform in production.

Taken together, the completed project functions as a credible portfolio piece for backend, platform, or data infrastructure engineering roles, and the individual modules serve as standalone references that can be revisited whenever a related problem — schema evolution, exactly-once processing, connector development — comes up in professional work.

Conclusion

Event-driven systems sit at the intersection of several of the hardest problems in backend engineering — ordering, delivery guarantees, schema evolution, and operational trust — and this course was designed to teach all of them through the lens of a single, coherent, incrementally built system. Rather than presenting isolated Kafka features in the abstract, each of the 60 lessons adds a working piece to the same architecture, so that by the end, the concepts of producer reliability, consumer correctness, stream processing, and production security are not just understood but implemented.

For engineers looking to move beyond basic Kafka familiarity into genuine systems-design capability, this curriculum offers a structured, thorough, and practically grounded path. The full lesson plan and published content are available through Hands On Kafka on Substack, with new lessons rolling out according to the course schedule.