Lesson 42: Defining the Topology — Wiring the Matching Engine
Module 4: The Matching Engine (Processor API) Prerequisites: Lessons 1–41 | Java 21 · Kafka Streams 3.6.x · H3 Resolution 9 · RocksDB
The Naive Approach: Why KTable Joins Fail at Geo-Scale
A developer who knows the Kafka Streams DSL writes this in under five minutes:
java
KStream<String, RideRequest> requests = builder.stream("ride-requests");
KTable<String, DriverLocation> drivers = builder.table("driver-locations");
requests.join(drivers, (req, driver) -> attemptMatch(req, driver))
.to("match-results");It compiles. It passes unit tests with two brokers and ten drivers. It ships to staging. Three failure modes detonate in sequence at production load:
Failure 1 — The Partition Locality Trap. KTable joins are co-partitioned. A KTable task assigned to partition 7 sees only the drivers whose keys hashed to partition 7. In our compound key scheme (h3CellRes5|driverId%8 established in Lesson 32), the join key is the partition key — not the geographic proximity key. A ride request at the boundary of H3 cell 8528340bfffffff will miss every driver in adjacent cell 85283473fffffff that landed on a different partition. The DSL has no concept of H3 k-ring expansion; it can only join on key equality.
Failure 2 — O(N) Scan on Rebalance. The DSL materializes the KTable as a RocksDB state store with key = the message key and value = latest driver location. To find all drivers within k-ring(1) of a request, you’d need to iterate the entire store, filter by H3 cell, then check distance. At 50,000 concurrent drivers across 24 partitions, each task stores ~2,100 driver records. A full scan per ride request at 3,000 req/sec means 6.3 million RocksDB get() calls per second per task — an IOPS budget that exhausts block cache and forces SSD seeks within minutes.
Failure 3 — Rebalance Blindness. During a consumer group rebalance (as examined in Lesson 37), the KTable state is migrated between tasks. The restoration time from changelog is proportional to state size. With 50K drivers updating every 4 seconds, the changelog topic accumulates ~12,500 records/sec. A fresh task restoring from offset 0 runs blind — returning NO_DRIVER for every request — until restoration completes. EAGER protocol makes this worse; COOPERATIVE helps but doesn’t solve the root problem: the data model is wrong.
The Uber-Lite Architecture: Global State + Prefix-Keyed RocksDB
The correct invariant: any StreamThread processing any ride-request partition must see all available drivers. This eliminates co-partitioning as a viable strategy.
Kafka Streams provides
addGlobalStore()precisely for this pattern. A global state store is populated from a source topic by a dedicatedGlobalStateUpdateProcessor, replicated to every running instance of the application. Each StreamThread has a local RocksDB copy of the full driver index. No inter-task communication. No network hop for driver lookup.
The topology splits into two independent sub-graphs:
[Global Sub-topology]
driver-locations → GlobalDriverSource → GeoIndexProcessor → h3-driver-index (RocksDB)
[Matching Sub-topology]
ride-requests → RideRequestSource → MatchingProcessor ─(reads)→ h3-driver-index
│
└─► MatchResultsSink → match-resultsThe compound RocksDB key encodes H3 cell in the first 8 bytes (big-endian long) followed by 32 bytes of driverId UTF-8. This layout is not accidental. RocksDB’s useFixedLengthPrefixExtractor(8) tells the LSM tree to build bloom filters keyed on the H3 cell prefix. An existence check for drivers in a given cell requires a single bloom filter probe — O(1), independent of total driver count.
