Lesson 41: DSL vs. Processor API — Why Spatial Matching Demands the Low Road
Module 4: The Matching Engine (Processor API)
The Naive Approach — And Why It Crashes at 10k Drivers/Sec
Every engineer who first encounters Kafka Streams reaches for the DSL. It’s clean, it’s fluent, it hides the plumbing. For matching a rider to nearby drivers, the naive instinct looks like this:
// DON'T DO THIS
KStream<String, RiderRequest> riders = builder.stream("rider-requests");
KTable<String, DriverLocation> drivers = builder.table("driver-locations");
riders.join(drivers, (rider, driver) -> match(rider, driver))
.to("match-results");
This code has three fatal flaws for spatial workloads.
Fatal Flaw 1: Key co-partitioning is impossible. DSL join() requires that both topics are partitioned by the same key. Our driver-locations topic uses the compound geo-partition key h3CellRes5|driverId%8 (established in Lesson 32). Our rider-requests topic is keyed by riderId. Kafka’s TopologyException will reject this topology at startup. You cannot join them with the DSL without rekeying — which means a full shuffle, doubling your network amplification.
Fatal Flaw 2: Temporal joins are not spatial joins. DSL join() with JoinWindows matches events that arrive within a time window — e.g., 5 seconds. A driver who last updated 8 seconds ago is invisible to the join, even if they’re sitting 200 meters from the rider. You need a stateful lookup, not a temporal intersection.
Fatal Flaw 3: No range scan API. Even if you rekeyed and forced co-partitioning, the DSL’s KTable abstraction gives you one operation: point lookup by exact key. K-Ring matching requires scanning all drivers in 7 H3 cells simultaneously — a lexicographic range scan. The DSL has no surface area for this. The RocksDB store is there, but the DSL puts a wall in front of it.
The result at 10k driver updates/sec: either a
TopologyExceptionon startup, or a full KTable scan per rider request — O(N) with N = total drivers in the partition. At 500 drivers/partition with 24 partitions, that’s 500 RocksDB point-lookups per rider request where 7 range scans would suffice.
