Week 6 : Schema Evolution, Serialization & Log Compaction (Days 26–30)
What We Will Build Today
One integrated StreamSocial data contract layer spanning Days 26–30.
A JSON Schema catalog that validates profile and interaction events before they enter Kafka.
Avro serialization with measurable payload shrink vs JSON.
A Protobuf vs Avro benchmark reflecting mobile client trade-offs.
Schema evolution rules (backward, forward, full) for rolling app releases.
Log-compacted user preference state—latest settings per user key.
A dashboard where every day’s metrics render as live cards, not raw JSON dumps.
Where This Fits in the 60-Day System
Weeks 1–5 moved events reliably through Kafka. Week 6 answers the next crisis at scale: what happens when ten teams ship schemas, mobile apps lag by three versions, and a 200-byte JSON post becomes a multi-megabyte firehose?
LinkedIn’s Kafka adoption, Uber’s schema registry, and Confluent’s production guidance all converge on one idea: the log is only as safe as the contract on the wire. This week builds that contract layer for StreamSocial.
Component Architecture
Core Concepts
1) JSON Schema Is Your Gate, Not Your Transport
Validation at the edge catches bad emails, missing user_id, and malformed interaction payloads before they poison downstream consumers. At billion-event scale, one unchecked field can force fleet-wide consumer crashes.
2) Avro Wins Bandwidth; Schema Registry Wins Coordination
Avro’s binary layout plus an ID in the message header lets consumers decode without embedding full schemas. StreamSocial’s interaction stream shrinks dramatically vs JSON—CPU for encode/decode is the price you model in capacity plans.
3) Protobuf vs Avro Is a Client and Team Decision
Mobile teams often prefer Protobuf’s codegen and field numbers; data platforms often standardize on Avro + Schema Registry. Neither is universally “faster”—measure bytes and nanoseconds on your payload shape, not blog benchmarks.
