Week 2 : Build a Consumer Reliability Stack That Survives Real Traffic
What we build today
A multi-pipeline consumer platform for StreamSocial (feed, engagement, rebalance, payments).
Horizontal scaling from one worker to many without code changes.
Custom offset checkpointing for analytics recovery.
Manual commit ordering that prevents silent data loss.
Rebalance-safe state handoff and idempotent payment processing.
A live dashboard that proves every subsystem is actually moving.
Where this sits in StreamSocial
Week 1 taught ingestion and partitioning. Week 2 teaches how to consume safely at scale. In production social systems, this layer sits between Kafka and product features (feed ranking, moderation, billing, notifications).
If this layer is weak, you get duplicate charges, stuck consumers, or replay storms after deploys.
Real-world parallels: Instagram feed fanout workers, LinkedIn engagement counters, Stripe-style payment consumers, and notification dispatchers.
Architecture diagram :
