Week 4 : Throughput Optimization, Transactions & Replication (Days 16–20)
What We Will Build Today
A producer pipeline that can push large event volumes using batching knobs.
A compression benchmark layer to compare size reduction and CPU trade-offs.
Atomic multi-topic writes for timeline + global feed correctness.
Async callback handling for delivery observability.
Replication failover simulation to validate ISR behavior.
Where This Fits in StreamSocial
This week is the producer reliability core of the full platform. Earlier weeks define events and consumers; Week 4 makes sure events are produced fast, safely, and consistently before downstream systems consume them. If this layer is weak, every later module inherits data loss, duplicates, or latency spikes.
Core Concepts (Day 16-20)
Day 16: Batching and Throughput Tuning
Batching increases network efficiency by grouping records before send. batch_size controls how much data we send per request, linger_ms controls how long we wait to fill a batch, and buffer_memory protects producer memory limits.
Practical insight: Bigger batches improve throughput until they hurt tail latency; your SLO decides the safe window.
Day 17: Compression Strategy
Compression lowers network and storage cost but adds CPU overhead. GZIP may compress best in some datasets, while LZ4/Snappy often win on speed.
Practical insight: Measure with your real payload shape (social metadata is repetitive and compresses extremely well).
Day 18: Transactional Producers
Transactions allow atomic writes across topics. Either both timeline and global_feed get committed, or neither does.
Practical insight: Transactions are not only a correctness feature; they are a product trust feature for feed consistency.
Day 19: Async Sends + Callbacks
Non-blocking sends keep producer threads efficient under high load. Callbacks capture per-batch success/failure and latency without blocking request paths.
Practical insight: Async is not complete without callback observability; otherwise failures become invisible.
Day 20: Replication + ISR Management
Replication protects durability. If leader fails, a follower becomes leader, and min.insync.replicas enforces safe write quorum.
Practical insight: Throughput numbers without failover behavior are vanity metrics.
Component Architecture
