What We're Building Today
By the end of this lesson, you'll have built a complete event processing system that handles 50,000 user events per second with different reliability guarantees. You'll implement three acknowledgment modes, create a real-time monitoring dashboard, and benchmark performance trade-offs.
Main Learning Points:
Implement three acknowledgment modes (acks=0, 1, all) for different event types
Build monitoring dashboard to visualize acknowledgment behavior
Test failure scenarios and measure data loss impact
Understand durability vs performance trade-offs in production systems
The StreamSocial Challenge
Your users are posting photos, commenting, and liking content. StreamSocial processes 50,000 user events per second during peak hours. The business demands 99.9% durability for critical events like user registration, premium subscriptions, and payment confirmations, while maintaining sub-100ms response times for social interactions.
Understanding Producer Acknowledgments
What Are Acknowledgments?
When your StreamSocial app sends an event to Kafka, it needs to know: "Did my message arrive safely?" Acknowledgments (acks) are Kafka's way of saying "Got it!" - but with different levels of certainty.
acks=0 (Fire and Forget)
Your producer sends the message and immediately moves on. No waiting, no confirmation. Like posting a letter without tracking - fast but risky.
acks=1 (Leader Confirmation)
The leader broker confirms receipt before your producer continues. Like getting a delivery receipt - balanced approach.
acks=all (Full Replication)
All replica brokers must confirm before your producer gets acknowledgment. Like requiring signatures from multiple recipients - slowest but most reliable.
The Durability vs Latency Spectrum
