What We Will Build Today
A Connect observability layer with health scores, lag, throughput, and P0/P1 alerts (Day 36).
A custom social source connector simulating Twitter and LinkedIn ingestion with rate limits (Day 37).
Change Data Capture from PostgreSQL user profiles via Debezium event patterns (Day 38).
JDBC-driven ML training sync with conflict resolution and feature extraction (Day 39).
Production-grade error handling: DLQ, exponential backoff retry, and circuit breaker for moderation (Day 40).
One integrated dashboard on port 5080 where the Demo button regenerates live KPIs across all five days.
Where This Fits in the 60-Day System
Week 7 taught you to move data in and out with Connect. Week 8 teaches you to operate that layer: see failures before users do, stream database truth into Kafka, keep ML features consistent, and quarantine poison messages without stalling the pipeline.
Netflix’s data platform, LinkedIn’s Brooklin, and Uber’s logging pipelines all share this pattern: CDC for source-of-truth, Connect for integration, DLQ for failure isolation. StreamSocial’s profile and moderation paths are the same architecture at classroom scale.
Component Architecture
Core Concepts
1) Observability Is a Product Feature, Not an Afterthought
Connect failures rarely crash Kafka—they stall silently: lag grows, tasks flap, error rates creep. Day 36’s health score combines error rate, task availability, and offset lag into one number SREs can page on. That composite score is how platforms like Confluent Cloud surface connector risk before data drift hits dashboards.
2) Custom Connectors Encode Business Rules at the Edge
Twitter and LinkedIn impose different rate limits and payload shapes. A custom source connector encapsulates polling, backoff, and partition keys so downstream topics stay ordered per account. Day 37’s token bucket is the same primitive API gateways use—only here it protects your integration tier, not HTTP clients.
3) CDC Turns the Database Into an Event Stream
Debezium reads PostgreSQL’s logical replication log and emits change events (create, update, delete, tombstone). Profile updates become Kafka messages without application double-writes. Day 38’s tombstone teaches log compaction: the latest key wins, critical for “current user state” topics.
4) ML Pipelines Need Sync Semantics, Not Just Copy
Training data drifts when the same user is updated concurrently from mobile and web. Last-write-wins and field-level merge (max followers, union interests) prevent feature corruption. Day 39 mirrors how feature stores reconcile streaming updates with batch backfills.
