Day 60: System Integration & Production Readiness
Taking StreamSocial to Production at Scale
Production Operations & Security
Welcome to the final day of our 60-day journey. Today, we’re taking everything you’ve built and making it production-ready. This isn’t about theory anymore—this is about deploying a system that can handle 100 million daily active users without breaking a sweat.
When Netflix deploys code 4,000 times per day to production, they’re not crossing their fingers and hoping it works. When Instagram serves 500 million daily active users with 99.9% uptime, they’re not getting lucky. When Twitter processes 500 million tweets daily across multiple datacenters, they have systems in place that guarantee reliability.
Today, you’re learning those systems.
What We’re Building Today
By the end of today, you’ll have a complete production deployment system for StreamSocial that includes:
Multi-Region Deployment Architecture Deploy across three geographic regions (US-East, US-West, EU-Central) with automated failover between them.
Zero-Downtime Deployment Pipeline Deploy new code to production without a single user noticing. If something goes wrong, rollback in 8 seconds.
Disaster Recovery System When an entire datacenter goes offline, automatically failover to a standby region in under 5 minutes.
Production Capacity Planning Use mathematics to calculate exactly how much infrastructure you need. No guessing, no over-provisioning.
Real-Time Monitoring Dashboard See every request, every error, every latency spike across all regions in real-time.
Load Testing Framework Simulate 1 million concurrent users to prove your system can handle the load before launch day.
This is what separates hobbyist projects from production systems that process billions of dollars in transactions.
Production Readiness: What It Really Means
Production readiness isn’t about writing perfect code. It’s about building systems that survive real-world chaos.
Think about it: when you write a program for a class assignment, what happens if it crashes? You restart it, fix the bug, and submit it again. No big deal.
But when Stripe processes $640 billion in annual payments, they can’t just “restart and try again.” Every second of downtime costs millions of dollars. Every bug impacts thousands of businesses and millions of customers.
Production-ready systems have three characteristics:
They survive failures without human intervention. A server crashes at 3am? The system automatically detects it, routes traffic elsewhere, and pages an engineer. By the time the engineer wakes up, the problem is already contained.
They deploy new code constantly without breaking. LinkedIn releases new code every 7 minutes while serving 900 million members. Users never see downtime, never lose data, never experience errors.
They know exactly when something is wrong. Before users complain on Twitter, before the CEO asks what’s happening, the monitoring system has already alerted the team with exact details about what broke and why.
Let’s build that.
The Three Pillars of Production Systems
Pillar 1: Deployment Resilience
Production deployments fail. It’s not a question of “if” but “when.” Hard drives die. Network cables get unplugged. Someone pushes code with a subtle bug that only appears under heavy load.
The companies that stay online during these failures are the ones that planned for them.
Blue-Green Deployments
Here’s how most companies used to deploy code: turn off the servers, deploy new code, turn them back on. Downtime: 15 minutes. Frustrated users: thousands.
Here’s how modern companies do it: run two identical production environments. Blue is serving traffic right now. Green is sitting idle, waiting. When you want to deploy new code, you deploy it to Green. Test it. Warm it up. Then, in 10 seconds, switch all traffic from Blue to Green. If anything breaks, switch back to Blue in 8 seconds.
Netflix does this thousands of times per day. Zero downtime.
Let me show you the timeline:
T-30 minutes: Deploy v2.5.0 to Green environment
Pull new code from repository
Install dependencies
Run database migrations
Start all services
Green is now running the new version, but getting zero traffic
T-20 minutes: Run integration tests against Green
API health checks (do all endpoints respond?)
Database connectivity (can we read and write?)
Kafka connectivity (can we produce and consume messages?)
Load test (can we handle 50,000 requests per second?)
End-to-end user journey (can a user create a post and see it in their timeline?)
Duration: 15 minutes of thorough testing
T-5 minutes: Warm up Green environment
Preload Redis cache with hot data
Establish database connection pools (500 connections ready)
Initialize service workers
T-0 minutes: Switch traffic from Blue to Green
Update load balancer configuration
Traffic shifts: 0% → 10% → 25% → 50% → 75% → 100%
Total switch time: 10 seconds
Blue is now idle, but still running
T+5 minutes: Monitor Green’s metrics
Error rate: Should be <0.1%
P99 latency: Should be <200ms
CPU usage: Should be <75%
Memory usage: Should be stable
T+30 minutes: Decision point
If metrics are healthy: Success! Keep Green, shut down Blue
If metrics show problems: Switch traffic back to Blue in 8 seconds
This is why Netflix can deploy 4,000 times per day. They’re not brave. They’re prepared.
Canary Deployments
For really risky changes—like updating the database schema or replacing a machine learning model—even blue-green isn’t careful enough. That’s when you use canary deployments.
Instead of switching all traffic at once, you release to a tiny fraction of users first:
Release to 1% of users for 1 hour (exposes to 1 million users)
Monitor error rates, latency, CPU usage
If metrics look normal, expand to 5% for 2 hours (5 million users)
Then 25% for 4 hours (25 million users)
Finally 100% after 24 hours of validation
Uber uses this exact progression when they update their driver allocation algorithm. One bad deployment could lose millions in ride requests. Canary deployments catch bugs when they impact thousands of users, not millions.
Pillar 2: Capacity Planning with Mathematics
Guessing at capacity wastes millions. Amazon doesn’t guess how many servers they need before launching a new region. They calculate it precisely.
Here’s the math that matters.
Calculating Peak Requests Per Second
Your system won’t receive uniform traffic throughout the day. Traffic follows patterns. On social media, most posts happen during evening hours. For StreamSocial with 100 million daily active users who each post 10 times per day:
Peak RPS = (DAU × Events_Per_User) / (Seconds_Per_Day × Peak_Traffic_Ratio)
Peak RPS = (100,000,000 × 10) / (86,400 × 0.2)
Peak RPS = 1,000,000,000 / 17,280
Peak RPS = 57,870 requests per second
Why divide by 0.2? Because typically 80% of traffic happens in 20% of the day. If you plan for average traffic, you’ll collapse during peak hours.
Calculating Kafka Capacity
Each Kafka partition can handle about 10,000 messages per second before performance degrades. For 57,870 RPS:
Minimum Partitions = 57,870 / 10,000 = 6 partitions
But we need replication for fault tolerance. With replication factor of 3:
Total Partitions = 6 × 3 = 18 partitions
Now we can lose 2 out of 3 replicas and still keep running.
Calculating Broker Requirements
Each partition is stored on a broker. With 18 partitions spread across 6 brokers in 3 availability zones:
Brokers per AZ = 6 / 3 = 2 brokers per availability zone
Now we can lose an entire datacenter and still operate.
Memory Requirements
Each consumer group maintains offset metadata. With 50 consumer groups and 18 partitions:
Offset Commits = 50 groups × 18 partitions = 900 offset commits/sec during rebalancing
Memory for Metadata = 2 GB per broker
Total Memory = 6 brokers × 2 GB = 12 GB just for metadata
Storage Calculation
For 1 billion events per day with 2KB average message size and 70% compression:
Daily Storage = (1,000,000,000 × 2 KB × 0.3) / 1024³
Daily Storage = 558 GB per day compressed
For 7-day retention:
Total Storage = 558 GB × 7 = 3.9 TB
Cost Estimation
AWS pricing (approximate):
r5.2xlarge instances (8 vCPU, 64GB RAM): $300/month each
6 brokers needed: $1,800/month
Storage (4TB EBS gp3): $400/month
Network bandwidth (180 Gbps peak): $18,000/month
Total infrastructure cost: $20,200/month Cost per million events: $0.0061
Instagram runs on similar architecture serving 500 million users daily.
Pillar 3: Disaster Recovery Without Hope
Hope is not a strategy. When AWS us-east-1 went down in 2017, companies with multi-region architectures stayed online. Those without lost millions per hour.
Your Kafka cluster already replicates data across brokers for fault tolerance. But what if the entire datacenter catches fire? Or a backhoe cuts through the fiber optic cables? Or a hurricane knocks out power for 48 hours?
That’s when disaster recovery saves you.
Multi-Region Replication Architecture
Your data doesn’t live in just one place. It lives in three:
Primary Region: US-East
Handles 100% of write traffic
6 Kafka brokers processing 150K events/second
MirrorMaker 2 continuously streams topics to other regions
Monitoring detects failures in 15 seconds
Standby Region: US-West
Consumes replicated events in real-time
Maintains PostgreSQL replica with <2 second lag
Fully warmed up and ready to take over
Can be promoted to primary in 5 minutes
Read Replica: EU-Central
Serves European users’ read traffic
Reduces latency from 140ms to 18ms for EU users
Can be promoted if both US regions fail
Handles 10% of global read traffic
The Replication Flow
When a user in New York posts a photo:
Post Service publishes event to Kafka in US-East
Event replicates to 2 other US-East brokers (in-region HA): 30ms
MirrorMaker 2 consumes from US-East: immediate
MirrorMaker 2 publishes to US-West and EU-Central: 1,500ms
All three regions now have the event: total lag 1.5 seconds
If US-East disappears, US-West is only 1.5 seconds behind. That’s your Recovery Point Objective (RPO): 1.5 seconds of potential data loss.
Automated Failover Procedure
When the primary region fails, here’s what happens automatically:
Detection: 15 seconds Health check endpoint fails three consecutive times with 5 second intervals between attempts.
DNS Update: 30 seconds
Route53 updates DNS records to point streamsocial.com → us-west-1. TTL of 60 seconds ensures global propagation.
Database Promotion: 90 seconds PostgreSQL replica in US-West promotes itself to master. All writes now go there.
Kafka Consumer Rebalance: 20 seconds Consumer groups detect broker failures and rebalance to consume from US-West brokers instead.
Smoke Tests: 120 seconds Automated tests verify all services responding, database writes working, Kafka producing and consuming correctly.
Traffic Redirect: 45 seconds Load balancers update rules to route traffic to US-West. CDN cache invalidates and refreshes.
Total Recovery Time Objective (RTO): 5 minutes from failure to full restoration.
During those 5 minutes, users might see a “Service Temporarily Unavailable” page. But their data is safe, and the system recovers automatically without human intervention.
Data Loss Estimation
At 150,000 requests per second with 1.5 second replication lag:
Potential Lost Events = 150,000 × 1.5 = 225,000 events
These are events that were acknowledged in US-East but hadn’t replicated to US-West yet when US-East went offline.
Mitigation strategies:
Clients automatically retry failed requests
Failed events are logged for manual replay
Critical financial transactions use synchronous cross-region replication
Netflix maintains three full backups in separate regions and can restore the entire platform in 6 hours if all regions somehow fail simultaneously.
StreamSocial Production Architecture
Let’s walk through the complete production architecture we’ve built.
Geographic Distribution
Three regions, each with a specific role:
US-East (Primary Region)
Handles 100% of write traffic
Serves North American read traffic
6 Kafka brokers: 2 in each of 3 availability zones
Each broker handles 50,000 events/second = 300K events/sec total capacity
12 microservices running: Post Service, User Service, Timeline Service, Analytics Service, Notification Service, Search Service, Cache Service, Metrics Service, Audit Service, ML Service, Gateway Service, Auth Service
PostgreSQL multi-master for user profiles
Redis cluster with 10GB total cache
Elasticsearch cluster for real-time search across 500 million posts
US-West (Hot Standby)
Receives replicated data from US-East with 1.5 second lag
Fully warmed up: connection pools established, caches populated
Can take over as primary in <5 minutes
Identical infrastructure to US-East, just waiting
EU-Central (Read Replica)
Serves European users’ read requests
Reduces latency: 18ms local vs 140ms cross-Atlantic
Handles 10-15% of global read traffic
Can be promoted to primary if both US regions fail
Data Flow Through the System
Let’s trace what happens when a user in New York posts a photo. This entire flow completes in under 200 milliseconds.
Step 1: Edge Layer (5ms) Request hits CloudFront CDN at edge location in New York. CDN routes to nearest region: US-East.
Step 2: API Gateway (3ms) Load balancer uses round-robin to distribute request to one of 20 running Post Service instances.
Step 3: Post Service (89ms total)
Validates request: 4ms
Generates unique post ID using timestamp + user ID: 1ms
Publishes to
posts-createdtopic in Kafka: 71msInserts into PostgreSQL: 13ms
Step 4: Kafka Broker (71ms breakdown)
Batch write to leader: 8ms
Replicate to follower in AZ-B: 31ms
Replicate to follower in AZ-C: 32ms
Acknowledge back to producer: immediate
MirrorMaker 2 streams to US-West and EU-Central: async (1,500ms, but producer doesn’t wait)
Step 5: Consumer Groups Process Event (120ms total, parallel) Eight different services consume the event simultaneously:
Timeline Service: Updates 500 follower timelines in 120ms using batch writes
Analytics Service: Increments hourly post counter in memory, flushes every 10 seconds
Search Indexer: Adds post to Elasticsearch within 2 seconds (async)
ML Service: Analyzes content for moderation, flags if needed
Notification Service: Triggers push notifications to followers’ phones
Metrics Collector: Updates Prometheus counters for monitoring
Audit Logger: Archives event to S3 for compliance (90-day retention)
Cache Warmer: Invalidates relevant Redis keys for this user’s followers
Step 6: Response (3ms) API Gateway returns success response to client: 200 OK with post ID.
From user click to database persistence: 187 milliseconds on average. Under 200ms at P99.
If US-East fails mid-operation, the user’s next retry automatically routes to US-West, which already has the replicated data (only 1.5 seconds behind).
Infrastructure Capacity: The Numbers
Let’s break down the exact infrastructure requirements for 100 million daily active users.
Throughput Requirements
Peak posts per second: 57,870 (during evening hours in each timezone)
Timeline fanout per post: Each post updates 500 follower timelines on average Timeline updates per second: 57,870 × 500 = 28,935,000 timeline updates/second
Kafka throughput needed:
Ingress: 2.3 GB/second (posts coming in)
Egress: 18.4 GB/second (8 consumer groups reading)
Storage growth: 480 GB/day compressed
With 3-year retention: 525 TB total storage needed
Network I/O: 180 Gbps during peak hours Infrastructure: Requires 2× 100 Gbps network links per datacenter for redundancy
Calculating Exact Costs
Infrastructure on AWS:
Compute:
6 Kafka brokers (r5.2xlarge): $1,800/month
20 microservice instances (c5.2xlarge): $6,000/month
3 PostgreSQL instances (db.r5.2xlarge): $3,600/month
3 Redis instances (cache.r5.2xlarge): $1,800/month
Storage:
Kafka storage (4TB EBS gp3): $400/month
PostgreSQL storage (2TB EBS gp3): $200/month
S3 archive (100TB): $2,300/month
Network:
Data transfer out (200TB/month): $18,000/month
Total monthly cost: $34,100 per region Three regions: $102,300/month Annual: $1.2 million
Cost per daily active user: $1.023/month Cost per event: $0.0031
For comparison, Instagram’s infrastructure costs are estimated at $2-3 per user per year. We’re in the right ballpark.
Zero-Downtime Deployment Strategy
Traditional deployments cause 5-15 minute outages. Unacceptable when users expect instant responses 24/7. Here’s how we achieve zero-downtime deployments.
Blue-Green Deployment in Detail
Think of Blue and Green as two identical production environments. Blue is serving traffic right now. Green is idle, waiting to be useful.
Setup Phase (One-Time)
Create two identical environments:
Blue Environment:
Current production running v2.4.1
Serving 100% of user traffic
All services running and healthy
Green Environment:
Identical infrastructure to Blue
Currently receiving zero traffic
Ready to deploy new versions
Deployment Phase (Every Release)
T-30 minutes: Deploy to Green
# Pull latest code
git pull origin release/v2.5.0
# Deploy to Green environment
kubectl set image deployment/post-service \
post-service=streamsocial/post-service:v2.5.0 \
--namespace=green
# Wait for rollout to complete
kubectl rollout status deployment/post-service --namespace=green
Green is now running v2.5.0, but getting zero traffic. Blue continues serving all users with v2.4.1.
T-20 minutes: Integration Tests
Run comprehensive tests against Green:
# API health checks
curl https://green.streamsocial.com/health
# Expected: 200 OK, all services responding
# Database connectivity test
curl -X POST https://green.streamsocial.com/test/db
# Expected: Can read and write successfully
# Kafka connectivity test
curl -X POST https://green.streamsocial.com/test/kafka
# Expected: Can produce and consume messages
# Load test: 50K RPS for 5 minutes
locust -f load_tests/main.py \
--host=https://green.streamsocial.com \
--users=50000 \
--spawn-rate=1000 \
--run-time=5m
# Expected: P99 latency <200ms, 0% errors
# End-to-end user journey
./tests/e2e/user_journey.sh green
# Expected: User can create post, see in timeline, receive notification
If any test fails, abort deployment. Keep Blue running. Try again tomorrow after fixing bugs.
Duration: 15 minutes of thorough testing.
T-5 minutes: Warm Up Green
Don’t switch cold traffic to a cold server. Warm it up first.
# Preload Redis cache with hot data
redis-cli -h green-cache.streamsocial.com \
--pipe < cache_warmup_data.txt
# Loads: user sessions, trending posts, popular profiles
# Establish database connection pools
curl -X POST https://green.streamsocial.com/admin/warmup-pools
# Opens 500 connections to each database
# Prevents connection storms when traffic hits
# Prime service workers
for i in {1..1000}; do
curl -s https://green.streamsocial.com/health > /dev/null
done
# Wakes up all worker processes
# JIT compiles hot paths
Green is now fully warmed and ready.
T-0 minutes: Switch Traffic
The critical moment. We’re switching 100% of traffic from Blue to Green.
# Update load balancer weights gradually
for weight in 0 10 25 50 75 90 100; do
aws elbv2 modify-target-group \
--target-group-arn $BLUE_TG_ARN \
--weight $((100 - weight))
aws elbv2 modify-target-group \
--target-group-arn $GREEN_TG_ARN \
--weight $weight
echo "Traffic: Blue=${$((100-weight))}%, Green=${weight}%"
sleep 1
done
Traffic shifts smoothly over 10 seconds. Users don’t notice the transition.
Blue is now idle, receiving zero traffic. But we keep it running.
T+5 minutes: Monitor Metrics
Watch Green carefully. Is it handling traffic properly?
# Check error rate
prometheus query 'rate(http_requests_total{status=~"5.."}[5m])' | \
jq '.data.result[0].value[1]'
# Expected: <0.1% (less than 1 error per 1000 requests)
# Check P99 latency
prometheus query 'histogram_quantile(0.99, http_request_duration_seconds)' | \
jq '.data.result[0].value[1]'
# Expected: <0.2 seconds (200ms)
# Check CPU usage
prometheus query 'avg(rate(cpu_usage_seconds_total[5m]))' | \
jq '.data.result[0].value[1]'
# Expected: <75% (healthy headroom for traffic spikes)
# Check memory usage
prometheus query 'avg(memory_usage_bytes / memory_total_bytes)' | \
jq '.data.result[0].value[1]'
# Expected: Stable (not growing over time)
If any metric looks unhealthy, we have our rollback option ready.
T+30 minutes: Decision Point
After 30 minutes of stable operation, make the call:
If metrics are healthy:
# Success! Drain Blue environment
kubectl scale deployment --all --replicas=0 --namespace=blue
# Keep Blue infrastructure allocated for 24 hours
# Quick rollback option if issues discovered later
If metrics show problems:
# Rollback: Switch traffic back to Blue
for weight in 100 90 75 50 25 10 0; do
aws elbv2 modify-target-group \
--target-group-arn $GREEN_TG_ARN \
--weight $weight
aws elbv2 modify-target-group \
--target-group-arn $BLUE_TG_ARN \
--weight $((100 - weight))
sleep 1
done
# Total rollback time: 8 seconds
# Users experienced the bad version for only 30 minutes
This is how Netflix deploys 4,000 times per day. Each deployment follows this exact pattern.
Canary Deployment for High-Risk Changes
For really scary changes—like database schema migrations or ML model replacements—blue-green isn’t careful enough. Use canary deployments instead.
Release to a tiny percentage of users first, then gradually expand:
# Hour 0: 1% of users get new version
aws elbv2 modify-target-group --weight-blue=99 --weight-green=1
# Monitor for 1 hour
# 1% of 100M users = 1M users exposed to new code
# Large enough sample to catch most bugs
# Small enough that damage is limited
# Hour 1: Expand to 5% if metrics healthy
aws elbv2 modify-target-group --weight-blue=95 --weight-green=5
# Hour 3: Expand to 25%
aws elbv2 modify-target-group --weight-blue=75 --weight-green=25
# Hour 7: Expand to 100%
aws elbv2 modify-target-group --weight-blue=0 --weight-green=100
Uber uses this exact progression for driver allocation algorithm updates. One bad deployment could lose millions in ride requests. Canary deployments catch bugs when they impact thousands, not millions.
Disaster Recovery Implementation
GitHub Link:-
https://github.com/sysdr/streamscial-p/tree/main/day60Now let’s handle the really scary scenario: an entire datacenter going offline.
Failure Detection System
You can’t fix what you don’t detect. The monitoring system continuously checks health.
def detect_region_failure(region):
consecutive_failures = 0
while consecutive_failures < 3:
try:
response = requests.get(f"https://{region}/health", timeout=5)
if response.status_code == 200:
return False # Region is healthy
except:
pass
consecutive_failures += 1
time.sleep(5) # Wait 5 seconds between attempts
# Three consecutive failures = region is down
logger.critical(f"Region {region} is DOWN")
return True
Detection time: 15 seconds (3 attempts × 5 seconds each).
Automated Failover Steps
When US-East fails, here’s the automated procedure:
Step 1: DNS Update (30 seconds)
# Update Route53 DNS records
aws route53 change-resource-record-sets \
--hosted-zone-id Z123456 \
--change-batch '{
"Changes": [{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "streamsocial.com",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "Z789012",
"DNSName": "us-west-1.streamsocial.com",
"EvaluateTargetHealth": true
}
}
}]
}'
# TTL of 60 seconds ensures global propagation
# Users automatically redirected to US-West
Step 2: Database Promotion (90 seconds)
# Promote PostgreSQL replica to master in US-West
aws rds promote-read-replica \
--db-instance-identifier streamsocial-uswest-replica
# Wait for promotion to complete
aws rds wait db-instance-available \
--db-instance-identifier streamsocial-uswest-replica
# Update application config to point to new master
kubectl set env deployment/post-service \
DATABASE_HOST=uswest.streamsocial.com
Step 3: Kafka Consumer Groups (20 seconds)
Consumer groups automatically detect broker failures and rebalance. But we can force it:
# Trigger consumer group rebalance
kafka-consumer-groups \
--bootstrap-server us-west-1.streamsocial.com:9092 \
--group timeline-processor \
--reset-offsets --to-latest --execute
# Consumers now reading from US-West brokers
# Resume from last committed offset
Step 4: Smoke Tests (120 seconds)
Verify the failover worked:
# Test API endpoints
curl https://streamsocial.com/health
# Expected: 200 OK from US-West
# Test database writes
curl -X POST https://streamsocial.com/api/posts \
-d '{"user_id": "test", "content": "Failover test"}'
# Expected: 201 Created
# Test Kafka
kafka-console-producer \
--bootstrap-server us-west-1.streamsocial.com:9092 \
--topic test
# Expected: Can produce messages
kafka-console-consumer \
--bootstrap-server us-west-1.streamsocial.com:9092 \
--topic test --from-beginning
# Expected: Can consume messages
# Test end-to-end flow
./tests/smoke/full_user_journey.sh
# Expected: User can create post and see in timeline
Step 5: Traffic Redirect (45 seconds)
# Update load balancer rules
aws elbv2 modify-rule \
--rule-arn $RULE_ARN \
--conditions Field=host-header,Values=streamsocial.com \
--actions Type=forward,TargetGroupArn=$USWEST_TG_ARN
# Invalidate CDN cache
aws cloudfront create-invalidation \
--distribution-id $DIST_ID \
--paths "/*"
# CDN will fetch fresh content from US-West
Total failover time: 305 seconds (5 minutes and 5 seconds).
Users experienced downtime: 5 minutes. Data lost: ~1.5 seconds worth of events (about 225,000 events at 150K RPS).
Those lost events are logged and can be replayed manually. Critical financial transactions use synchronous replication so they’re never lost.
Testing Disaster Recovery
You can’t wait until a real disaster to find out if your DR plan works. Run drills regularly:
# Simulate US-East failure
./scripts/simulate_region_failure.sh us-east-1
# Monitor automated failover
tail -f /var/log/failover/events.log
# Expected timeline:
# T+0:00 - Failure detected
# T+0:15 - Failover initiated
# T+0:45 - DNS updated
# T+2:15 - Database promoted
# T+2:35 - Kafka rebalanced
# T+4:35 - Smoke tests passed
# T+5:20 - Traffic redirected
# T+5:20 - Failover complete
Run this drill monthly. When a real disaster happens, you’ll be ready.
Production Monitoring and Observability
You can’t fix what you can’t see. Production monitoring is your window into the system’s soul.
The Four Golden Signals
Google’s SRE team identified four metrics that matter most:
Latency: How Long Do Requests Take?
Don’t just track average latency. Averages lie. If 99% of requests take 50ms but 1% take 5 seconds, your average is still 95ms. Sounds fine, right? But 1% of 150,000 requests per second is 1,500 users per second getting a terrible experience.
Track percentiles instead:
P50 (median): 50% of requests faster than this
P95: 95% of requests faster than this
P99: 99% of requests faster than this
P99.9: 99.9% of requests faster than this
For StreamSocial, our targets:
P50: <50ms
P99: <200ms
P99.9: <500ms
A single slow request doesn’t matter. But if P99 degrades, thousands of users per second are unhappy.
Traffic: How Many Requests Are We Handling?
Track requests per second across all endpoints. Know your baseline: 50K RPS is normal, 150K RPS during peak hours. Anything above 200K RPS might be a DDoS attack.
Also track:
Events per second through Kafka
WebSocket connections (for real-time notifications)
Database queries per second
Cache operations per second
Errors: What’s Breaking?
Track two types of errors separately:
HTTP 5xx errors: Your fault. Server crashed, database timeout, Kafka unavailable. These are bugs you need to fix.
HTTP 4xx errors: Client fault. Invalid input, missing authentication, resource not found. These might indicate API confusion or attempted attacks.
Set alerts: If error rate exceeds 0.1%, page the on-call engineer immediately.
Saturation: How Close Are We to Capacity?
Track resource utilization:
CPU usage: Alert at 70%, critical at 85%
Memory: Alert at 80%, OOM kills happen at 95%
Disk I/O: Alert at 80% IOPS, latency degrades badly beyond this
Network: Alert at 70% bandwidth, packet loss starts at 85%
Saturation is a leading indicator. It predicts errors before they happen. If CPU hits 85%, you have 5-15 minutes before requests start timing out. That’s your window to auto-scale or shed load.
Distributed Tracing: Following a Request’s Journey
When an API call touches 12 microservices, how do you debug slowness? Distributed tracing shows you the complete path.
Example trace for POST /api/posts:
POST /api/posts (total: 187ms)
├─ API Gateway: 3ms
├─ Auth Service: 12ms
│ └─ Redis cache hit: 2ms
├─ Post Service: 89ms
│ ├─ Schema validation: 4ms
│ ├─ Kafka publish: 71ms ← SLOWDOWN HERE
│ └─ Database insert: 14ms
└─ Timeline Service: 83ms
├─ Fetch followers: 31ms
└─ Fanout to 500 timelines: 52ms
You immediately see Kafka publish is slow (normally 5ms, now 71ms). Drill down: broker disk I/O is saturated at 95%. Solution: add more brokers or reduce replication traffic.
Without distributed tracing, you’d waste hours trying to find the slow component.
Load Testing: Validating Capacity Before Launch
Never deploy to production without load testing. When Disney+ launched, they expected 10 million signups on day one. They got 10 million in 8 hours and crashed completely. Load testing would’ve caught this.
Test Scenarios We Need
Scenario 1: Normal Load (Baseline)
Target: 50,000 RPS sustained for 1 hour
Purpose: Validate baseline performance
Success Criteria: P99 <200ms, 0% errors, CPU <60%
Scenario 2: Peak Load (Daily Peak)
Target: 150,000 RPS sustained for 30 minutes
Purpose: Simulate daily peak traffic
Success Criteria: P99 <300ms, <0.01% errors, CPU <75%
Scenario 3: Spike Load (Viral Event)
Target: Ramp from 50K to 300K RPS over 5 minutes, sustain for 10 minutes
Purpose: Test auto-scaling and elasticity
Success Criteria: Graceful degradation, no cascading failures
Scenario 4: Endurance Test (Memory Leaks)
Target: 75,000 RPS sustained for 24 hours
Purpose: Detect memory leaks and resource exhaustion
Success Criteria: Flat memory usage, no performance degradation
Scenario 5: Chaos Test (Fault Tolerance)
Target: Kill random broker during peak load, partition network between regions, fill disk to 95%
Purpose: Validate fault tolerance
Success Criteria: System auto-recovers, no user-visible errors
Load Testing Tools
Use Locust or k6 for load generation. For 300K RPS, you’ll need about 50 load-testing instances generating 6K RPS each.
Example k6 test script:
import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
stages: [
{ duration: '2m', target: 50000 }, // Ramp up to 50K
{ duration: '5m', target: 50000 }, // Stay at 50K
{ duration: '2m', target: 150000 }, // Ramp to peak
{ duration: '5m', target: 150000 }, // Stay at peak
{ duration: '2m', target: 0 }, // Ramp down
],
};
export default function() {
const payload = JSON.stringify({
user_id: `user_${__VU}`,
content: `Test post from VU ${__VU} at ${Date.now()}`
});
const res = http.post('https://streamsocial.com/api/posts', payload, {
headers: { 'Content-Type': 'application/json' },
});
check(res, {
'status is 201': (r) => r.status === 201,
'response time < 200ms': (r) => r.timings.duration < 200,
});
sleep(1);
}
Run the test:
k6 run --vus 50000 --duration 20m load_test.js
Interpreting Results
Good results look like this:
✓ status is 201..................: 99.97%
✓ response time < 200ms..........: 99.45%
http_req_duration.............avg=153ms p(95)=287ms p(99)=412ms
http_reqs.....................149,876/s
99.97% success rate, P99 latency 412ms, actual throughput 149,876 RPS. Close to our 150K target. System is ready.
Bad results look like this:
✗ status is 201..................: 94.12%
✗ response time < 200ms..........: 67.34%
http_req_duration.............avg=847ms p(95)=3.2s p(99)=8.9s
http_reqs.....................98,234/s
Only 94% success rate (6% errors!), P99 latency 8.9 seconds, actual throughput only 98K RPS. System is not ready. Need to investigate why it’s failing.
Production Readiness Checklist
Before you flip the switch to production, validate every single item on this checklist:
Infrastructure
[ ] Multi-region deployment active (3+ regions)
[ ] Auto-scaling configured (2-50 instances per service)
[ ] Load balancers healthy in all regions
[ ] DNS failover tested and working
[ ] SSL certificates valid for 90+ days
[ ] CDN configured with proper cache rules
Data Layer
[ ] Database replication lag <2 seconds
[ ] Kafka replication factor = 3 (can lose 2 brokers)
[ ] Backup restoration tested successfully
[ ] Disaster recovery drill completed (RTO validated)
[ ] Data retention policies configured
[ ] Encryption at rest enabled
Application
[ ] All services have health check endpoints
[ ] Circuit breakers configured (fail fast, don’t cascade)
[ ] Rate limiting active (1000 requests per minute per user)
[ ] Request timeouts set (default 30 seconds)
[ ] Graceful shutdown implemented (drain connections before kill)
[ ] Feature flags ready (enable/disable features without deployment)
Monitoring
[ ] Metrics dashboard created for all services
[ ] Alerts configured with proper thresholds
[ ] On-call rotation scheduled
[ ] Runbooks written for common incidents
[ ] Distributed tracing working end-to-end
[ ] Log aggregation capturing all services
Security
[ ] Authentication on all endpoints
[ ] HTTPS only (HTTP redirects to HTTPS)
[ ] API rate limiting per user
[ ] Input validation on all fields
[ ] SQL injection protection
[ ] XSS protection headers
[ ] Secrets in vault, not in code
Performance
[ ] Load testing passed at 2x expected peak
[ ] Caching strategy validated (90%+ hit rate)
[ ] Database indexes on all query patterns
[ ] CDN cache hit rate >85%
[ ] API latency P99 <200ms
[ ] No N+1 query problems
Compliance
[ ] GDPR data export/delete implemented
[ ] Audit logs retained for required period
[ ] Privacy policy reviewed
[ ] Terms of service updated
[ ] Data processing agreement signed
[ ] Security audit completed
Skip even one item and you’ll regret it at 3am when you’re paged for an outage.
Real-World Production Patterns
These patterns are battle-tested by companies handling billions of requests daily.
Pattern 1: The Bulkhead
Don’t let one failing component sink the entire ship. Netflix isolates each microservice with dedicated thread pools. If the recommendation service is slow, it doesn’t block video streaming.
Implementation: Configure separate consumer groups for critical vs non-critical processing. Timeline updates are critical—use dedicated Kafka brokers. Analytics are nice-to-have—they can share resources with other batch jobs.
Pattern 2: The Circuit Breaker
When a downstream service fails, stop calling it immediately. Otherwise you waste resources making requests that are guaranteed to fail.
After 50 consecutive failures, open the circuit for 60 seconds. Try again. If still failing, wait 120 seconds. Back off exponentially up to 30 minutes.
Spotify uses this to handle backend failures gracefully. When their personalized playlist service fails, users still get generic playlists instead of errors.
Pattern 3: Backpressure
When consumers are falling behind, slow down producers before Kafka runs out of disk space.
When consumer lag exceeds 1 million messages, publish a backpressure event. Producers see it and throttle from 10K messages/sec to 5K messages/sec. Consumers catch up in 3 minutes, then producers return to normal speed.
LinkedIn pioneered this for their activity stream processing. During traffic spikes, they gracefully degrade non-critical features rather than crashing completely.
Performance Optimization: The Final 20%
You’ve built a working system. Now make it fast.
Optimization 1: Batch Processing
Sending 1 million database inserts individually takes 16 minutes (each insert has network overhead). Batching 1,000 inserts per transaction takes 90 seconds. Same data, 10× faster.
Apply this to Kafka too. Producer sending messages individually: 10K messages/sec. Producer batching 100 messages every 10ms: 250K messages/sec. Same hardware, 25× throughput.
Optimization 2: Make Everything Async
Blocking I/O wastes threads. With 100 concurrent requests using synchronous calls, you need 100 threads consuming 1.2GB RAM. With async/await, you need 1 thread consuming 12MB RAM. Same throughput, 100× less memory.
Update the timeline synchronously—user waits. Send notification asynchronously—user doesn’t wait. Saves 80ms per request.
Optimization 3: Multiple Cache Layers
L1 Cache: In-process cache with 10ms expiry for extremely hot data (current user’s session)
L2 Cache: Redis with 100ms expiry for popular content (trending posts)
L3 Cache: CDN with 1 hour expiry for static assets (profile pictures)
With this setup, 99% of requests never touch the database. Query performance becomes irrelevant when cache hit rate is 99.2%.
Optimization 4: Database Read Replicas
For every write operation, there are typically 10 read operations on social media platforms. Write to the master database, read from replicas.
One master handles 10K writes/sec. Five read replicas each handle 50K reads/sec = 250K total reads/sec. You’ve just scaled read capacity 25×.
Add a replica in each geographic region. EU users read from EU replica: 18ms latency instead of 120ms cross-Atlantic.
Optimization 5: Compression
Kafka messages compress 5:1 on average (JSON compresses very well). Throughput without compression: 10 GB/sec. With compression: 2 GB/sec. That’s 80% less network bandwidth, 80% lower costs.
Use Snappy for good compression with low CPU overhead. Use zstd if you need better compression and can afford the CPU cost.
Implementation Walkthrough
Now let’s build this system step by step.
Setting Up the Development Environment
Create the project structure:
mkdir -p day60_production_readiness/{src,tests,config,monitoring,scripts,logs,data}
cd day60_production_readiness
Your directories:
src/- Production application codetests/- Test suiteconfig/- Multi-region configurationmonitoring/- Metrics and dashboardsscripts/- Automation scriptslogs/- Application logsdata/- Persistent storage
Create Python virtual environment:
python3.11 -m venv venv
source venv/bin/activate
Install dependencies:
cat > requirements.txt << 'EOF'
kafka-python==2.0.2
confluent-kafka==2.3.0
fastapi==0.109.0
uvicorn[standard]==0.27.0
pydantic==2.5.3
pytest==7.4.4
psutil==5.9.8
prometheus-client==0.19.0
redis==5.0.1
psycopg2-binary==2.9.9
EOF
pip install -r requirements.txt
Region Configuration Files
Create configuration for each region:
mkdir -p config/regions
# US-East configuration
cat > config/regions/us_east.json << 'EOF'
{
"region_name": "us-east-1",
"role": "primary",
"kafka_brokers": ["localhost:9092", "localhost:9093", "localhost:9094"],
"capacity": {
"max_rps": 150000,
"brokers": 6,
"partitions": 18
}
}
EOF
This defines our primary region with exact capacity specifications.
Building the Circuit Breaker
The circuit breaker prevents cascading failures:
class CircuitBreaker:
def __init__(self, failure_threshold=5, timeout=60):
self.failure_count = 0
self.failure_threshold = failure_threshold
self.timeout = timeout
self.last_failure_time = None
self.state = "CLOSED" # CLOSED, OPEN, HALF_OPEN
def call(self, func, *args, **kwargs):
if self.state == "OPEN":
if time.time() - self.last_failure_time > self.timeout:
self.state = "HALF_OPEN"
else:
raise Exception("Circuit breaker is OPEN")
try:
result = func(*args, **kwargs)
if self.state == "HALF_OPEN":
self.state = "CLOSED"
self.failure_count = 0
return result
except Exception as e:
self.failure_count += 1
self.last_failure_time = time.time()
if self.failure_count >= self.failure_threshold:
self.state = "OPEN"
raise
After 5 failures, circuit opens for 60 seconds. This gives the downstream service time to recover.
Creating the Production Monitoring Dashboard
The dashboard uses FastAPI with WebSockets for real-time updates:
from fastapi import FastAPI, WebSocket
import psutil
import asyncio
app = FastAPI()
class MetricsCollector:
def __init__(self):
self.regions = {
"us-east-1": {"role": "primary", "traffic_percent": 100},
"us-west-1": {"role": "standby", "traffic_percent": 0},
"eu-central-1": {"role": "read_replica", "traffic_percent": 0}
}
def collect_metrics(self):
cpu_percent = psutil.cpu_percent(interval=0.1)
memory = psutil.virtual_memory()
# Collect metrics for each region
for region_name, region in self.regions.items():
region["cpu"] = round(cpu_percent * random.uniform(0.9, 1.1), 1)
region["memory"] = round(memory.percent * random.uniform(0.9, 1.1), 1)
# Additional metrics...
return {
"timestamp": datetime.utcnow().isoformat(),
"regions": self.regions
}
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
while True:
metrics = collector.collect_metrics()
await websocket.send_json(metrics)
await asyncio.sleep(1) # Update every second
The dashboard updates every second with live metrics from all regions.
Running the Complete Implementation
Everything above has been packaged into an automated script. Run it:
# Make script executable
chmod +x day60_implementation.sh
# Run complete setup
./day60_implementation.sh
# Expected output:
# [1/12] Creating virtual environment...
# [2/12] Installing dependencies...
# [3/12] Creating configuration files...
# ...
# [12/12] Running tests...
# ✓ All tests passed
# ✓ Day 60 Implementation Complete!
This creates the complete production system including all services, deployment tools, disaster recovery, monitoring, and testing.
Building, Testing, and Demonstration
Building the System
After running the implementation script:
cd day60_production_readiness
source venv/bin/activate
# Run the build script
bash scripts/build.sh
Expected output:
Building StreamSocial Production System...
Running test suite...
test_circuit_breaker_opens_after_failures PASSED
test_deployment_workflow PASSED
test_failover_timing_meets_rto PASSED
test_capacity_calculation_for_100m_users PASSED
test_load_test_achieves_target_rps PASSED
✓ Build completed successfully
✓ All tests passed
Production system ready for deployment
Running the Comprehensive Demo
The demo script walks through all components:
bash scripts/demo.sh
This runs five demonstrations:
Demo 1: Capacity Planning Calculator
Shows exact infrastructure requirements for 100M daily active users:
Calculating requirements for 100M DAU...
Peak RPS: 57,870
Recommended Partitions: 18
Recommended Brokers: 6
Total Storage: 3.9 TB
Monthly Cost: $22,600
Cost per million events: $0.0068
Demo 2: Blue-Green Deployment Simulation
Executes complete zero-downtime deployment:
[DEPLOY_START] Deploying v2.5.0 to Green
[TEST_PASS] API health checks
[TEST_PASS] Database connectivity
[TEST_PASS] Load test (1000 RPS)
[SWITCH_COMPLETE] Traffic on Green
[DEPLOY_SUCCESS] Deployment completed in 305.2s
Demo 3: Disaster Recovery Drill
Simulates datacenter failure and automatic recovery:
[FAILURE_DETECTED] us-east-1 confirmed down after 3 checks
[FAILOVER_START] us-east-1 → us-west-1
[DNS_UPDATE] Route53 updated (30s)
[DATABASE_PROMOTION] PostgreSQL replica promoted (90s)
[FAILOVER_COMPLETE] Completed in 298.3s
RTO Target: 300s ✓ RTO Met: True
Demo 4: Load Testing Scenarios
Validates system under different traffic loads:
Scenario: Normal Load (50K RPS)
Actual RPS: 49,876
Success Rate: 99.97%
P99 Latency: 178ms ✓
Scenario: Peak Load (150K RPS)
Actual RPS: 148,234
Success Rate: 99.94%
P99 Latency: 287ms ✓
Demo 5: Production Monitoring Dashboard
Starts the real-time dashboard:
Starting real-time dashboard...
Dashboard available at: http://localhost:8000
Dashboard features:
- Live metrics from all regions
- Real-time traffic distribution
- Deployment status tracking
- Recent alerts panel
- System health overview
Open http://localhost:8000 in your browser to see the dashboard.
Starting the Production Dashboard
To run the dashboard separately:
bash scripts/start.sh
This starts the FastAPI server with WebSocket support. You’ll see:
✓ Dashboard started (PID: 12345)
Dashboard available at: http://localhost:8000
Press Ctrl+C to stop
The dashboard shows:
Three region cards (US-East, US-West, EU-Central)
Real-time RPS, error rates, latency percentiles
CPU and memory usage
Deployment status (Blue/Green traffic split)
Recent alerts
Metrics update every second via WebSocket connection.
Stopping Services
When you’re done:
bash scripts/stop.sh
This gracefully shuts down the dashboard and cleans up processes.
Running Individual Components
You can also run components separately:
# Capacity planning
python src/monitoring/capacity_planner.py
# Blue-green deployment
python src/deployment/blue_green_deployer.py
# Disaster recovery
python src/disaster_recovery/failover_manager.py
# Load testing
python src/monitoring/load_tester.py
# Dashboard
python src/monitoring/production_dashboard.py
Each component runs independently and shows its specific functionality.
Key Takeaways
Production readiness is about preparation, not hope. The best engineering teams don’t fight fires—they prevent them.
What You’ve Accomplished
You built a complete production system that:
Deploys with zero downtime using blue-green strategy
Recovers from datacenter failures automatically in <5 minutes
Handles 150,000+ requests per second across three regions
Monitors all metrics in real-time with automated alerting
Calculates exact infrastructure costs before deployment
Validates capacity through comprehensive load testing
The Patterns You’ve Mastered
These are the same patterns used by companies processing billions of events daily:
Blue-green deployments let you deploy 100× per day without fear. Netflix deploys 4,000 times daily using this exact pattern.
Capacity planning with mathematics tells you exactly what infrastructure you need. No guessing, no over-provisioning, no surprises.
Multi-region replication keeps you online when datacenters fail. Instagram serves 500M users with 99.9% uptime using this architecture.
Circuit breakers prevent cascading failures. When one service fails, it doesn’t take down the entire system.
Real-time monitoring catches problems before users complain. Metrics, alerts, and distributed tracing show you exactly what’s happening.
Real-World Applications
This architecture powers:
Twitter: 500 million tweets per day
Instagram: 500 million daily active users
Netflix: 4,000+ deployments per day
Stripe: $640 billion annual payment volume with 99.999% uptime
You’re now equipped to build systems at the same scale.
Assignment: Final System Review
Time to prove your system is production-ready.
Your Task
Conduct a comprehensive production readiness review of your StreamSocial deployment.
Requirements
Run load tests at 2× your expected peak capacity
Perform disaster recovery drill (kill primary region, validate failover)
Generate latency reports (P50/P95/P99/P99.9) for all API endpoints
Create runbooks for the top 5 most likely production incidents
Document your capacity plan with growth projections for 12 months
Deliverables
Submit these five items:
Load Test Results
Throughput achieved (RPS)
Success rate percentage
Latency percentiles (P50, P95, P99, P99.9)
Error rate and types of errors
Disaster Recovery Timeline
Detection time (how long to notice failure)
Failover time (how long to switch regions)
Validation time (how long to verify recovery)
Total RTO (detection + failover + validation)
Estimated data loss (RPO in seconds)
Production Monitoring Dashboard Screenshot
All three regions visible
Current RPS and latency metrics
Deployment status
Recent alerts panel
Capacity Planning Spreadsheet
Current capacity (DAU, RPS, storage)
Growth projections (month by month for 12 months)
Infrastructure requirements at each milestone
Cost projections
Incident Response Runbooks Create one page for each incident:
High API Latency (P99 >500ms)
High Error Rate (>0.1%)
Database Replication Lag (>10 seconds)
Kafka Consumer Lag (>1M messages)
Primary Region Failure (complete datacenter down)
Each runbook should include:
How to detect the incident
How to diagnose root cause
Steps to mitigate impact
Steps to resolve permanently
How to prevent recurrence
Success Criteria
Your system passes if:
Sustains 2× peak load for 30 minutes with <0.1% errors
Failover completes in <5 minutes with <10 seconds data loss
P99 latency <200ms for all critical endpoints
Complete documentation for on-call engineers
Capacity plan shows you can handle 12 months of growth
Solution Hints and Next Steps
Running Load Tests
Use the built-in load tester:
python src/monitoring/load_tester.py
Or use k6 for more sophisticated testing:
k6 run --vus 1000 --duration 30m load-test.js
# Monitor metrics during test
watch -n 1 'curl -s localhost:9090/metrics | grep ^api_latency_p99'
Expected results:
Normal load (50K RPS): P99 <200ms, 99.9% success rate
Peak load (150K RPS): P99 <300ms, 99.5% success rate
Spike test (300K RPS): Graceful degradation, no crashes
Disaster Recovery Drill
Simulate primary region failure:
# Simulate US-East going down
docker-compose -f us-east.yml down
# Verify automatic failover to US-West
curl -I https://streamsocial.com/health
# Should return 200 OK from us-west-1
# Check replication offset matches
kafka-consumer-groups --bootstrap-server us-west-1:9092 \
--group timeline-processor --describe
# Lag should be near zero
Expected timeline:
Detection: 15 seconds
DNS update: 30 seconds
Database promotion: 90 seconds
Kafka rebalance: 20 seconds
Smoke tests: 120 seconds
Total: ~5 minutes
Capacity Planning Formula
For your spreadsheet:
Current Month (January):
- DAU: 100M
- Events per user: 10
- Peak RPS: 57,870
- Brokers needed: 6
- Storage needed: 3.9 TB
- Monthly cost: $22,600
Growth: 10% month over month
Month 12 (December):
- DAU: 314M (100M × 1.1^12)
- Peak RPS: 181,000
- Brokers needed: 18
- Storage needed: 12 TB
- Monthly cost: $71,000
Creating Runbooks
Template for each incident:
INCIDENT: High API Latency (P99 >500ms)
DETECTION:
- Alert fires when P99 >500ms for 5 consecutive minutes
- Slack notification to #incidents channel
- Page on-call engineer if sustained >15 minutes
DIAGNOSIS:
1. Check Grafana dashboard for which service is slow
2. Check Jaeger traces for slow requests
3. Check Kafka consumer lag
4. Check database query performance
5. Check external API dependencies
MITIGATION:
- If Kafka lag high: Scale up consumer instances
- If database slow: Enable query cache, add read replica
- If CPU high: Trigger auto-scaling
- If external API slow: Enable circuit breaker
RESOLUTION:
- Root cause analysis within 24 hours
- Post-mortem document created
- Preventive measures implemented
- Runbook updated with learnings
Congratulations
You’ve completed the 60-day Kafka Mastery journey. You’re now equipped to:
Design event-driven architectures at scale
Deploy production systems with confidence
Handle disasters and failures gracefully
Calculate capacity requirements precisely
Monitor and optimize system performance
This is the foundation for senior engineering roles at companies like Netflix, Uber, Stripe, and Airbnb.
Welcome to the world of production-grade distributed systems.
Commands Reference
Quick reference for all the commands you’ll need:
Initial Setup
chmod +x day60_implementation.sh
./day60_implementation.sh
cd day60_production_readiness
source venv/bin/activate
Build and Test
bash scripts/build.sh # Run full build and tests
python -m pytest tests/ -v # Run tests only
python -m pytest tests/ --cov=src # Run with coverage
Run Demonstrations
bash scripts/demo.sh # Run all 5 demos
python src/monitoring/capacity_planner.py # Demo 1
python src/deployment/blue_green_deployer.py # Demo 2
python src/disaster_recovery/failover_manager.py # Demo 3
python src/monitoring/load_tester.py # Demo 4
python src/monitoring/production_dashboard.py # Demo 5
Dashboard Operations
bash scripts/start.sh # Start dashboard
bash scripts/stop.sh # Stop dashboard
# Open http://localhost:8000 in browser
Individual Components
python src/services/post_service.py # Test post service
python src/deployment/blue_green_deployer.py # Test deployment
python src/disaster_recovery/failover_manager.py # Test DR
python src/monitoring/capacity_planner.py # Calculate capacity
python src/monitoring/load_tester.py # Run load tests
Testing
python -m pytest tests/test_production_readiness.py::TestPostService -v
python -m pytest tests/test_production_readiness.py::TestBlueGreenDeployment -v
python -m pytest tests/test_production_readiness.py::TestDisasterRecovery -v
python -m pytest tests/test_production_readiness.py::TestCapacityPlanning -v
python -m pytest tests/test_production_readiness.py::TestLoadTesting -v
This comprehensive foundation prepares you for operating production systems at any scale.