Every outage during deployment is a choice. Here are the patterns that make production deploys invisible to users.
Why Deployments Cause Downtime
The three most common causes:
- Brief gap between stopping old version and starting new version
- Database migrations that lock tables or break the old app version
- Environment variable / config changes that aren't backward compatible
Blue-Green Deployment
Run two identical production environments. Route traffic to the idle one after deployment:
Production:
Load Balancer
-> Blue (currently serving 100% traffic, old version)
-> Green (idle, deploy new version here)
Deploy process:
1. Deploy new version to Green
2. Run smoke tests against Green
3. Switch load balancer to Green (instant, no downtime)
4. Blue becomes idle (ready for rollback)
# Nginx blue-green switch
upstream active {
server green.internal:3000; # Change to 'blue' to rollback
}
server {
listen 80;
location / {
proxy_pass http://active;
}
}
To switch: update nginx config and run nginx -s reload (zero-downtime config reload).
Canary Releases
Deploy to a small percentage of users first:
# Send 5% of traffic to canary
upstream production {
server prod-v1.internal:3000 weight=95;
server prod-v2.internal:3000 weight=5; # canary
}
Watch error rates. If canary is healthy, gradually shift weight. If errors spike, remove canary instantly.
Database Migrations: The Hard Part
The key rule: your database schema must be compatible with both the old and new app version simultaneously. Deploys take time, and during rollout, both versions run concurrently.
Safe Migration Sequence for Adding a Column
Phase 1: Add nullable column (old app ignores it)
Migration: ALTER TABLE users ADD COLUMN phone TEXT;
Deploy: old app continues running fine
Phase 2: Deploy new app (reads and writes phone)
Both versions coexist temporarily -- old ignores, new uses
Phase 3: Make non-null if needed (only after all old instances gone)
Migration: ALTER TABLE users ALTER COLUMN phone SET NOT NULL;
Safe Migration Sequence for Removing a Column
Phase 1: Deploy new app that stops reading the column
Code change: remove column from SELECT, ORM model, etc.
Phase 2: Drop the column (only after ALL old instances are gone)
Migration: ALTER TABLE users DROP COLUMN legacy_field;
Never drop a column in the same deploy as the code change.
Safe Migration Sequence for Renaming a Column
Phase 1: Add new column alongside old one
Migration: ALTER TABLE users ADD COLUMN full_name TEXT;
Phase 2: Backfill new column
UPDATE users SET full_name = name WHERE full_name IS NULL;
Phase 3: Keep both in sync (trigger or dual-write in app)
Phase 4: Deploy new app using full_name
Phase 5: Drop old column after all old instances gone
ALTER TABLE users DROP COLUMN name;
Feature Flags
Decouple deployment from release:
// Simple feature flag with environment variable
function isEnabled(flag: string): boolean {
return process.env[`FEATURE_${flag.toUpperCase()}`] === 'true'
}
// Usage
if (isEnabled('NEW_CHECKOUT')) {
return <NewCheckout />
} else {
return <OldCheckout />
}
With feature flags:
- Deploy new code (behind flag = off) -- no user impact
- Enable for internal users first
- Enable for 5% of users (canary)
- Monitor, then roll out to 100%
- Remove old code path after full rollout
Health Check Timing
# Kubernetes: wait for health check before routing traffic
readinessProbe:
httpGet:
path: /ready
port: 3000
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3 # Remove from load balancer after 3 failures
livenessProbe:
httpGet:
path: /health
port: 3000
initialDelaySeconds: 30
periodSeconds: 15
Your /ready endpoint should check: database connection, cache connection, any warm-up tasks. Only return 200 when the app can handle traffic.
Rollback Plan
Every deploy needs a tested rollback procedure:
- Blue-green: switch load balancer back to previous environment
- Canary: remove canary from upstream pool
- Database: maintain backward-compatible schemas (rollback is trivially safe)
- Feature flags: flip the flag off
Document the rollback steps before you deploy, not after something goes wrong.