Monitoring and Observability for Node.js Apps in Production

You can't fix what you can't see. Here's how to get full visibility into what your Node.js app is doing in production.

The Three Pillars

1. Logs -- What happened (events, errors, request details)
2. Metrics -- How much (request rate, error rate, latency, resource usage)
3. Traces -- Where time was spent (which function, which DB query, which service)

Structured Logging with Pino

import pino from 'pino'

const logger = pino({
    level: process.env.LOG_LEVEL || 'info',
    transport: process.env.NODE_ENV === 'development' ? {
        target: 'pino-pretty'  // Human-readable in dev
    } : undefined  // JSON in production (machine-parseable)
})

// Use child loggers with context
app.use((req, res, next) => {
    req.log = logger.child({
        requestId: crypto.randomUUID(),
        method: req.method,
        url: req.url,
        userId: req.user?.id
    })
    next()
})

// Log with context
req.log.info({ productId }, 'Product fetched')
req.log.error({ err, productId }, 'Failed to fetch product')

Never use console.log in production. Structured JSON logs can be queried in Datadog, Loki, CloudWatch.

Metrics with Prometheus

import client from 'prom-client'

// Default Node.js metrics (memory, CPU, event loop lag)
client.collectDefaultMetrics()

// Custom business metrics
const httpRequests = new client.Counter({
    name: 'http_requests_total',
    help: 'Total HTTP requests',
    labelNames: ['method', 'route', 'status']
})

const httpDuration = new client.Histogram({
    name: 'http_request_duration_seconds',
    help: 'HTTP request duration',
    labelNames: ['method', 'route', 'status'],
    buckets: [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5]
})

// Middleware to track all requests
app.use((req, res, next) => {
    const end = httpDuration.startTimer()
    res.on('finish', () => {
        const labels = {
            method: req.method,
            route: req.route?.path || 'unknown',
            status: res.statusCode
        }
        httpRequests.inc(labels)
        end(labels)
    })
    next()
})

// Expose metrics endpoint for Prometheus to scrape
app.get('/metrics', async (req, res) => {
    res.set('Content-Type', client.register.contentType)
    res.send(await client.register.metrics())
})

Distributed Tracing with OpenTelemetry

// tracing.js -- load this BEFORE any other imports
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http'
import { ExpressInstrumentation } from '@opentelemetry/instrumentation-express'
import { PgInstrumentation } from '@opentelemetry/instrumentation-pg'

const sdk = new NodeSDK({
    serviceName: 'my-api',
    traceExporter: new OTLPTraceExporter({
        url: 'http://localhost:4318/v1/traces'  // Jaeger or Tempo
    }),
    instrumentations: [
        new HttpInstrumentation(),
        new ExpressInstrumentation(),
        new PgInstrumentation()  // Auto-traces all DB queries
    ]
})

sdk.start()
process.on('SIGTERM', () => sdk.shutdown())

// Custom span for a specific operation
import { trace } from '@opentelemetry/api'

const tracer = trace.getTracer('my-api')

async function processOrder(orderId: string) {
    const span = tracer.startSpan('processOrder')
    span.setAttribute('order.id', orderId)

    try {
        const result = await doWork()
        span.setStatus({ code: SpanStatusCode.OK })
        return result
    } catch (err) {
        span.setStatus({ code: SpanStatusCode.ERROR, message: err.message })
        span.recordException(err)
        throw err
    } finally {
        span.end()
    }
}

Alerting Rules (Prometheus/Grafana)

# prometheus/alerts.yml
groups:
  - name: api_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m])
          / rate(http_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for 2 minutes"

      - alert: SlowP95Response
        expr: |
          histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 response time above 1s"

      - alert: HighMemoryUsage
        expr: process_resident_memory_bytes / 1024 / 1024 > 512
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Process using more than 512MB RAM"

Grafana Dashboard

Create a dashboard with these panels:

• Request rate: rate(http_requests_total[5m]) by status
• Error rate: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])
• P50/P95/P99 latency: histogram_quantile(0.95, ...)
• Active connections: nodejs_active_handles_total
• Event loop lag: nodejs_eventloop_lag_seconds

What to Alert On

Alert on symptoms, not causes:

• Error rate > 1% (users see errors)
• P95 latency > 2s (users feel slowness)
• Memory increasing for >30min (memory leak)

Don't alert on CPU usage alone -- high CPU isn't always bad. Alert when it causes high latency or errors.