Kubernetes Local Development That Doesn't Drive You Insane

Local Kubernetes development has a reputation for being painful. It doesn't have to be. The right tooling makes it almost as fast as running locally without K8s.

Why Local K8s?

If production runs on Kubernetes, testing locally without K8s hides a whole class of bugs: resource limits, readiness/liveness probes, service discovery, environment variable injection from secrets/configmaps. Better to catch these locally.

k3d: Lightweight Local Cluster

k3d runs a full K3s cluster inside Docker. Much lighter than minikube:

# Install k3d
curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash

# Create a cluster with a local registry
k3d cluster create dev \
    --registry-create registry.localhost:5000 \
    --agents 2 \
    --port "8080:80@loadbalancer"

# Verify
kubectl get nodes

The local registry at registry.localhost:5000 means you push images there and K8s can pull them instantly -- no waiting for Docker Hub.

Tilt: Live Reloading for K8s

Tilt watches your files and automatically rebuilds and redeploys when code changes:

# Tiltfile (Python-like DSL)
# Build and push to local registry
docker_build(
    'registry.localhost:5000/myapp',
    '.',
    live_update=[
        # Sync source files directly into container without rebuild
        sync('./src', '/app/src'),
        # Run a command after sync (e.g., restart process)
        run('npm run compile', trigger=['./src/**/*.ts']),
    ]
)

# Apply K8s manifests
k8s_yaml(['k8s/deployment.yaml', 'k8s/service.yaml'])

# Port forward for local access
k8s_resource('myapp', port_forwards='3000:3000')

tilt up

Tilt's UI at localhost:10350 shows build/deploy status, logs, and resource health for every service.

Production-Parity Manifests

Start with your production manifests and override for local:

# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 1  # Override for local (3+ in prod)
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: registry.localhost:5000/myapp  # Local registry
          ports:
            - containerPort: 3000
          env:
            - name: NODE_ENV
              value: development
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: myapp-secrets
                  key: database-url
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "500m"
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 15
            periodSeconds: 20

Local Secrets Management

# Create a secret from your .env.local file
kubectl create secret generic myapp-secrets \
    --from-env-file=.env.local

# Or from literal values
kubectl create secret generic myapp-secrets \
    --from-literal=database-url='postgresql://localhost/devdb' \
    --from-literal=api-key='dev-key-123'

Skaffold Alternative

Skaffold is more CI/CD focused but works well for local dev too:

# skaffold.yaml
apiVersion: skaffold/v4beta7
kind: Config

build:
  artifacts:
    - image: registry.localhost:5000/myapp
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: /app/src

deploy:
  kubectl:
    manifests:
      - k8s/*.yaml

portForward:
  - resourceType: service
    resourceName: myapp
    port: 3000

Debugging in the Cluster

# Exec into a running container
kubectl exec -it deploy/myapp -- sh

# Stream logs
kubectl logs -f deploy/myapp

# Port forward ad-hoc
kubectl port-forward deploy/myapp 5432:5432

# Describe why a pod isn't starting
kubectl describe pod myapp-xxx

# Check events
kubectl get events --sort-by='.lastTimestamp'

Teardown

tilt down      # Stop Tilt and clean up resources
k3d cluster delete dev  # Remove the entire cluster

The initial setup takes about an hour. After that, k3d cluster start dev && tilt up is all you need. Changes deploy in seconds, you get production-parity resource limits and networking, and you stop being surprised by things that work locally but fail in K8s.