Implementing Secret Rotation

Implementing Secret Rotation

Secret rotation limits exposure duration if secrets are compromised. Manual rotation processes don't scale and often lead to skipped rotations. Automated rotation ensures regular updates without human intervention. However, rotation must be coordinated between secret stores and applications to avoid service disruptions.

Application-level rotation strategies vary by secret type. Database credentials might use dual accounts with overlapping validity periods. API keys might support multiple active keys during rotation. Certificates require coordination between issuance and trust store updates. Each pattern requires specific implementation approaches in Kubernetes.

# CronJob for automated secret rotation
apiVersion: batch/v1
kind: CronJob
metadata:
  name: secret-rotation
  namespace: security-system
spec:
  schedule: "0 2 * * 0"  # Weekly at 2 AM on Sundays
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: secret-rotator
          containers:
          - name: rotator
            image: secret-rotator:latest
            command:
            - /bin/sh
            - -c
            - |
              # Generate new secret
              NEW_PASSWORD=$(openssl rand -base64 32)
              
              # Update external system (e.g., database)
              mysql -h $DB_HOST -u admin -p$ADMIN_PASSWORD -e \
                "ALTER USER 'app'@'%' IDENTIFIED BY '$NEW_PASSWORD';"
              
              # Update Kubernetes secret
              kubectl create secret generic database-credentials \
                --from-literal=password=$NEW_PASSWORD \
                --dry-run=client -o yaml | \
                kubectl apply -f -
              
              # Trigger rolling update of dependent deployments
              kubectl rollout restart deployment/api-server
              kubectl rollout restart deployment/worker
              
              # Verify rollout completion
              kubectl rollout status deployment/api-server
              kubectl rollout status deployment/worker
          restartPolicy: OnFailure

---
# Certificate rotation with cert-manager
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: app-tls
  namespace: production
spec:
  secretName: app-tls-secret
  duration: 720h    # 30 days
  renewBefore: 168h # Renew 7 days before expiry
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: app.example.com
  dnsNames:
  - app.example.com
  - api.example.com
  usages:
  - digital signature
  - key encipherment
  - server auth
  # Rotation behavior
  rotationPolicy: Always
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always

Zero-downtime rotation requires careful orchestration. Applications must handle both old and new secrets during rotation periods. This might involve checking multiple credentials, implementing graceful reconnection logic, or using versioned secrets. Health checks should verify applications successfully use new secrets before removing old ones.