Observability for Security

Observability for Security

Service mesh observability provides unprecedented visibility into service communications, essential for security monitoring. Automatic metrics collection captures request rates, error rates, and latencies for every service interaction. This telemetry enables detection of anomalous behaviors indicating potential security incidents.

Distributed tracing reveals complete request paths through microservices architectures. Security teams can trace suspicious requests across multiple services, understanding attack patterns and impact. Trace analysis identifies unusual call patterns, excessive fan-out, or suspicious service interactions that might indicate compromise.

# Telemetry configuration for security monitoring
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: security-metrics
  namespace: production
spec:
  metrics:
  - providers:
    - name: prometheus
    overrides:
    # Track authentication failures
    - match:
        metric: REQUEST_COUNT
        customDimensions:
          response_code: "401 | 403"
      tagOverrides:
        security_event:
          value: "auth_failure"
    
    # Monitor unusual request patterns
    - match:
        metric: REQUEST_COUNT
      tagOverrides:
        source_workload:
          value: source.workload.name | "unknown"
        request_protocol:
          value: request.protocol | "unknown"
    
    # Track request sizes for data exfiltration detection
    - match:
        metric: REQUEST_SIZE
      tagOverrides:
        monitoring_priority:
          value: |
            has(request.size) && request.size > 10485760 ? "high" : "normal"

---
# Access logging for security audit
apiVersion: v1
kind: ConfigMap
metadata:
  name: access-log-config
  namespace: istio-system
data:
  access-log-format.json: |
    {
      "timestamp": "%START_TIME%",
      "method": "%REQ(:METHOD)%",
      "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
      "protocol": "%PROTOCOL%",
      "response_code": "%RESPONSE_CODE%",
      "response_flags": "%RESPONSE_FLAGS%",
      "bytes_received": "%BYTES_RECEIVED%",
      "bytes_sent": "%BYTES_SENT%",
      "duration": "%DURATION%",
      "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
      "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%",
      "user_agent": "%REQ(USER-AGENT)%",
      "request_id": "%REQ(X-REQUEST-ID)%",
      "authority": "%REQ(:AUTHORITY)%",
      "upstream_host": "%UPSTREAM_HOST%",
      "source_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
      "source_workload": "%DOWNSTREAM_PEER_SUBJECT%",
      "destination_workload": "%UPSTREAM_PEER_SUBJECT%",
      "jwt_claims": "%DYNAMIC_METADATA(envoy.filters.http.jwt_authn:payload)%"
    }

Security-focused dashboards aggregate mesh telemetry for threat detection. Visualizations highlight authentication failures, authorization denials, and unusual traffic patterns. Real-time alerting notifies security teams of potential incidents. Integration with SIEM platforms enables correlation with other security data sources.