Building Comprehensive Kubernetes Security Monitoring

Building Comprehensive Kubernetes Security Monitoring

Security monitoring in Kubernetes requires data collection from multiple sources across the stack. The Kubernetes API audit logs provide visibility into control plane operations, while container runtime logs reveal workload behavior. Network flow logs expose communication patterns, and admission controller logs show policy violations. Correlating these diverse data sources enables comprehensive security visibility.

The ephemeral nature of Kubernetes workloads complicates traditional monitoring approaches. Containers may exist for seconds or minutes, requiring immediate log collection before destruction. Pod IP addresses change with each restart, making network flow correlation challenging. Dynamic scaling creates variable baseline behaviors that static thresholds cannot accommodate. Effective monitoring strategies must account for these Kubernetes-specific characteristics.

Centralized logging architectures ensure security data availability despite workload ephemerality. Log forwarding agents running as DaemonSets collect logs from all nodes. These agents must handle high-volume log streams while minimizing resource consumption. Proper buffering and retry mechanisms prevent log loss during network disruptions or backend outages.

# Fluent Bit DaemonSet for comprehensive log collection
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: monitoring
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Systemd_Filter _SYSTEMD_UNIT=docker.service
        Read_From_Tail On

    [INPUT]
        Name tail
        Tag kube.*
        Path /var/log/containers/*.log
        Parser cri
        DB /var/log/flb-kube.db
        Mem_Buf_Limit 50MB
        Skip_Long_Lines On
        Refresh_Interval 10

    [INPUT]
        Name tail
        Tag audit.*
        Path /var/log/kubernetes/audit/audit.log
        Parser json
        DB /var/log/flb-audit.db
        Mem_Buf_Limit 100MB

    [FILTER]
        Name kubernetes
        Match kube.*
        Kube_URL https://kubernetes.default.svc:443
        Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix kube.var.log.containers.
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [FILTER]
        Name modify
        Match audit.*
        Add k8s.audit.log true
        Add environment ${CLUSTER_NAME}

    [OUTPUT]
        Name forward
        Match *
        Host log-aggregator.monitoring.svc.cluster.local
        Port 24224
        net.keepalive on
        net.keepalive_idle_timeout 30
        net.keepalive_max_recycle 10000
        tls on
        tls.verify on
        tls.ca_file /fluent-bit/ssl/ca.crt
        tls.crt_file /fluent-bit/ssl/client.crt
        tls.key_file /fluent-bit/ssl/client.key

---
# DaemonSet deployment
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:2.0
        imagePullPolicy: Always
        securityContext:
          runAsNonRoot: false
          runAsUser: 0
          capabilities:
            drop:
            - ALL
            add:
            - DAC_OVERRIDE
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config
          mountPath: /fluent-bit/etc/
        - name: ssl
          mountPath: /fluent-bit/ssl/
          readOnly: true
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: config
        configMap:
          name: fluent-bit-config
      - name: ssl
        secret:
          secretName: fluent-bit-ssl