Kubernetes Audit Logs

Docs > Integrations > Kubernetes Audit Logs

Supported OS Linux Mac OS Windows

Overview

Collect Kubernetes audit logs to track everything that happens inside your Kubernetes clusters, including every call made to the Kubernetes API by any service. This includes the control plane (built-in controllers, the scheduler), node daemons (the kubelet, kube-proxy, and others), cluster services (such as the cluster autoscaler), users making kubectl requests, and even the Kubernetes API itself.

With the Kubernetes audit logs integration, you can diagnose permission issues, identify RBAC policies that need to be updated, and track slow API requests that are impacting your whole cluster. Deep dive into these topics with the Datadog talk at KubeCon 2019.

Setup

This integration is available for Agent >6.0

Configuration

For more information about setting up Kubernetes audit logs, see Kubernetes Auditing.

To enable audit logs in Kubernetes:

Audit logs are disabled by default in Kubernetes. To enable them in your API server configuration, specify an audit policy file path:

kube-apiserver
  [...]
  --audit-log-path=/var/log/kubernetes/apiserver/audit.log
  --audit-policy-file=/etc/kubernetes/audit-policies/policy.yaml

Create the policy file at /etc/kubernetes/audit-policies/policy.yaml to specify the types of API requests you want to capture in your audit logs. Audit policy rules are evaluated in order. The API server follows the first matching rule it finds for each type of operation or resource. Example of an audit policy:

# /etc/kubernetes/audit-policies/policy.yaml

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
    # do not log requests to the following
    - level: None
      nonResourceURLs:
          - '/healthz*'
          - '/logs'
          - '/metrics'
          - '/swagger*'
          - '/version'

    # limit level to Metadata so token is not included in the spec/status
    - level: Metadata
      omitStages:
          - RequestReceived
      resources:
          - group: authentication.k8s.io
            resources:
                - tokenreviews

    # extended audit of auth delegation
    - level: RequestResponse
      omitStages:
          - RequestReceived
      resources:
          - group: authorization.k8s.io
            resources:
                - subjectaccessreviews

    # log changes to pods at RequestResponse level
    - level: RequestResponse
      omitStages:
          - RequestReceived
      resources:
          # core API group; add third-party API services and your API services if needed
          - group: ''
            resources: ['pods']
            verbs: ['create', 'patch', 'update', 'delete']

    # log everything else at Metadata level
    - level: Metadata
      omitStages:
          - RequestReceived

This example policy file configures the API server to log at the highest level of detail for certain types of cluster-changing operations (update, patch, create, delete). It also tracks requests to the subjectaccessreviews resource at the highest level to help troubleshoot authentication delegation issues.

You may want to reduce the level of verbosity to Metadata for endpoints that contain sensitive data, such as the tokenreviews resource. Datadog also omits the RequestReceived stage from logs.

In the last section, for everything that was not explicitly configured by the previous rules, the policy is configured to log at Metadata level. As audit logs might be verbose, you can choose to exclude less critical actions/verbs, such as operations that don’t change the cluster state like list, watch, and get.

Log collection

Install the Agent on your Kubernetes environment.
Log collection is disabled by default. Enable it in the env section of your DaemonSet:
```
env:
    # (...)
    - name: DD_LOGS_ENABLED
      value: 'true'
```

Mount the audit log directory as well as a directory that the Agent uses to store a pointer to know which log was last sent from that file. To do this, add the following in the volumeMounts section of the daemonset:

 # (...)
    volumeMounts:
      # (...)
      - name: pointdir
        mountPath: /opt/datadog-agent/run
      - name: auditdir
        mountPath: /var/log/kubernetes/apiserver
      - name: dd-agent-config
        mountPath: /conf.d/kubernetes_audit.d
  # (...)
  volumes:
    # (...)
    - hostPath:
        path: /opt/datadog-agent/run
      name: pointdir
    - hostPath:
        path: /var/log/kubernetes/apiserver
      name: auditdir
    - name: dd-agent-config
        configMap:
          name: dd-agent-config
          items:
            - key: kubernetes-audit-log
              path: conf.yaml
  # (...)

This also mounts the conf.d folder which is used to configure the Agent to collect logs from the audit log file.

Configure the Agent to collect logs from that file with a ConfigMap:

kind: ConfigMap
apiVersion: v1
metadata:
    name: dd-agent-config
    namespace: default
data:
    kubernetes-audit-log: |-
        logs:
          - type: file
            path: /var/log/kubernetes/apiserver/audit.log
            source: kubernetes.audit
            service: audit