Auditing GKE operations? Configure Data Access audit logs

If you’re setting up GKE audit logging, you are probably following the instructions on this page. It describes two levels of audit logging that are available via GCP: the ‘Admin Activity log’ and the ‘Data Access log’. The documentation says:

Admin Activity logging is enabled by default and has no extra cost. Data Access logging is disabled by default, and enabling it can result in extra billing. To learn more about enabling Data Access logging, and the associated costs, see ‘Configuring Data Access Logs’.

The ‘Configuring Data Access Logs’ link points to the general Data Access logging page for all GCP services, and has no Kubernetes-specific information. A more useful page to understand exactly how the logging policy works can be found here. This page clarifies that:

  • Entries that represent create, delete, and update requests go to your Admin Activity log.
  • Entries that represent get, list, and updateStatus requests go to your Data Access log.

While this might seem reasonable on the face of it (most destructive or concerning operations will go into the Admin Activity logs), the Admin Activity logs are missing get operations on Secret objects by default. So for example, if you store a service account password in your cluster as a Kubernetes secret, a kubectl get secret service_account_password -o yaml will get an attacker the entire secret without logging a single line into the audit logs. For this reason alone (if you use Kubernetes secrets for anything sensitive) it is probably essential that you enable the Data Access logging as well.

Interestingly, at the end of the [GKE how-to] on audit logging, they specify a method for adding Data Access audit logging that will probably generate way more log data than you actually need (assuming you are only interested data access logging from GKE).

Instead of updating the project’s IAM policy with:

auditConfigs:
- auditLogConfigs:
  - logType: ADMIN_READ
  - logType: DATA_WRITE
  - logType: DATA_READ
  service: allServices

as the page (currently) suggests, you can get away with the much less verbose:

auditConfigs:
- auditLogConfigs:
  - logType: ADMIN_READ
  - logType: DATA_READ
  - logType: DATA_WRITE
  service: container.googleapis.com

You can also do this via the GUI by following the instructions here.

*****
Written by Feroz Salam on 10 February 2022