In the context of critical infrastructure, it is especially important to implement a multi-layered security concept. We have explored how log-based alerts can be implemented when dealing with an extensive cloud landscape consisting of many Google Cloud projects grouped under a common organization.

Below, I will demonstrate how to aggregate logs from all Google Cloud projects to create centralized, project-wide log-based alerts. I will also address some of the hurdles we encountered.

Overview

We create a Log Router at the organization level that directs all logs to another project. It is necessary to adjust the filter of the _Default sink, grant permissions, and create a log alert. Optionally, all logs can then be stored in a logging bucket.

Log Router at the Organization Level

We install a Log Router (sink) on the GCP organization that collects all logs, including those of the underlying projects (include_children), and directs them to a Google Cloud project. We leave the parameter intercept_children at its default false to continue accessing logs in the respective projects. Theoretically, it is also possible to redirect directly to a logging bucket, but then it is not possible to install log alerts on it. Thus, the “detour” via a project is necessary.

1data "google_organization" "org" {
2  domain = "example.com"
3}
4
5resource "google_project" "logging" {
6  // Your project in which you collect all aggregated logs and implement log alerts
7}
8
9resource "google_logging_organization_sink" "org_aggregation" {
10  name             = "organization-logs-aggregation-sink"
11  description      = "Route organization logs to logging project"
12  org_id           = data.google_organization.org
13  destination      = "logging.googleapis.com/projects/${google_project.logging.id}"
14  include_children = true
15}

Setting Permissions Correctly

The logs are written by a dedicated service account that needs the appropriate permissions for the logging project.

1resource "google_project_iam_member" "log_writer" {
2  project = google_project.logging.id
3  role    = "roles/logging.logWriter"
4  member  = google_logging_organization_sink.org_aggregation.writer_identity
5}

Adjusting the _Default Log Router

All logs forwarded to another project are processed by the log routers installed there. By default, each project has a log router named _Default. This is generally not an issue unless you want to create a log-based alert on the audit logs. The _Default sink has a predefined filter that filters out audit logs:

1NOT LOG_ID("cloudaudit.googleapis.com/activity") AND NOT \
2LOG_ID("externalaudit.googleapis.com/activity") AND NOT \
3LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT \
4LOG_ID("externalaudit.googleapis.com/system_event") AND NOT \
5LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT \
6LOG_ID("externalaudit.googleapis.com/access_transparency")

For example, if you want to create an alert on the SetIamPolicy audit logs, you need to remove cloudaudit.googleapis.com/activity from the filter. I recommend importing the log router into the Terraform state to easily edit the filter.

1import {
2  id = "projects/<project-id>/sinks/_Default"
3  to = google_logging_project_sink.default
4}
5resource "google_logging_project_sink" "default" {
6  name        = "_Default"
7  project     = google_project.logging.id
8  destination = "logging.googleapis.com/projects/${google_project.logging.id}/locations/global/buckets/_Default"
9  filter      = <<-EOT
10  NOT LOG_ID("externalaudit.googleapis.com/activity") AND NOT
11  LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT
12  LOG_ID("externalaudit.googleapis.com/system_event") AND NOT
13  LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT
14  LOG_ID("externalaudit.googleapis.com/access_transparency")
15  EOT
16}

View Logs in the Log Explorer

To view the aggregated logs, open the Log Explorer and switch the scope to _Default.

Implement Log Alert

Continuing with the SetIamPolicy example, a log-based alert could look like this:

1resource "google_monitoring_alert_policy" "default" {
2  display_name = "IAM role assignment to user"
3  severity     = "WARNING"
4  project      = google_project.logging.id
5
6  alert_strategy {
7    auto_close = "1800s"
8
9    notification_rate_limit {
10      period = "300s"
11    }
12  }
13
14  combiner = "OR"
15  conditions {
16    display_name = "Log match condition"
17
18    condition_matched_log {
19      filter = <<-EOT
20      protoPayload.methodName="SetIamPolicy" AND
21      protoPayload.serviceData.policyDelta.bindingDeltas.action="ADD" AND
22      protoPayload.serviceData.policyDelta.bindingDeltas.member:"user:"
23      EOT
24    }
25  }
26
27  documentation {
28    content   = "An IAM role has been assigned directly to a user. Roles should always be assigned to security groups and not individuals."
29    mime_type = "text/markdown"
30  }
31
32  notification_channels = [
33    <your-notification-channels>
34  ]
35}

Optional Routing to a Logging Bucket

If you also want to store the logs, it is possible to create a logging bucket and specify it as the target in the log router. It is important to highlight that the service account needs special permissions to write to the bucket.

1resource "google_logging_project_bucket_config" "org_aggregation" {
2  project        = google_project.logging.id
3  location       = var.region
4  retention_days = 30
5  bucket_id      = "organization-logs-aggregation"
6}
7
8resource "google_logging_project_sink" "org_aggregation" {
9  name        = "organization-logs-aggregation-sink"
10  description = "Route aggregated organization logs to Cloud Logging bucket"
11  project     = google_project.logging.id
12  destination = "logging.googleapis.com/${google_logging_project_bucket_config.org_aggregation.id}"
13}
14
15resource "google_project_iam_member" "log_writer" {
16  project = google_project.logging.id
17  role    = "roles/logging.bucketWriter"
18  member  = google_logging_project_sink.org_aggregation.writer_identity
19}

Filter Logs

For cost reasons, it might make sense not to copy all logs from all projects. For the above example, which is only interested in the SetIamPolicy logs, it would suffice to provide a filter to the google_logging_organization_sink.org_aggregation.

1filter = "protoPayload.methodName=\"SetIamPolicy\""