GCP Cloud Logging

The shape of a log entry

Everything in Cloud Logging is a structured entry, even when what you wrote was a line of text. The fields that matter operationally are the log name, the monitored resource (type plus labels identifying which instance, cluster, function or service produced it), the timestamp, the severity, an insert identifier used for deduplication and ordering, optional labels, and the payload.

The payload comes in three forms. A text payload is an unstructured string. A JSON payload is a structured object, which is what you want. A proto payload is a typed message, used by audit logs.

The resource type is worth understanding early because it is the primary axis for querying, routing and access control -- filters overwhelmingly begin with resource.type=. It is set by whatever wrote the entry rather than by you, so knowing that a Cloud Run request log is a different resource type from a GKE container log, which is different again from a load balancer log, is most of what makes querying feel fluent rather than exploratory.

Certain JSON fields are recognised specially when an application writes structured logs: a severity field populates the entry's severity rather than sitting inside the payload, an HTTP request object renders as request metadata, and trace and span identifiers link the entry to a distributed trace. Using those field names instead of your own is free and it is the difference between logs that correlate with traces and logs that do not.

The log router, and the two default buckets

Every entry written into a project goes to the log router before it is stored anywhere. The router evaluates sinks -- each a filter plus a destination -- and exclusions, and an entry can match several sinks and be written to all of them, or match an exclusion and be dropped entirely.

Two sinks exist in every project by default, and the difference between them is structural rather than a setting.

_Required receives Admin Activity audit logs, System Event audit logs and Access Transparency logs. It retains them for 400 days, it is free, and it cannot be modified, deleted, or excluded from. That immutability is deliberate: these are the records of who changed what in your infrastructure, and they are not permitted to be the thing an attacker turns off. Knowing they are always there, always free, and always 400 days is genuinely useful during an investigation.

_Default receives everything else that is not excluded, retains it for 30 days by default, and is billable on ingestion. This is the bucket the bill comes from, and it is the one you can and should shape.

The critical property: an exclusion applied at the router avoids the ingestion charge, because the entry is dropped before it is stored. A filter applied when querying does not. That asymmetry is the whole of Cloud Logging cost control, and it is why the router is the first place to look rather than retention settings.

Buckets, retention and views

Beyond the two defaults you can create custom log buckets: named storage with a chosen region and a retention between one day and ten years. Sinks route matching entries into them, which is how you give different log classes different lifetimes -- application logs for two weeks, security-relevant logs for two years, everything in a specific region for data-residency reasons.

Retention is billed after a free window, so long retention on high-volume logs is expensive by design and the intended pattern is to route the small, valuable subset to a long-retention bucket rather than extending retention on everything. Buckets can also be locked, which makes retention immutable and the bucket undeletable until it expires -- the write-once control for compliance obligations, and irreversible in exactly the way that implies.

Log views are the access-control mechanism on top of a bucket: a view is a saved filter, and permissions are granted on views rather than on the whole bucket. This is how a team is given access to logs from their own service without seeing everything in a shared project, and it is a much better answer than splitting logs across projects for access reasons.

One property to internalise before designing around it: individual log entries cannot be deleted. There is no delete-by-filter. If sensitive data is logged, the options are to wait out retention or to delete the entire bucket. That makes keeping secrets and personal data out of logs a preventive discipline with no cleanup path, which is worth stating plainly to teams before it is tested.

Sinks and where logs go next

A sink is a filter plus a destination, and four destinations cover essentially every use case.

A log bucket, including one in another project, keeps entries in Cloud Logging with its query interface. BigQuery puts them in a dataset for SQL analysis and joining against business data. Cloud Storage writes hourly-partitioned JSON files, which is the cheap archive for compliance. Pub/Sub streams them to anything else -- a third-party SIEM, a custom processor, another cloud.

The recurring operational failure with sinks is permissions. Each sink has a writer identity, a service account that Cloud Logging creates, and that identity must be granted write permission on the destination. Create a sink to a BigQuery dataset without granting the writer identity the data editor role and the sink exists, reports no error in its own configuration, and silently delivers nothing. When a sink appears to do nothing, check the writer identity's permission first; it is the cause more often than the filter is.

Aggregated sinks at the folder or organisation level are the centralisation mechanism: one sink, defined once, that captures matching logs from every project underneath and routes them to a central bucket or dataset. This is how a security team gets organisation-wide audit log coverage without asking every project owner to configure anything, and it should be set up on day one of an organisation rather than retrofitted after an incident.

Querying

The Logging query language is a filter expression over entry fields, not SQL:

resource.type="k8s_container"
resource.labels.cluster_name="prod-eu"
severity>=ERROR
jsonPayload.order_id="A-4471"
timestamp>="2026-08-01T00:00:00Z"

Terms combine with implicit AND, and explicit boolean operators, comparisons, regular expression matching and substring search are all available. Severity is ordered, so severity>=ERROR means error and above.

Two habits make queries fast rather than slow. Constrain the time range first -- the range is the primary partition, and a query over 30 days is thirty times the work of one over a day even when the filter is identical. And filter on indexed fields: log name, resource type and labels, severity and timestamp are indexed, while arbitrary payload fields are not, so a search on a JSON field alone scans. Adding an index on specific payload fields is possible and is worth doing for the handful of identifiers you routinely search by, such as a request or order identifier.

For anything analytical -- counting, grouping, joining, windowing -- the filter language is the wrong tool and the answer is the next section.

Log Analytics and SQL over logs

Upgrading a log bucket for Log Analytics makes its contents queryable with SQL through a linked BigQuery dataset, without moving or duplicating the data and without a separate storage charge. Queries run against the logs where they already are.

This changes what logs are good for. Aggregations across a day, joins between application logs and load balancer logs, percentile latency computed from request logs, counting distinct users hitting an error path -- all trivial in SQL and impossible in a filter expression.

It also changes the design conversation about sinking logs to BigQuery. The classic pattern was a sink into a dataset, paying ingestion into Logging and then storage in BigQuery for a second copy. With Log Analytics, the reasons to still export are narrower and specific: joining logs against business tables in the same project, retention beyond what the bucket holds, or feeding a downstream pipeline. For 'I want to run SQL over my logs', the upgrade is the cheaper answer and it is worth revisiting existing export-to-BigQuery pipelines against it.

Audit logs — the four kinds and the one that is off

Audit logs answer who did what, where and when, and there are four types with quite different behaviour.

Admin Activity records configuration and metadata changes -- creating a VM, changing an IAM policy, deleting a bucket. Always on, cannot be disabled, free, and routed to _Required for 400 days.

System Event records Google-initiated actions such as automatic migrations. Also always on and free.

Policy Denied records access denied by security policy. On by default and billable.

Data Access records reads and writes of user data -- who read which object, who ran which query. It is off by default for almost every service, with BigQuery the notable exception, and it must be enabled per service and per data type in the IAM audit configuration. It is off by default because it is enormous: on a busy project it can dwarf every other log source combined, and enabling it broadly without an exclusion strategy is a well-known way to multiply a logging bill.

The practical consequence is the one to remember: if an investigation needs to know who read a particular object last month, and Data Access logging was not enabled for that service, the record does not exist and cannot be reconstructed. Deciding which services need it is a security design decision to make in advance, narrowly and deliberately -- enable it for the data stores that matter, exclude the high-volume noise at the router, and route what remains to a locked long-retention bucket.

Writing good logs from applications

Write JSON, not text. A structured payload is queryable by field, indexable, and usable in log-based metrics. A text line is a string someone will write a regular expression against at three in the morning. Most language logging libraries have a Cloud Logging formatter or can emit JSON to stdout, which the collection agents parse automatically.

Use the recognised field names. Severity, message, HTTP request, trace and span identifiers, and source location are all promoted into first-class entry fields when named correctly, which gives you severity-based filtering and log-to-trace correlation for free.

Set severity honestly. Severity drives alerting and exclusion filters, and a codebase where everything is INFO or everything is ERROR makes both useless. The practical test: ERROR should mean someone would want to know.

Log identifiers, not payloads. Order and request identifiers make entries correlatable; whole request bodies make them expensive and turn logs into a personal-data store with no deletion path.

On GKE and serverless, write to stdout and stderr and let the platform collect. Writing directly to the Logging API from application code adds latency, a failure mode, and a dependency for no benefit.

Log-based metrics and alerting

A log-based metric turns matching entries into a time series. Counter metrics count entries matching a filter; distribution metrics extract a numeric field and build a histogram of it. Both become ordinary metrics available to dashboards and alerting policies.

This is the bridge from logs to monitoring, and it is the right tool when the signal exists only in logs -- a specific error string, a business event, a condition no instrumented metric covers. Alerting directly on a log-based metric is how 'page me when payment failures exceed ten a minute' gets built.

Two cautions. Labels create cardinality: extracting a field with unbounded values into a metric label produces a time series per value, with the same explosion consequences as in any metrics system, and there are enforced limits that will silently drop data when exceeded. Keep labels to bounded dimensions such as service, region or error class. And a metric only counts what was ingested -- if an exclusion filter drops the entries at the router, the metric sees nothing, which is a genuinely surprising interaction between the cost lever and the alerting story. Check your exclusions against your log-based metrics whenever either changes.

Controlling cost

Cloud Logging bills primarily on volume ingested into billable buckets, with storage charged beyond the included retention. The levers, in order of effectiveness:

1. Exclusion filters at the router. The only lever that avoids the ingestion charge entirely. The usual candidates are load balancer logs for health checks, successful request logs at high volume where a sample suffices, verbose debug output from a chatty dependency, and Data Access logs for services where they are not required. Exclusions support percentage sampling, so 'keep ten percent of successful requests and all errors' is expressible directly and is often the single biggest saving.

2. Fix the source. Turn down log levels in production, stop logging full payloads, remove per-iteration debug lines. Cheaper than any configuration and it improves signal at the same time.

3. Archive rather than retain. Route what must be kept for years to Cloud Storage, which is far cheaper per gigabyte than extended log retention, and keep short retention in Logging for the interactive window.

4. Watch what enabling Data Access logs does before rolling it out widely -- enable on one service, measure the volume, then decide.

Instrument the spend itself: system metrics report ingested bytes by resource type and by log name, which turns 'logging is expensive' into 'this one service produced sixty percent of it' in about a minute. That query is the right first step in every logging cost investigation.

Pitfalls

Sinks that silently deliver nothing because the writer identity was never granted permission on the destination.

Exclusions that hide the signal. An exclusion added to save money can remove the entries a log-based metric or an alert depends on, and nothing warns you.

Assuming Data Access logs exist. They do not unless you enabled them, and the gap is unrecoverable.

Sensitive data in logs. No per-entry deletion, so the only remedies are retention expiry or deleting a whole bucket. Prevent at the source.

Retention set on _Default instead of routing. Extending _Default's retention applies to everything and multiplies storage cost; route the valuable subset to a long-retention bucket instead.

Timestamp confusion. Entries carry both the time the event occurred and the time it was received. Querying by the wrong one during an incident with delayed ingestion produces a misleading picture.

Locked buckets chosen casually. A retention lock cannot be undone and commits you to storing that data for the full period.