Why architecture matters here
GKE fails on cost (unbounded autoscale), security (unsigned images, wide IAM), and upgrades (poorly planned control plane windows). Architecture matters because Autopilot vs Standard, workload identity, and Binary Authorization decide operational overhead + safety.
The architecture: every piece explained
The top strip is compute. Cluster mode — Standard (node-managed) or Autopilot (Google manages nodes). Control plane is Google-managed masters. Node pools for hardware variety. Autoscaler horizontal + cluster.
The middle row is identity + network. Workload identity maps Kubernetes ServiceAccount to GCP Service Account. VPC-native gives pods alias IPs. Gateway / Ingress L7 with policies. Binary Authorization enforces signed images.
The lower rows are governance. Config Sync + policy GitOps. Observability in Google Cloud stack. Ops upgrades + cost + regional design.
What Google runs and what you run
The single most useful mental model for GKE is the line drawn through the Kubernetes control plane. Google runs the API server, etcd, the scheduler, and the controller manager. You never see those processes, you cannot SSH to them, and there is no master node in your project's VM list. What you get is a stable API endpoint, a kubeconfig, and a service level objective. Google owns etcd backup and restore, certificate issuance and rotation, control-plane version upgrades, and the vertical resizing of the control plane as your object count grows. That last item is worth knowing about because it is not free: when GKE decides your control plane needs more capacity, it performs a resize that can make the API endpoint briefly unavailable. Workloads keep running - the kubelet does not need the API server to keep existing pods alive - but kubectl, admission webhooks that call out to the API, and any controller doing a watch will see a gap.
The flip side of a managed control plane is that the knobs are gone. You do not set API server flags, you do not choose the etcd compaction interval, you cannot install a custom audit policy or arbitrary feature gates, and alpha APIs are only available on short-lived alpha clusters that are excluded from upgrades and support. Admission control is the exception that matters: your own validating and mutating webhooks still run as pods in your cluster, which means a webhook that fails closed and has no healthy backends can wedge the entire API surface - including the reconciliation GKE itself needs during an upgrade.
Below the line, you own workloads, node configuration in Standard mode, networking design, IAM, and the resource requests that everything else keys off. GKE adds managed behaviour on your side of the line too: node auto-repair recreates a node whose kubelet has been unhealthy for several consecutive health checks, and node auto-upgrade keeps the node version inside the supported skew of the control plane. Both are on by default in most configurations, and both will delete and recreate VMs underneath you, which is the correct default and the reason anything holding state on a node's local disk is a liability.
Standard versus Autopilot - two responsibility boundaries
Standard and Autopilot are not tiers of the same product. They are two different places to draw the second line, the one between "node" and "pod", and they bill differently as a direct consequence.
In Standard, nodes are Compute Engine VMs in your project. You choose machine families, disk types, image type, and how many nodes exist. You are billed for those VMs whether or not a single pod is scheduled on them. That makes bin-packing your problem and your opportunity: a Standard cluster whose pods request half of what they use, or a quarter of what the node offers, quietly burns money on unallocated capacity. In exchange you keep every capability that requires touching a node: DaemonSets with host mounts, privileged containers, custom kernel parameters through node system configuration, local SSD, node SSH for debugging, and the ability to run a security agent that genuinely needs host visibility.
In Autopilot, the pod is the unit. You do not create node pools; you submit pods and GKE provisions the capacity underneath. Billing follows the pod's resource requests - CPU, memory, and ephemeral storage - for the pod's lifetime, so unallocated node capacity is Google's problem, not yours. Node-level operations are removed rather than discouraged: there is no node list to manage, no SSH, and the set of pod specifications that are accepted is narrower.
Where Autopilot bites
Three things surprise teams migrating in. First, requests are normalised. Autopilot enforces minimum requests per pod and permitted CPU-to-memory ratios per compute class, so a pod asking for 0.25 vCPU with 8 GiB of memory does not run as written - it is bumped to a legal shape, and you are billed for the bumped shape. Second, the workload patterns that assume node access simply fail admission: privileged containers, most host namespaces, and DaemonSets that expect to poke at the host are restricted. If your observability or security tooling is a privileged DaemonSet, check it against Autopilot's allow list before committing. Third, because you cannot over-commit against a node you did not buy, sloppy requests cost real money immediately rather than showing up later as low utilisation. Autopilot rewards accurate requests; Standard rewards dense packing. Choosing between them is mostly a question of which of those two disciplines your team can actually sustain.
Node pools - the unit of machine type and lifecycle
A node pool is a group of identical nodes backed by a managed instance group. Everything that is a property of a machine rather than a property of a pod lives on the pool: machine type and family, boot disk size and type, image type (Container-Optimized OS with containerd, or Ubuntu when you need something COS will not give you), the node service account, Shielded VM and metadata settings, taints and labels applied at node creation, autoscaling bounds, and the node version. Changing most of those means creating a new pool and migrating, not editing in place.
That is why real clusters end up with several pools rather than one. A typical shape is a small on-demand pool for system and singleton workloads, a general application pool, a memory-optimised pool for the JVM or cache tier, an accelerator pool with GPUs or TPUs and the corresponding taint so nothing else lands on expensive hardware, a spot pool for batch, and sometimes an Arm pool once the images are multi-arch. Each boundary you want to enforce with a taint, each machine shape you need, and each independent upgrade cadence is another pool.
Two mechanical details cause repeated confusion. The node count you configure is per zone, not per pool: a pool spanning three zones with a minimum of one node runs three nodes, and a maximum of ten runs up to thirty. And taints applied through the node pool configuration are reapplied when GKE recreates a node; taints you add manually with kubectl taint are not, so they vanish on the next repair or upgrade and workloads drift onto hardware they were supposed to avoid.
Zonal, multi-zonal, and regional clusters
The cluster's availability type is chosen at creation and cannot be changed afterwards, which makes it one of the few genuinely irreversible GKE decisions.
A zonal cluster has one control-plane replica in one zone. Nodes live in that same zone. If the zone has a problem, or the control plane is being upgraded, the API endpoint is unavailable. Running pods keep serving, because kubelet and kube-proxy do not need the API to maintain steady state, but during that window nothing reconciles: no deployments roll, no failed pods get rescheduled, the cluster autoscaler cannot add nodes, and Horizontal Pod Autoscaler decisions stop. A multi-zonal cluster keeps the single-replica control plane but spreads nodes across zones, which protects the data plane from a zone failure while leaving the control plane as a single point of failure.
A regional cluster replicates the control plane across three zones in the region behind one endpoint, with etcd replicated between them. That buys two things. Zone loss no longer takes the API away, and control-plane upgrades roll one replica at a time, so the endpoint stays served throughout. Node pools in a regional cluster replicate across the region's zones by default, which is where the cost surprise lives: the same per-zone node count now multiplies by three, and pod-to-pod traffic that crosses a zone boundary is charged as cross-zone traffic. Both are manageable - reduce per-zone counts, and use topology-aware routing or zone-aware service placement for chatty paths - but neither is automatic.
The practical rule is that anything with a real availability target should be regional, and zonal clusters belong to development, experiments, and CI where an hour of missing control plane is an inconvenience rather than an incident.
Four autoscalers that have to agree
GKE ships four independent scaling controllers, and most autoscaling incidents are really conflicts between them.
The cluster autoscaler operates on nodes. Its input is Pending pods: if a pod cannot be scheduled because no node has enough allocatable CPU, memory, or a matching label, taint toleration, or GPU, the autoscaler simulates adding a node to each eligible pool and grows the one that would fit the pod, within that pool's maximum. Scale-down is the mirror image and is far pickier - a node is removed only after it has been underutilised for a sustained period and every pod on it can be evicted elsewhere. Several things pin a node open forever: pods with no controlling workload object, pods using local storage, kube-system pods that lack a PodDisruptionBudget, and any pod whose PDB refuses the eviction. Critically, the cluster autoscaler only reacts to schedulability. If pods are Pending because the project is out of CPU quota or the pod IP range is exhausted, it will keep trying and never succeed, and the symptom looks identical from a dashboard.
Node auto-provisioning sits one level up: instead of scaling pools you defined, it creates and deletes node pools with machine shapes it selects from the pending pods' requests. It needs cluster-wide resource limits so it knows when to stop, and it is most valuable when workload shapes are unpredictable - a research cluster where jobs ask for wildly different CPU-to-memory ratios or specific accelerators. It is least valuable when you have carefully curated pools and want them respected.
HPA scales replica counts against a metric, most often CPU utilisation expressed as a percentage of the pod's request. VPA changes the requests themselves, in recommendation-only mode, at pod admission, or continuously.
The VPA versus HPA fight
Point both at CPU and you have built an oscillator. HPA measures utilisation as usage divided by request. VPA notices a pod using less than it requested and lowers the request. Lowering the request raises the measured utilisation percentage without anything about the actual load changing, so HPA scales out. More replicas means less load per replica, so VPA lowers requests again, so utilisation climbs again, so HPA scales out again. The cluster ends up with many small starved pods and a replica count with no relationship to demand.
The rule is to keep them on disjoint signals. Let HPA own the replica count against CPU or, better, a business metric such as queue depth or requests per second delivered through Managed Service for Prometheus. Let VPA own memory, or run it in recommendation mode only and feed its output into your manifests through review rather than letting it act. Remember also that VPA's continuous mode has historically resized by evicting and recreating pods, which interacts with PDBs and with anything that is expensive to restart.
Spot node pools and designing for eviction
Spot VMs are the same machine types at a large discount with one condition attached: Compute Engine can reclaim them whenever it needs the capacity, giving roughly thirty seconds of notice. Preemptible VMs are the older variant of the same idea with a hard maximum lifetime. GKE makes spot pools workable by labelling and tainting them automatically, so nothing lands there unless it tolerates the spot taint - a good default, because it means adding a spot pool cannot accidentally move your database onto reclaimable hardware.
Designing for spot is designing for eviction as a normal event rather than an incident. Keep terminationGracePeriodSeconds inside the notice window, because a pod that wants two minutes to drain will not get them. Handle SIGTERM properly: stop accepting new work, finish or checkpoint what is in flight, exit. Spread replicas with topology spread constraints so a single zone's reclamation does not take a whole service. Set PodDisruptionBudgets, but set them loosely enough that voluntary evictions can still proceed. And never put the system pool, singletons, or anything with a persistent disk it cannot cheaply reattach onto spot.
The failure mode people underestimate is capacity, not preemption. Spot capacity for a given machine type in a given zone can simply be unavailable, and the cluster autoscaler will not silently fall back to on-demand for you. The standard mitigation is to run both a spot pool and a smaller on-demand pool that can serve the same pods, and to configure the autoscaler's expander so the spot pool is preferred while the on-demand pool remains a legal answer. Batch and stateless serving with generous replica counts are excellent spot candidates; stateful sets, long non-checkpointed training runs, and anything whose restart costs more than the discount saves are not.
End-to-end flow
End-to-end: Autopilot cluster created. Pods get workload identity mapped to GCP SA reading secrets. VPC-native means pods reachable inside VPC. Gateway routes external traffic with WAF policies. Binary Authorization ensures only signed images ship. Config Sync applies GitOps. Metrics + logs in Cloud Logging. Cluster autoscales; regional for HA.
VPC-native networking and the IP plan that bites later
A VPC-native cluster gives pods real VPC addresses through alias IP ranges rather than routing pod traffic over cluster-managed routes. The subnet carries a primary range for nodes and secondary ranges for pods and for services. Because pod addresses are first-class in the VPC, VPC firewall rules apply to pod traffic, on-premises peers can reach pods directly over interconnect or peering, and load balancers can target pods through network endpoint groups instead of bouncing through a node port. That last property, container-native load balancing, removes a hop and a round of source NAT, gives health checks a view of the actual pod rather than the node, and makes session affinity behave the way the documentation says it does.
The cost of all this is address planning, and it is unforgiving because most of it is fixed at creation time. Each node is carved a slice of the pod secondary range sized by the pool's maximum pods per node - the default of 110 pods per node consumes a /24 per node, while capping at 32 pods per node needs only a /26. Multiply by the largest node count your autoscaler is allowed to reach, add the services range, and add the /28 the managed control plane needs on a private cluster, which must not collide with anything you now or later peer with. Get this wrong and the failure is abrupt: the cluster autoscaler tries to add a node, there is no free pod CIDR to assign, the node never joins, and pods stay Pending with an error that says nothing about addresses. Additional pod ranges can be attached to node pools later, but a node's maximum pods per node cannot be changed after the pool exists, so the cheapest fix is usually a new pool.
Dataplane V2
GKE Dataplane V2 replaces the iptables-based kube-proxy data path with an eBPF implementation derived from Cilium. Service resolution and policy enforcement move into eBPF programs attached in the kernel, which scales far better than a linearly growing iptables ruleset once a cluster has thousands of services and endpoints. It also brings network policy enforcement and network policy logging into the platform rather than requiring a separate policy agent. The tradeoffs are real: it is selected at cluster creation, your accumulated troubleshooting instincts around iptables-save and kube-proxy no longer apply, and any third-party CNI or agent that assumes the classic data path needs verifying.
Gateway versus Ingress
GKE Ingress programs a Google Cloud HTTP(S) load balancer from an Ingress object, with the interesting configuration - timeouts, connection draining, CDN, IAP, security policy binding, session affinity - expressed through BackendConfig and FrontendConfig custom resources plus a layer of annotations. It works, and it is what most existing clusters use. The Gateway API is the successor and is worth choosing for new work. It splits the infrastructure concern from the routing concern: a platform team owns the Gateway and its GatewayClass (external global, external regional, or internal), while application teams own HTTPRoutes that attach to it. Header-based routing and weighted traffic splitting for canaries become first-class fields rather than annotation conventions, policies attach through typed objects such as GCPBackendPolicy and HealthCheckPolicy, and multi-cluster gateways let one address front services in several clusters. If you need a WAF, rate limiting, or DDoS policy in front of that load balancer, attach a Cloud Armor security policy to the backend rather than reimplementing it in the mesh - see Google Cloud Armor architecture for how those policies evaluate.
Workload Identity instead of node service accounts
The legacy way for a pod to call a Google Cloud API was to inherit the node's attached service account through the Compute Engine metadata server. It works, and it is wrong, because the granularity is the node. Every pod scheduled onto that node - including anything an attacker manages to run there - gets exactly the same credentials as the most privileged workload on it. Blast radius equals node, and it grows every time someone adds a role to make one deployment work.
Workload Identity fixes the granularity by binding a Kubernetes ServiceAccount to a Google Cloud identity. The GKE metadata server runs as a DaemonSet, intercepts the metadata endpoint on a per-pod basis, and issues tokens for the identity mapped to that pod's KSA. Application Default Credentials in the Google client libraries then work with no key file anywhere - which is the real prize, since exported service account keys are the credential most likely to end up in a repository.
# 1. Enable the workload identity pool on the cluster and the node pool.
gcloud container clusters update prod \
--workload-pool=PROJECT_ID.svc.id.goog
# 2. Let the Kubernetes SA impersonate the Google SA.
gcloud iam service-accounts add-iam-policy-binding \
ingest@PROJECT_ID.iam.gserviceaccount.com \
--role=roles/iam.workloadIdentityUser \
--member="serviceAccount:PROJECT_ID.svc.id.goog[data/ingest-sa]"
# 3. Point the Kubernetes SA at it.
kubectl annotate serviceaccount ingest-sa -n data \
iam.gke.io/gcp-service-account=ingest@PROJECT_ID.iam.gserviceaccount.comNewer projects can skip the intermediary entirely and grant IAM roles directly to the Kubernetes identity as a principal in the workload identity pool, which removes one object and one indirection per workload. Either way, the node pool still needs its own service account, because the kubelet uses it to pull images and write logs and metrics; give it exactly Artifact Registry reader plus the logging and monitoring writer roles and nothing else.
Three failure modes account for most of the debugging time. Workload Identity enabled on the cluster but not on a node pool means pods on that pool silently keep using the node identity - and it works, which is why nobody notices until an audit. A missing or misspelled binding produces a 403 from the metadata server that mentions the Google service account, not the Kubernetes one, so people go looking in the wrong place. And a network policy or an hostNetwork pod that cannot reach the link-local metadata address fails with a timeout rather than a permission error. In all three cases the fastest diagnostic is to exec into the pod and curl the metadata server's service-accounts endpoint to see which identity it actually believes it has.
Upgrades - channels, surge, windows, and the PDB that blocks you
GKE upgrades are continuous rather than an event you schedule once a year, and the controls exist to shape when and how fast, not whether. A release channel - rapid, regular, stable, or extended - fixes the version cadence you sign up for: rapid gets new minor versions first and is where you validate, stable trails and is where production usually lives, extended stretches the window for a version you need to stay on longer. You can pin a specific version within a channel, but you cannot opt out of upgrading indefinitely.
The control plane upgrades first; nodes follow, and must stay within the supported version skew of the control plane. Maintenance windows constrain when automatic upgrades may start, and maintenance exclusions block them entirely for a bounded period - the correct tool for a retail freeze or a regulatory audit, and one that must be set before the freeze rather than during it.
Node upgrades happen per node pool, using one of two strategies. Surge upgrade is controlled by two numbers per zone: maxSurge, how many extra nodes may be created above the pool's size, and maxUnavailable, how many existing nodes may be down at once. Surge 1 with unavailable 0 adds one node, drains one old node, and repeats - the safest setting and, on a pool of eighty nodes, a very long afternoon. Raising surge trades quota and brief extra cost for wall-clock time. Blue-green upgrade instead provisions a complete new set of nodes, moves workloads over, holds a soak period during which you can roll back to the still-existing old nodes, and only then deletes them. It costs double capacity for the duration and is the right choice for workloads where a bad node image needs a fast, whole-pool reversal.
Every node drain evicts pods through the Eviction API, which means it must satisfy PodDisruptionBudgets - and this is where upgrades stop dead. A PDB with minAvailable equal to the Deployment's replica count can never allow a voluntary eviction; neither can maxUnavailable: 0; and the single most common instance is a one-replica Deployment with minAvailable: 1, which reads as a safety measure and functions as a permanent block. The drain retries, the upgrade stalls, and eventually the node is force-recreated anyway or the operation times out - the worst of both outcomes, since you got the disruption without the graceful handover. Audit PDBs the same way you audit resource requests: every PDB must leave at least one pod evictable at the workload's minimum replica count, and single-replica workloads that genuinely cannot tolerate a restart need a second replica, not a stricter budget.
Storage - persistent disks, storage classes, and zone affinity
GKE provisions block storage through the Compute Engine Persistent Disk CSI driver. The modern default StorageClass is backed by balanced persistent disks with ReadWriteOnce access; older clusters may still carry an in-tree class backed by standard spinning-disk-class PDs, which is a performance trap for anything latency sensitive. SSD classes exist for databases, and on newer machine families Hyperdisk decouples provisioned IOPS and throughput from capacity, which matters because on classic PDs performance scales with the disk's size - a 10 GiB volume is slow no matter what is on it - and is additionally capped by the VM's own limits.
The property that causes the most confusing incidents is zone affinity. A zonal persistent disk exists in exactly one zone, and a pod that mounts it can only run on a node in that zone. With volumeBindingMode: WaitForFirstConsumer, the scheduler picks the node first and the disk is created in the zone it chose, which is almost always what you want. With Immediate binding, the disk is created as soon as the claim exists, in whatever zone the provisioner picks, and if that zone later has no schedulable capacity in the right node pool, the pod stays Pending forever with an error about volume node affinity conflicts. Regional persistent disks replicate synchronously across two zones and let a StatefulSet survive the loss of one, at the cost of write latency and stricter attach rules.
Volumes can grow online when the class sets allowVolumeExpansion; they can never shrink. For shared read-write-many access, Filestore through its CSI driver is the managed NFS answer, and the Cloud Storage FUSE driver mounts buckets for read-heavy workloads such as model weights or datasets - convenient, but with object-store semantics and latency, not file-system semantics. Local SSD is available on Standard node pools for scratch and cache and is genuinely fast, with the obvious condition that its contents die with the node - which, given auto-repair, auto-upgrade, autoscaler scale-down, and spot reclamation, happens more often than people plan for.
Observability and cost attribution
Cloud Logging and Cloud Monitoring integration is on by default, and the first tuning decision is which components you actually want. Container logs, system logs, and the optional control-plane logs - API server, scheduler, controller manager - are separately toggleable, and the control-plane audit stream in particular is both the thing you want during a security investigation and a substantial log volume the rest of the time. Log volume is a real line item, so route deliberately: exclusion filters at the sink keep debug chatter out of Logging, and a log sink to Cloud Storage or BigQuery keeps high-volume streams queryable without paying to index them.
For metrics, GKE's built-in system metrics cover nodes, pods, and control-plane health, while Managed Service for Prometheus handles application metrics with PromQL and none of the storage operations. Managed collection scrapes your pods based on PodMonitoring resources, which keeps the familiar Prometheus workflow and exporters while removing the part of Prometheus that pages you at three in the morning about retention.
Cost attribution is the piece teams most often discover too late. GKE cost allocation, enabled on the cluster, breaks node cost down by namespace and by workload label in the billing export, apportioning each node's cost across the pods on it according to their requests. That converts a single opaque GKE line item into per-team numbers you can actually act on - and it depends entirely on requests being honest, which is the same discipline HPA, the cluster autoscaler, and Autopilot billing all separately demand. The gap between requested and actually used resources is the direct measure of your bin-packing waste; it is the number that tells you whether the answer is smaller requests, a different machine shape, or a move to Autopilot. The crude alternative, one node pool per team, does produce clean invoices, but at the price of the utilisation that made a shared cluster worth building.
Choosing the shape
Most GKE design work collapses into four decisions made early, in this order. Regional or zonal: regional for anything with an availability target, and the choice is permanent. Autopilot or Standard: Autopilot unless a workload genuinely needs node-level access or you have the operational maturity to bin-pack Standard well. The IP plan: size the pod range against the autoscaler's maximum, not today's node count, and reserve the control-plane range where it will not collide with future peering. And the identity model: Workload Identity from the first cluster, because retrofitting it means auditing every deployment that quietly depended on the node service account.
GKE is also not always the answer. A stateless HTTP service with bursty traffic and no need for cluster primitives is usually cheaper and simpler on Cloud Run, which scales to zero and has no cluster to upgrade. Event-driven glue belongs on Cloud Functions or Eventarc. Managed Spark and Hadoop have Dataproc. GKE earns its operational surface when you need the Kubernetes API itself - operators, custom resources, precise scheduling against GPUs or Arm or spot, service mesh, or portability across clouds - and when you have enough workloads that a shared, well-packed cluster beats per-service serverless pricing.