Config Connector, in depth: managing Google Cloud resources as Kubernetes objects

The model: cloud resources as Kubernetes objects

Kubernetes already has a general pattern for "make the world look like this object": a custom resource definition (CRD) declares a new kind, and a controller watches objects of that kind and reconciles them. Config Connector ships one CRD per supported Google Cloud resource type, grouped by API, for example storage.cnrm.cloud.google.com or pubsub.cnrm.cloud.google.com. The format is called the Kubernetes Resource Model (KRM): every object has metadata, a spec describing what you want, and a status that the controller writes.

Three consequences follow. First, every Kubernetes tool works on cloud infrastructure: RBAC decides who may create a SQLInstance, admission policies can reject a public bucket, kubectl get lists your topics, and GitOps tools sync it all from Git. Second, the object's namespace decides which Google Cloud project the resource lands in, so the namespace becomes your tenancy boundary. Third, by default the object's metadata.name is the cloud resource's name; when the cloud name has characters or length rules Kubernetes names cannot meet, many kinds accept spec.resourceID to set it explicitly.

Architecture and identity

The runtime lives in the cnrm-system namespace. A manager process runs controllers for every installed CRD; in namespaced mode there is one controller manager per watched namespace, each with its own identity. A webhook validates and defaults objects on admission, and a deletion-defender component guards against cloud resources being deleted when Config Connector CRDs are removed.

Identity is the part to get right first. Controllers authenticate to Google Cloud as an IAM service account through GKE Workload Identity: the Kubernetes service account cnrm-controller-manager (or cnrm-controller-manager-NAMESPACE in namespaced mode) is allowed to impersonate a Google service account, and that account's IAM roles bound what Config Connector can do. Whatever the YAML says, the controller cannot exceed those roles, which makes the service account the real blast-radius control. See workload identity for the general pattern and Google Cloud IAM for role design.

Installing: add-on, operator or Config Controller

There are three ways to get Config Connector. The GKE add-on is a checkbox on a Standard cluster, but Google's own documentation warns that its version "is often behind by up to 12 months or more", so new resource kinds and fixes arrive late. The manual operator install is the recommended route: you install the operator, then create a ConfigConnector object that picks the mode. Config Controller is a hosted offering where Google runs the cluster and upgrades Config Connector for you, which suits teams that want the API without operating a cluster.

Mode is the important choice. Cluster mode uses a single cluster-wide service account; every namespace acts with the same permissions. Namespaced mode, recommended for most uses, gives each namespace its own controller and service account, so team A's manifests physically cannot touch team B's project.

apiVersion: core.cnrm.cloud.google.com/v1beta1
kind: ConfigConnector
metadata:
  name: configconnector.core.cnrm.cloud.google.com
spec:
  mode: namespaced
  stateIntoSpec: Absent
---
apiVersion: core.cnrm.cloud.google.com/v1beta1
kind: ConfigConnectorContext
metadata:
  name: configconnectorcontext.core.cnrm.cloud.google.com
  namespace: team-payments
spec:
  googleServiceAccount: "kcc-payments@platform-host.iam.gserviceaccount.com"
  stateIntoSpec: Absent

Two commands finish the wiring: allow the per-namespace Kubernetes service account to impersonate the Google account, and annotate the namespace with the target project.

gcloud iam service-accounts add-iam-policy-binding \
  kcc-payments@platform-host.iam.gserviceaccount.com \
  --member="serviceAccount:platform-host.svc.id.goog[cnrm-system/cnrm-controller-manager-team-payments]" \
  --role="roles/iam.workloadIdentityUser"

kubectl annotate namespace team-payments \
  cnrm.cloud.google.com/project-id=payments-prod

Anatomy of a resource and its references

A Config Connector object reads like any Kubernetes object. The example below creates a bucket and grants a service account read access to it. Notice that the IAM object does not repeat the bucket's name as a string; it points at the other Kubernetes object with a resourceRef.

apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  name: payments-prod-receipts
  namespace: team-payments
  annotations:
    cnrm.cloud.google.com/deletion-policy: abandon
spec:
  location: EU
  uniformBucketLevelAccess: true
  versioning:
    enabled: true
---
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicyMember
metadata:
  name: receipts-reader
  namespace: team-payments
spec:
  member: serviceAccount:receipts-api@payments-prod.iam.gserviceaccount.com
  role: roles/storage.objectViewer
  resourceRef:
    kind: StorageBucket
    name: payments-prod-receipts

References are how dependency ordering works. Config Connector has no global plan; each object reconciles independently. If the IAM member is applied before the bucket exists, its controller sees an unresolved reference, reports a not-ready condition and retries later. Apply the whole directory at once and the graph converges on its own, usually within a few reconcile cycles. Most reference fields also accept an external form, a plain identifier for a resource Config Connector does not manage.

Reconciliation, drift and state-into-spec

Each controller reconciles an object whenever it changes and again after a jittered interval whose average is the per-kind default listed on that resource's reference page. Since version 1.102 you can override it per object with cnrm.cloud.google.com/reconcile-interval-in-seconds; a value of 0 stops periodic reconciles once the object reaches UpToDate, which trades drift correction for API quota. Google's best-practice guide suggests raising the interval, for example to an hour, when large fleets hit API quota errors.

On each pass the controller reads the live resource, compares it with the spec and patches the differences. That means console edits are reverted: if an engineer disables bucket versioning by hand, the next reconcile turns it back on. That is the point of the tool, and also the source of most surprises.

The subtle setting is state-into-spec. Many fields are optional; the API fills them with defaults. With Merge, after a successful reconcile the controller copies those server-side values back into your spec, so the object in the cluster no longer matches the file in Git and a GitOps tool may fight it. With Absent, unspecified fields stay out of the spec and are left alone. Google recommends Absent; Merge is the default for CRDs supported in version 1.113.0 and earlier, which is why the install above sets stateIntoSpec: Absent explicitly, and why the best-practice guide asks for the cnrm.cloud.google.com/state-into-spec: absent annotation on ContainerCluster and ContainerNodePool.

Worked example: a team's Pub/Sub pipeline

Take a concrete case. The payments team needs an event pipeline: a topic, a dead-letter topic, a push subscription to a Cloud Run service and a service account that may publish. The platform team owns the cluster; the payments team owns a directory in Git.

1. The platform team creates namespace team-payments, the ConfigConnectorContext above, and grants kcc-payments only the roles it needs in project payments-prod: Pub/Sub admin, service-account admin, and project IAM admin if the team must bind project-level roles. Nothing in other projects.
2. The team commits four objects: PubSubTopic orders, PubSubTopic orders-dlq, IAMServiceAccount orders-publisher, and a PubSubSubscription whose topicRef and dead-letter policy reference the two topics.
3. GitOps syncs the directory. The topics and service account become ready on the first pass; the subscription may report a dependency-not-ready reason until both topics exist, then succeeds.
4. They add an IAMPolicyMember granting roles/pubsub.publisher on the topic to the new service account, referenced by resourceRef. The audit trail for that grant is now a Git commit plus a Kubernetes event.
5. Months later someone raises the subscription's ack deadline in the console during an incident. The next reconcile sets it back. The fix is a pull request, not a console edit, and the on-call guide says so.

kubectl -n team-payments get pubsubsubscription orders-push -o yaml | yq '.status'
# conditions:
# - type: Ready
#   status: "True"
#   reason: UpToDate
#   message: The resource is up to date
kubectl -n team-payments describe pubsubsubscription orders-push   # recent events

Wait on readiness in pipelines with kubectl wait --for=condition=Ready pubsubsubscription/orders-push --timeout=10m rather than sleeping; it fails loudly when a reference never resolves.

Adopting existing resources and deleting safely

Most organisations start with resources that already exist. When you apply an object whose name matches an existing resource in the target project, folder or organisation, Config Connector acquires it and starts managing it. That is convenient and dangerous in equal measure: an object that is missing a field you care about will not delete the field, but a field set differently in YAML will overwrite production on the first reconcile.

The safe path is to export first. config-connector bulk-export uses Cloud Asset Inventory to discover resources in a project, folder or organisation and writes Config Connector YAML for the kinds it supports (config-connector print-resources lists them). Commit the export, review it, add deletion-policy: abandon to anything stateful, and apply it to a namespace whose service account is read-mostly until you trust the diff.

Going the other way, removing an object deletes the cloud resource by default. Before moving a resource between namespaces, renaming it or uninstalling Config Connector, set cnrm.cloud.google.com/deletion-policy: abandon and let it reconcile, so deleting the Kubernetes object leaves the bucket, database or Spanner instance in place. Google's guide warns that recreating resources such as SpannerInstance or BigtableTable loses data.

Failure modes

• Two owners, one resource. Terraform and Config Connector, or two namespaces, both manage the same bucket and overwrite each other every cycle. Pick one owner per resource. The management-conflict-prevention-policy: resource annotation exists for this, but the default is none, so do not assume protection.
• Spec churn under GitOps. With Merge, defaults written into spec make Argo CD or Config Sync report the object out of sync forever, or revert values the API needs. Use Absent.
• Accidental deletion. A refactor that renames a directory, a namespace delete, or a GitOps prune removes objects and therefore cloud resources. Abandon policies on stateful kinds, plus admission rules requiring them, are the guard.
• Permission gaps. The object reports an update failure with a 403 in the message. The fix is a role on the controller's service account, never broadening it to Owner because one kind failed.
• Immutable fields. Changing a field the API cannot update in place (many GKE cluster settings, for example) produces an update failure rather than a silent recreate. Plan replacements explicitly; Google recommends separate ContainerNodePool objects instead of node config on the cluster.
• Stale add-on. A feature in the docs does not exist on your cluster because the add-on lags. Check the installed version before debugging the YAML.

Operating it, and when to choose it

Run Config Connector like any controller platform. Alert on objects whose Ready condition is not True for longer than a few reconcile intervals, grouped by reason; that single signal catches permissions, quota and reference errors. Keep controller logs and Kubernetes events in your central logging. Upgrade on a cadence with a staging cluster that applies the same Git tree, because new versions occasionally change defaults or how fields are read.

Put policy in front of it: an admission controller (Gatekeeper or Kyverno) can require uniformBucketLevelAccess, forbid allUsers members and require abandon policies on databases. This is the real advantage over a CI-only tool: the rule is enforced at the API server whoever applies the object.

Many organisations use both: Terraform bootstraps folders, projects and the management cluster, and Config Connector handles per-team resources after that. The model generalises; config drift reconciliation covers the trade-offs of continuous correction, and GKE covers the cluster you will be running it on.

What to do next

1. Decide the tenancy model: one namespace per team and project, namespaced mode, one Google service account per namespace with only the roles that team needs.
2. Install with the operator (or use Config Controller) rather than the add-on, and record the installed version.
3. Set stateIntoSpec: Absent on the ConfigConnector and every ConfigConnectorContext.
4. Create one bucket and one IAM binding in a sandbox namespace; delete the bucket's object with and without the abandon annotation and watch what happens.
5. Run config-connector bulk-export on a non-production project and review what it would adopt before applying anything.
6. Add admission policies requiring abandon on stateful kinds and forbidding public members.
7. Alert on Ready not True by reason, and write the runbook line: fix drift in Git, not in the console.