Config + Secrets Manager Integration, in depth: secret references as a config layer, a redacting Secret type, Spring sm@ and safe rotation for ADK Java

References, not values

Start by deciding what a config file may contain. The rule is simple: never a secret value, only a pointer to one. A reference names the store, the secret and, ideally, the version:

# agent.yaml (committed; safe to read in code review)
model:
  name: gemini-2.5-flash
  apiKey: secret://projects/acme-prod/secrets/gemini-api-key/versions/7
tools:
  crm:
    baseUrl: https://crm.internal.example.com
    token: secret://projects/acme-prod/secrets/crm-token/versions/latest
  db:
    url: jdbc:postgresql://10.0.0.5/agent
    password: secret://projects/acme-prod/secrets/agent-db-password/versions/3

The secret:// scheme here is this page's own convention, not a library feature; any unambiguous prefix works. Its payload is the Secret Manager resource name, so there is no second naming system to keep in sync. Because references are ordinary strings, all the layering rules you already have still apply: an environment can override tools.crm.token with a different reference, and precedence works exactly as described in configuration layering and precedence. The only new step is that, after layers merge, any value that is still a reference is resolved.

A Secret type that cannot be printed

The second rule is that a resolved secret is never a String in your config object. A String ends up in toString() output, exception messages, debug dumps of the config and structured logs that serialise whole objects. A small wrapper type closes all of those paths at once:

public final class Secret {
  private final char[] value;
  private final String ref;        // resource name incl. version, safe to log
  private Secret(char[] value, String ref) { this.value = value; this.ref = ref; }

  static Secret of(String value, String ref) { return new Secret(value.toCharArray(), ref); }

  /** The only way out. Call it at the client boundary, nowhere else. */
  public String reveal() { return new String(value); }
  public String ref() { return ref; }

  @Override public String toString() { return "Secret[" + ref + "]"; }
  @Override public boolean equals(Object o) { return this == o; }
  @Override public int hashCode() { return System.identityHashCode(this); }
}

public record ModelConfig(String name, Secret apiKey) {}
public record AgentConfig(ModelConfig model, Map<String, ToolConfig> tools) {}

Records print their components, so AgentConfig.toString() now shows Secret[...] with a resource name instead of a key. Identity-based equals stops a secret from being used as a map key or compared in a test assertion message that prints both sides. A char[] does not make the JVM forget the value, because the client libraries will copy it into strings anyway; the type exists to control printing and to make every use of reveal() easy to find in review.

The resolver: parallel, verified, recorded

The resolver walks the merged tree, collects every reference, fetches them in parallel with a deadline, verifies each payload's checksum, and records provenance. The Secret Manager calls below follow Google's Java sample for accessing a secret version:

public final class SecretResolver implements AutoCloseable {
  private static final Pattern REF = Pattern.compile(
      "secret://projects/([^/]+)/secrets/([^/]+)/versions/([^/]+)");
  private final SecretManagerServiceClient client;
  private final ExecutorService pool = Executors.newVirtualThreadPerTaskExecutor(); // Java 21+

  public SecretResolver() throws IOException { client = SecretManagerServiceClient.create(); }

  public Map<String, Secret> resolveAll(Map<String, String> refsByKey, Duration deadline)
      throws InterruptedException {
    Map<String, Future<Secret>> pending = new LinkedHashMap<>();
    refsByKey.forEach((key, ref) -> pending.put(key, pool.submit(() -> fetch(ref))));
    Map<String, Secret> out = new LinkedHashMap<>();
    List<String> errors = new ArrayList<>();
    long end = System.nanoTime() + deadline.toNanos();
    for (var e : pending.entrySet()) {
      try {
        out.put(e.getKey(), e.getValue().get(Math.max(0, end - System.nanoTime()), NANOSECONDS));
      } catch (ExecutionException | TimeoutException ex) {
        Throwable cause = ex instanceof ExecutionException ? ex.getCause() : ex;
        errors.add(e.getKey() + " <- " + refsByKey.get(e.getKey())
            + ": " + cause.getClass().getSimpleName());          // never the payload
      }
    }
    if (!errors.isEmpty()) throw new IllegalStateException("unresolved secrets: " + errors);
    return out;
  }

  private Secret fetch(String ref) {
    Matcher m = REF.matcher(ref);
    if (!m.matches()) throw new IllegalArgumentException("bad secret reference");
    var name = SecretVersionName.of(m.group(1), m.group(2), m.group(3));
    AccessSecretVersionResponse resp = client.accessSecretVersion(name);
    byte[] data = resp.getPayload().getData().toByteArray();
    CRC32C crc = new CRC32C();
    crc.update(data, 0, data.length);
    if (resp.getPayload().getDataCrc32C() != crc.getValue())
      throw new IllegalStateException("checksum mismatch");
    // record the name the API returns; check once that it names a numeric version for latest
    return Secret.of(resp.getPayload().getData().toStringUtf8(), resp.getName());
  }

  @Override public void close() { client.close(); pool.shutdown(); }
}

Three details matter. Collect all failures before throwing, so a deploy with three missing permissions fails once with three lines instead of three times; this is the same collect-all discipline as configuration validation at startup. Error messages name the key and reference, never the value. And the Secret records the version name the API returned rather than the reference. The access API documents that name only as a version resource name, so check once in your project that a latest reference comes back as a numeric version; if it does not, provenance for latest references needs another source.

Wiring the snapshot into ADK

With a snapshot in hand, construct clients from it, and only there call reveal(). For the model, ADK Java's Gemini builder accepts an API key directly, so the key never has to pass through an environment variable:

AgentConfig cfg = ConfigLoader.load(Path.of("agent.yaml"), System.getenv(), resolver);
Gemini model = Gemini.builder()
    .modelName(cfg.model().name())
    .apiKey(cfg.model().apiKey().reveal())
    .build();
LlmAgent agent = LlmAgent.builder()
    .name("support").model(model)
    .instruction("Help with orders.")
    .tools(OrderTools.create(cfg.tools().get("crm")))   // tool gets its client, not a key
    .build();
log.info("config loaded: {}", cfg);  // prints Secret[...versions/7], not the key

On Vertex AI, use application default credentials and the builder's vertexCredentials or apiClient options instead of a key; the best secret is one you do not have. Tools should receive constructed clients, not raw keys, so the only code that sees a value is the factory that builds the client.

The Spring route: sm@ config data

If the agent runs inside Spring Boot, Spring Framework on Google Cloud already implements the reference layer. Add spring-cloud-gcp-starter-secretmanager and enable the config data import. Since version 6.0.0 the recommended syntax is sm@; the older sm:// form still works but logs a warning:

spring.config.import=sm@
agent.model.api-key=${sm@projects/acme-prod/secrets/gemini-api-key/versions/7}
agent.tools.crm.token=${sm@crm-token}
# regional secret, latest version, default project
agent.tools.db.password=${sm@locations/europe-west1/agent-db-password}

Short forms take the project from spring.cloud.gcp.secretmanager.project-id or from application default credentials. Two behaviours affect agents. A missing secret fails startup unless you set spring.cloud.gcp.secretmanager.allow-default-secret=true, which lets a ${...:DEFAULT} fallback apply; leave it off in production, because a silent default key is worse than a failed start. Refresh through the actuator's /actuator/refresh endpoint updates only @ConfigurationProperties beans annotated with @RefreshScope, never @Value fields, and it does not rebuild an agent that has already captured a key in its model client. Wire agents as described in ADK Java with Spring and rebuild the client when the properties change.

Pinned versions, latest and reload

Whether to reference a pinned version or latest is the decision that determines how rotation behaves. Cloud Run makes the trade-off explicit in its own docs: secrets exposed as environment variables are resolved at instance startup, so Google recommends pinning a version; secrets mounted as volumes are fetched from Secret Manager when read, so they follow latest.

A sound default for agents is to pin keys that gate the model and data stores, so a rotation is a reviewed config change that rolls out with the deploy, and to use latest with reload only for credentials rotated automatically by another system. When you reload, build a complete new snapshot with the same resolver, validate it, and swap it atomically, so invocations already in flight keep the snapshot they started with; dynamic configuration reload shows the swap. Keep the previous secret version enabled until no instance reports it in its provenance log.

Worked example: rotating a tool token

Worked example: rotating the CRM token without an outage.

1. The CRM issues a second token. You add it to Secret Manager as version 5 of crm-token; version 4 stays enabled.
2. The reference is versions/latest with a five-minute reload. Within one interval, each instance builds a new snapshot, the CRM client factory builds a client with version 5, and the provenance log records the version each instance loaded (this is why the numeric-name check above matters).
3. You query the logs until every instance reports version 5, then revoke the old token at the CRM and disable version 4.
4. If the new token had been wrong, tool calls would fail with authentication errors. Disabling version 5 makes latest resolve to version 4 again on the next reload, with no deploy.

Failure modes

Trade-offs

A resolver in your own code costs a dependency on the Secret Manager client and a service account permission per secret, in exchange for provenance, checksums and one behaviour across Spring and plain Java. Platform injection, such as Cloud Run environment variables, needs no code but hides which version an instance holds unless you pin. Spring's sm@ layer is the least code inside Spring, but its refresh model does not reach clients that captured a key. Fetching at startup makes Secret Manager a startup dependency; fetching lazily moves the failure into the first tool call of a user's conversation, which is usually worse. For tests, give the loader a map-backed fake resolver so no test ever needs real credentials.

What to do next

1. Grep your agent for System.getenv and literal keys, and list every credential it uses.
2. Move each one into config as a reference, choosing pinned or latest deliberately for each.
3. Introduce a Secret type and change config records to use it; let the compiler find every call site.
4. Resolve all references at startup, in parallel with a deadline, verifying checksums and collecting all errors.
5. Build the model and tool clients from the snapshot, and log the snapshot to confirm only references appear.
6. Grant each service account accessor rights on exactly the secrets it references.
7. Write a rotation runbook that uses the provenance log as the gate for disabling old versions, and rehearse it in staging.