Prompt assembly

The orchestrator combines: system instruction, session history, tool descriptions, and the current turn. It applies context-window trimming rules if the total exceeds the model's limit.

Advertisement

Model client abstraction

The runtime never talks to Gemini/OpenAI/etc directly — it goes through a ModelClient. You can swap this out for testing or to route requests through a proxy.

public class LoggingModelClient implements ModelClient {
  private final ModelClient delegate;
  @Override
  public ModelResponse generate(Request req) {
    log.info("prompt_tokens={}", req.tokenCount());
    ModelResponse r = delegate.generate(req);
    log.info("completion_tokens={}", r.tokenCount());
    return r;
  }
}
Advertisement

Retry semantics

By default the runtime retries transient model errors (5xx, quota) with exponential backoff. It does NOT retry 4xx — those indicate a bad prompt or missing auth and should surface fast.