Hook after model call

@Override
protected void afterModelCall(ModelResponse resp, AgentContext ctx) {
  tracer.span("model.call")
      .setAttribute("prompt_tokens", resp.usage().promptTokens())
      .setAttribute("completion_tokens", resp.usage().completionTokens())
      .setAttribute("prompt_hash", hash(ctx.effectivePrompt()));
}
Advertisement

Store full prompts sampled

Full prompts to S3 or a log bucket, tagged by trace_id. Sample 1% for cost. Store 100% in dev.

Advertisement

Hash + reference

Instead of storing the prompt in the trace (huge), store a hash + a reference URL. Trace stays small.