The stream

Stream stream = llm.generateStream(req);
stream.forEach(delta -> {
  switch (delta.type()) {
    case TEXT -> writeToUser(delta.text());
    case TOOL_CALL -> queueForDispatch(delta.toolCall());
    case USAGE -> recordCost(delta.usage());
  }
});
Advertisement

Text deltas

Fragments of the model's text output. Concatenate to reconstruct. Emitted as they arrive.

Advertisement

Tool-call deltas

The model requested a tool. This may arrive mid-stream. The runtime dispatches while the stream is still open.