Skip to main content

Streaming

Classic RAG streams generated text after retrieval and optional reranking have completed.

Pipeline streaming

Explicit history is supported:

Session streaming

The session commits the concatenated answer only after the stream is exhausted. Closing or abandoning the iterator does not leave a partial turn in history.

Provider support

The built-in LLM integrations implement the streaming contract:
  • Ollama
  • OpenAI and compatible endpoints
  • LM Studio
  • Mistral
  • Gemini
  • AWS Bedrock
Custom LLMs implement generate_streaming(input) and yield strings.

Current response contract

generate() returns structured events, timings, sources, and a trace ID. stream() currently returns text chunks only. Transport-level streaming event models belong to future delivery applications and are not advertised as part of the framework API.
A streaming iterator may hold provider resources. Consume it fully or close it explicitly when the caller cancels the request.