Streaming

SSE (Server-Sent Events) streaming delivers tokens in real-time as the LLM generates them, instead of waiting for the complete response.

Basic Usage

Provide Stream = True and an OnDelta callback:

type
  TMyHandler = class
    procedure HandleDelta(const Delta: RawUtf8);
  end;

procedure TMyHandler.HandleDelta(const Delta: RawUtf8);
begin
  Write(Delta);  // print each token as it arrives
end;

var
  Handler: TMyHandler;
  Response: TLLMResponse;
begin
  Handler := TMyHandler.Create;
  try
    Response := LLM.Chat('openai/gpt-4o',
      Messages,          // TLLMMessageDynArray
      Tools,             // TLLMToolDynArray (can be empty)
      '',                // ApiKey (from .env)
      '',                // ApiBase (default)
      4096,              // MaxTokens
      1.0,               // Temperature
      True,              // Stream
      Handler.HandleDelta);  // OnDelta callback

    // Response is complete here — Content, ToolCalls, FinishReason, Usage populated
    // Note: Created and RawResponse are NOT populated via the streaming path
    WriteLn;
    WriteLn('Finish: ', Response.FinishReason);
    WriteLn('Usage: ', Response.Usage.ToString);
  finally
    Handler.Free;
  end;
end;

The OnDelta callback is TOnLLMStreamDelta = procedure(const Delta: RawUtf8) of object — it must be a method, not a standalone procedure.

How It Works

Architecture

TLiteLLMHTTPHandler.ChatStream
  |
  v
THttpClientSocket.Request(... OutStream := SSE)
  |                              |
  |  HTTP chunks arrive  ------> TLiteLLMSSEStream.Write()
  |                              |
  |                              v
  |                         ProcessBuffer() — parse SSE lines
  |                              |
  |                              v
  |                         HandleLine() — accumulate event/data
  |                              |
  |                              v
  |                         DispatchEvent() — on empty line
  |                              |
  |                              v
  |                         ILiteLLMProviderConfig.ProcessSSEEvent()
  |                              |
  |                              +---> OnDelta(text fragment)
  |                              +---> accumulate tool calls
  |                              +---> accumulate usage
  v
SSE.GetResponse() -> TLLMResponse (complete)

TLiteLLMSSEStream

TLiteLLMSSEStream (in mormot.ai.litellm.streaming.pas) is a TStream subclass that intercepts Write calls from THttpClientSocket's chunked transfer decoding. It:

1. Buffers incoming bytes
2. Splits on \n boundaries (per W3C SSE spec)
3. Parses event: and data: fields
4. On empty line: dispatches the accumulated event to the provider's ProcessSSEEvent
5. Handles the [DONE] marker (OpenAI convention)

SSE Protocol

Per the W3C Server-Sent Events specification:

• Lines starting with : are comments (ignored)
• event: <type> sets the event type for the next dispatch
• data: <payload> sets the data (multiple data: lines are joined with \n)
• An empty line dispatches the current event
• [DONE] in data field signals stream end (OpenAI convention, handled by base stream)

Provider-Specific Processing

Each provider overrides ProcessSSEEvent(EventType, Data, Accumulator, OnDelta):

OpenAI: Ignores EventType. Parses choices[0].delta.content for text, choices[0].delta.tool_calls for tool call fragments. Usage comes from the final chunk when stream_options.include_usage is enabled.

Anthropic: Uses EventType as a state machine driver (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, error). Text deltas arrive in content_block_delta with type: text_delta. Tool arguments arrive as input_json_delta fragments.

Gemini: Ignores EventType. Each chunk contains a complete candidates[0].content.parts[] array. Text parts fire OnDelta, function call parts create tool call entries. Usage metadata (usageMetadata) is updated from each chunk.

Streaming with Tool Calls

Tool calls are accumulated during streaming and available in the final response:

Response := LLM.Chat('anthropic/claude-sonnet-4-20250514',
  Messages, Tools, '', '', 4096, 1.0, True, Handler.HandleDelta);

if Response.HasToolCalls then
begin
  // Tool calls are fully parsed at this point
  for i := 0 to High(Response.ToolCalls) do
    WriteLn('Tool: ', Response.ToolCalls[i].Name);
end;

Tool call arguments arrive as JSON string fragments during streaming. The provider accumulates them in TLLMToolCall.RawData and parses the complete JSON when the tool call block finishes (content_block_stop for Anthropic, finish_reason for OpenAI).

Streaming via ChatRaw

For proxy scenarios where you want to pass raw parameters:

var
  Params: variant;
begin
  Params := _Obj([
    'max_tokens', 2048,
    'temperature', 0.5]);

  Response := LLM.ChatRaw('gemini/gemini-2.5-flash',
    Messages, Tools, Params,
    '', '',   // ApiKey, ApiBase
    True,     // Stream
    Handler.HandleDelta);
end;

Callbacks During Streaming

If you've registered ILiteLLMCallback implementations, the handler fires:

• OnStart — before the HTTP request
• OnSuccess — after streaming completes (with the final TLLMResponse)
• OnFailure — if an error occurs during streaming

Note: ILiteLLMCallback.OnStreamDelta is available on the callback interface but must be explicitly called by your application if needed — the handler does not fire it automatically during streaming (to avoid double-dispatching with OnDelta).

Error Handling During Streaming

If the HTTP response status is not 2xx, an ELiteLLM exception is raised after the request completes (the status is checked after Sock.Request returns, not before streaming begins). If the stream encounters an error event (Anthropic event: error), the provider raises ELiteLLM from within ProcessSSEEvent.

try
  Response := LLM.Chat('anthropic/claude-sonnet-4-20250514',
    Messages, Tools, '', '', 4096, 1.0, True, Handler.HandleDelta);
except
  on E: ELiteLLMRateLimit do
    WriteLn('Rate limited during stream: ', E.Message);
  on E: ELiteLLM do
    WriteLn('Stream error: ', E.StatusCode, ' ', E.Message);
end;

Timeouts

The default timeout for streaming is 600000ms (10 minutes), set in TLiteLLMHTTPHandler.ChatStream. This applies to both the initial connection and the ongoing receive. For long-running generations, this should be sufficient. The timeout is not currently configurable through TLiteLLM.Chat — for custom timeouts, use Handler.ChatStream directly.