Providers

The SDK ships with three built-in providers, all registered automatically by TLiteLLM.Create:

Provider Interface

Every provider implements ILiteLLMProviderConfig (defined in mormot.ai.litellm.provider.pas):

ILiteLLMProviderConfig = interface
  function TransformRequest(const Model: RawUtf8;
    const Messages: TLLMMessageDynArray;
    const Tools: TLLMToolDynArray;
    const Params: variant): variant;
  function TransformResponse(const Model: RawUtf8;
    const RawResponse: RawUtf8): TLLMResponse;
  function GetCompleteUrl(const ApiBase, Model, ApiKey: RawUtf8;
    Stream: Boolean): RawUtf8;
  function GetAuthHeaders(const ApiKey: RawUtf8): variant;
  function GetSupportedParams: TRawUtf8DynArray;
  function MapParams(const OpenAIParams: variant): variant;
  function GetProviderName: RawUtf8;
  procedure ProcessSSEEvent(const EventType, Data: RawUtf8;
    var Accumulator: TLLMResponse; OnDelta: TOnLLMStreamDelta);
end;

The handler calls these methods in sequence: TransformRequest -> HTTP POST -> TransformResponse (sync) or ProcessSSEEvent (streaming).

OpenAI

Unit: mormot.ai.litellm.provider.openai.pas

OpenAI is the canonical format for the SDK. Transformations are quasi pass-through.

Request Transformation

• Messages converted via TLiteLLMProviderBase.MessagesToVariant (handles content=null for assistant messages with tool calls)
• Tools converted via TLiteLLMProviderBase.ToolsToVariant (OpenAI native format)
• For reasoning models (o1, o3): max_tokens is mapped to max_completion_tokens
• When streaming: adds stream_options.include_usage = true for token counts in the final chunk

Response Transformation

• choices[0].message.content -> TLLMResponse.Content
• choices[0].message.tool_calls[].function -> TLLMResponse.ToolCalls[]
• usage.prompt_tokens / completion_tokens / total_tokens -> TLLMUsage
• choices[0].finish_reason -> mapped directly (already standard format)
• If tool calls present but finish_reason = 'stop', normalized to 'tool_calls'

Streaming

OpenAI uses standard SSE with data: {...} lines. The [DONE] marker signals completion.

Each chunk contains choices[0].delta with incremental content and tool call fragments. Tool call arguments arrive as JSON string fragments across multiple chunks, accumulated in RawData and parsed when finish_reason is received.

Supported Parameters

temperature, max_tokens, top_p, frequency_penalty, presence_penalty, stop, n, stream, response_format, seed, tools, tool_choice

Note: logprobs is declared in GetSupportedParams but is not currently forwarded by TransformRequest.

Authentication

Standard Bearer token: Authorization: Bearer <api_key>

Endpoint

POST /chat/completions (relative to API base)

Anthropic

Unit: mormot.ai.litellm.provider.anthropic.pas

Anthropic's Messages API has significant format differences from OpenAI. The provider handles all transformations transparently.

Request Transformation

System messages:

• All system and developer role messages are extracted from the array and concatenated
• The result is placed in the top-level system field (Anthropic requirement)
• The messages array is filtered to exclude system/developer messages

Content blocks:

• User messages become [{"type": "text", "text": "..."}]
• Assistant messages with tool calls become [{"type": "tool_use", "id": "...", "name": "...", "input": {...}}] blocks
• Tool results become [{"type": "tool_result", "tool_use_id": "...", "content": "..."}]
• Tool role is mapped to user role (tool results are user messages in Anthropic)

Consecutive same-role merging:

• Anthropic API rejects consecutive messages with the same role
• The provider automatically merges content blocks from consecutive same-role messages

Tool format:

• OpenAI's parameters is renamed to input_schema
• Tool choice mapping: "required" -> {"type": "any"}, "auto" -> {"type": "auto"}

max_tokens is always required:

• If not provided, defaults based on model: opus-4 = 32768, sonnet-4 = 16384, opus = 4096, others = 8192

Response Transformation

• content[] blocks parsed: text -> Content, tool_use -> ToolCalls[]
• stop_reason mapping: end_turn -> stop, tool_use -> tool_calls, max_tokens -> length
• usage.input_tokens -> PromptTokens, usage.output_tokens -> CompletionTokens
• Cache tokens: cache_creation_input_tokens and cache_read_input_tokens preserved in TLLMUsage
• Thinking blocks are ignored (accessible via RawResponse)

Streaming

Anthropic uses event-typed SSE (not just data: lines). The provider implements a state machine:

Supported Parameters

temperature, max_tokens, top_p, stop (mapped to stop_sequences), stream, tools, tool_choice

Note: system is not a params field — system messages are extracted automatically from the messages array. top_k and metadata are not currently forwarded by TransformRequest.

Authentication

Custom headers (not Bearer token):

x-api-key: <api_key>
anthropic-version: 2023-06-01
content-type: application/json

Endpoint

POST /messages (relative to API base)

Gemini

Unit: mormot.ai.litellm.provider.gemini.pas

Google Gemini uses a completely different API structure. The provider handles all conversions.

Request Transformation

Message format:

• Messages become contents[].parts[] with role field
• assistant role is mapped to model
• tool role is mapped to user
• Text content: {"text": "..."}
• Tool calls: {"functionCall": {"name": "...", "args": {...}}}
• Tool results: {"functionResponse": {"name": "...", "response": {"result": "..."}}}

System prompt:

• Extracted to systemInstruction.parts[].text (separate top-level field)

Consecutive same-role merging:

• Gemini API rejects consecutive turns with same role
• Parts from consecutive same-role messages are merged into a single turn

Tool format:

• OpenAI tools wrapped in: tools[].functionDeclarations[]
• Each declaration has name, description, parameters
• Tool choice mapping: "required"/"any" -> mode: "ANY", "none" -> mode: "NONE", "auto" -> mode: "AUTO"
• Specific function: {"type":"function","function":{"name":"foo"}} -> mode: "ANY", allowedFunctionNames: ["foo"]

Generation config:

• max_tokens -> maxOutputTokens
• temperature, topP, topK in generationConfig object
• stop -> stopSequences (string converted to single-element array)
• response_format.type = "json_object" -> responseMimeType: "application/json"
• response_format.type = "json_schema" -> responseMimeType: "application/json" + responseSchema

thoughtSignature preservation:

• When replaying tool calls, if the original response contained thoughtSignature, it's stored in TLLMToolCall.RawData as the complete Gemini Part object
• The BuildParts method checks for _raw_fc in tool call data and uses it verbatim for round-trip fidelity

Response Transformation

• candidates[0].content.parts[] parsed: text -> Content, functionCall -> ToolCalls[]
• finishReason mapping: STOP -> stop, MAX_TOKENS -> length, SAFETY -> content_filter
• Tool call IDs are synthetic: call_0, call_1, etc. (Gemini doesn't provide IDs)
• usageMetadata.promptTokenCount -> PromptTokens, candidatesTokenCount -> CompletionTokens
• promptFeedback.blockReason raises ELiteLLM (400) if present

Streaming

Gemini uses standard SSE with data: lines (when alt=sse query param is set). Each chunk contains a complete candidates[] structure. The provider processes text parts and function calls from each chunk incrementally.

Supported Parameters

temperature, max_tokens, top_p, top_k, stop (mapped to stopSequences), response_format (mapped to responseMimeType/responseSchema), stream, tools, tool_choice

Authentication

Query parameter auth (not headers):

• Sync: /models/{model}:generateContent?key={api_key}
• Stream: /models/{model}:streamGenerateContent?alt=sse&key={api_key}

The GetAuthHeaders method returns only content-type: application/json (no auth headers).

Endpoint

Dynamic path based on model name and streaming flag:

• Sync: /models/<model>:generateContent?key=<api_key>
• Stream: /models/<model>:streamGenerateContent?alt=sse&key=<api_key>

Custom Providers

You can register additional providers at runtime:

type
  TMyProvider = class(TLiteLLMProviderBase)
  public
    function TransformRequest(const Model: RawUtf8;
      const Messages: TLLMMessageDynArray;
      const Tools: TLLMToolDynArray;
      const Params: variant): variant; override;
    function TransformResponse(const Model: RawUtf8;
      const RawResponse: RawUtf8): TLLMResponse; override;
    function GetCompleteUrl(const ApiBase, Model, ApiKey: RawUtf8;
      Stream: Boolean): RawUtf8; override;
    function GetSupportedParams: TRawUtf8DynArray; override;
    function GetProviderName: RawUtf8; override;
    // Override ProcessSSEEvent for streaming support
    // Override GetAuthHeaders if not using Bearer token
  end;

// Register
LLM.RegisterProvider('my-provider', TMyProvider.Create);

// Use
Response := LLM.Chat('my-provider/my-model', Messages);

TLiteLLMProviderBase provides default implementations:

• GetAuthHeaders: returns Authorization: Bearer <key> (override for non-Bearer auth)
• MapParams: pass-through (available for external use but not called by the handler — the handler calls TransformRequest directly)
• ProcessSSEEvent: empty (override for streaming support)

And shared helper class functions:

• MessagesToVariant: converts TLLMMessageDynArray to TDocVariant array (OpenAI format)
• ToolsToVariant: converts TLLMToolDynArray to TDocVariant array (OpenAI format)
• ExtractSystemMessages: removes system/developer messages from array, returns concatenated content

Custom API Base URLs

Override the API base URL for any provider (useful for self-hosted models, proxies, or OpenAI-compatible endpoints):

// OpenAI-compatible endpoint (e.g. vLLM, Ollama)
Response := LLM.Chat('openai/my-local-model', Messages,
  'no-key-needed',
  'http://localhost:8000/v1');

// Self-hosted Gemini proxy
Response := LLM.Chat('gemini/gemini-2.5-flash', Messages,
  'my-key',
  'https://my-proxy.example.com/v1beta');

Note: The OpenAI provider always appends /chat/completions to the base URL and uses Bearer auth. It does not support Azure OpenAI's deployment-based URL format or API key header — for Azure, use a custom provider or a local proxy.