API Reference

Complete reference for all public types, classes, records, and functions in the SDK.

Unit: mormot.ai.litellm.types

Foundation types, records, enums, exceptions, and helper functions.

Enumerations

TLLMFinishReason

TLLMFinishReason = (frStop, frLength, frToolCalls, frContentFilter, frError);

Finish reason for an LLM completion. Used internally; the SDK uses RawUtf8 strings in TLLMResponse.FinishReason for flexibility.

TLLMRoutingStrategy

TLLMRoutingStrategy = (
  rsSimpleShuffle,   // weighted random selection
  rsLowestLatency,   // pick deployment with lowest avg latency
  rsLowestCost,      // pick deployment with cheapest model
  rsLeastBusy,       // pick deployment with fewest requests
  rsUsageBased);     // pick deployment with lowest RequestCount among those with Tpm > 0

Records

TLLMMessage

TLLMMessage = packed record
  Role: RawUtf8;         // 'system', 'user', 'assistant', 'tool', 'developer'
  Content: RawUtf8;      // text content
  Name: RawUtf8;         // tool name (for role='tool')
  ToolCallId: RawUtf8;   // for tool result messages
  ToolCalls: variant;    // TDocVariant array (for assistant messages with tool calls)
end;
TLLMMessageDynArray = array of TLLMMessage;

JSON mapping: role, content, name, tool_call_id, tool_calls

TLLMToolFunction

TLLMToolFunction = packed record
  Name: RawUtf8;
  Description: RawUtf8;
  Parameters: variant;   // JSON Schema as TDocVariant
end;

JSON mapping: name, description, parameters

TLLMTool

TLLMTool = packed record
  Type_: RawUtf8;              // 'function'
  Function_: TLLMToolFunction;
end;
TLLMToolDynArray = array of TLLMTool;

JSON mapping: type, function (nested TLLMToolFunction)

TLLMToolCall

TLLMToolCall = packed record
  Id: RawUtf8;           // unique call ID
  Type_: RawUtf8;        // 'function'
  Name: RawUtf8;         // function name
  Arguments: variant;    // parsed JSON arguments as TDocVariant
  RawData: variant;      // provider-specific raw data
end;
TLLMToolCallDynArray = array of TLLMToolCall;

JSON mapping: id, type, name, arguments, raw_data

TLLMUsage

TLLMUsage = packed record
  PromptTokens: Int64;
  CompletionTokens: Int64;
  TotalTokens: Int64;
  CacheCreationInputTokens: Int64;   // Anthropic prompt caching
  CacheReadInputTokens: Int64;       // Anthropic prompt caching
  procedure Reset;                    // zero all fields
  procedure Add(const Other: TLLMUsage);  // accumulate
  function ToString: RawUtf8;         // 'prompt:X completion:Y total:Z'
end;

JSON mapping: prompt_tokens, completion_tokens, total_tokens, cache_creation_input_tokens, cache_read_input_tokens

TLLMResponse

TLLMResponse = packed record
  Id: RawUtf8;
  Model: RawUtf8;
  Content: RawUtf8;
  ToolCalls: TLLMToolCallDynArray;
  FinishReason: RawUtf8;    // 'stop', 'length', 'tool_calls', 'content_filter', 'error'
  Usage: TLLMUsage;
  Created: Int64;           // Unix timestamp
  Cost: Double;             // calculated cost in USD
  RawResponse: variant;     // full provider JSON as TDocVariant
  function HasToolCalls: Boolean;
end;

TLLMModelInfo

TLLMModelInfo = packed record
  ModelKey: RawUtf8;
  Provider: RawUtf8;
  MaxTokens: Integer;
  MaxInputTokens: Integer;
  MaxOutputTokens: Integer;
  InputCostPerToken: Double;
  OutputCostPerToken: Double;
  SupportsVision: Boolean;
  SupportsFunctionCalling: Boolean;
  SupportsStreaming: Boolean;
  SupportsPromptCaching: Boolean;
  SupportsReasoning: Boolean;
  SupportsSystemMessages: Boolean;
end;

JSON mapping: model_key, provider, max_tokens, max_input_tokens, max_output_tokens, input_cost_per_token, output_cost_per_token, supports_vision, supports_function_calling, supports_streaming, supports_prompt_caching, supports_reasoning, supports_system_messages

TLLMDeployment

TLLMDeployment = packed record
  // Configuration
  ModelName: RawUtf8;    // logical model name for routing
  ModelId: RawUtf8;      // unique deployment ID
  Provider: RawUtf8;     // 'openai', 'anthropic', 'gemini'
  ApiKey: RawUtf8;
  ApiBase: RawUtf8;
  Model: RawUtf8;        // actual model name for API
  Tpm: Integer;          // tokens per minute limit (used by rsUsageBased filter only)
  Rpm: Integer;          // requests per minute limit (actively enforced)
  Weight: Integer;       // weight for shuffle strategy
  // Runtime (managed by router)
  CooldownUntil: Int64;
  FailCount: Integer;
  TotalLatencyMs: Int64;
  RequestCount: Int64;
end;
TLLMDeploymentDynArray = array of TLLMDeployment;

Callback Types

TOnLLMStreamDelta

TOnLLMStreamDelta = procedure(const Delta: RawUtf8) of object;

Callback fired for each text delta during streaming. Must be a method (not a standalone procedure).

Exceptions

ELiteLLM                       Base class (StatusCode, Provider, Model fields)
  ELiteLLMBadRequest           400
    ELiteLLMContextWindow      400 + context window keywords
  ELiteLLMAuthentication       401, 403
  ELiteLLMNotFound             404
  ELiteLLMTimeout              408
  ELiteLLMRateLimit            429
  ELiteLLMServiceUnavailable   503
  ELiteLLMBudgetExceeded       Budget exceeded

Constructor: ELiteLLM.Create(aStatusCode: Integer; const aMessage, aProvider, aModel: RawUtf8)

Constants

LITELLM_DEFAULT_OPENAI_BASE    = 'https://api.openai.com/v1';
LITELLM_DEFAULT_ANTHROPIC_BASE = 'https://api.anthropic.com/v1';
LITELLM_DEFAULT_GEMINI_HOST    = 'https://generativelanguage.googleapis.com';
LITELLM_DEFAULT_GEMINI_BASE    = '/v1beta';
LITELLM_ANTHROPIC_VERSION      = '2023-06-01';

Helper Functions

ParseProviderModel

procedure ParseProviderModel(const ModelString: RawUtf8;
  out Provider, Model: RawUtf8);

Splits "provider/model" into parts. If no /, defaults to Provider='openai'.

FinishReasonToStr / StrToFinishReason

function FinishReasonToStr(Reason: TLLMFinishReason): RawUtf8;
function StrToFinishReason(const S: RawUtf8): TLLMFinishReason;

Handles provider-specific values: 'end_turn' -> frStop, 'tool_use' -> frToolCalls, 'STOP' -> frStop, 'MAX_TOKENS' -> frLength, 'SAFETY' -> frContentFilter.

ComputeBackoffMs

function ComputeBackoffMs(Attempt: Integer;
  BaseMs: Integer = 500; MaxMs: Integer = 8000): Integer;

Exponential backoff: min(base * 2^attempt, max) + random(0..750). Matches Python LiteLLM's jitter behavior.

RaiseForStatus

function RaiseForStatus(StatusCode: Integer;
  const Body, Provider, Model: RawUtf8): ELiteLLM;

Creates the appropriate ELiteLLM subclass based on status code. Extracts error message from JSON body (error.message or error field).

IsRetryableStatus

function IsRetryableStatus(StatusCode: Integer): Boolean;

Returns True for 408, 409, 429, and >= 500.

ParseApiBase

procedure ParseApiBase(const ApiBase: RawUtf8;
  out Server, Port, BasePath: RawUtf8; out UseTLS: Boolean);

Parses https://api.example.com:443/v1 into components.

BuildHeaderString

function BuildHeaderString(const Headers: variant): RawUtf8;

Converts a TDocVariant object of name-value pairs into HTTP header string (Name: Value\r\n).

CheckResponseError

procedure CheckResponseError(const Body, Provider, Model: RawUtf8);

Checks if a 200-OK response body contains an error object and raises the appropriate exception.

Unit: mormot.ai.litellm.provider

ILiteLLMProviderConfig

ILiteLLMProviderConfig = interface
  ['{7A8B9C0D-1E2F-4A5B-8C7D-9E0F1A2B3C4D}']
  function TransformRequest(const Model: RawUtf8;
    const Messages: TLLMMessageDynArray;
    const Tools: TLLMToolDynArray;
    const Params: variant): variant;
  function TransformResponse(const Model: RawUtf8;
    const RawResponse: RawUtf8): TLLMResponse;
  function GetCompleteUrl(const ApiBase, Model, ApiKey: RawUtf8;
    Stream: Boolean): RawUtf8;
  function GetAuthHeaders(const ApiKey: RawUtf8): variant;
  function GetSupportedParams: TRawUtf8DynArray;
  function MapParams(const OpenAIParams: variant): variant;
  function GetProviderName: RawUtf8;
  procedure ProcessSSEEvent(const EventType, Data: RawUtf8;
    var Accumulator: TLLMResponse; OnDelta: TOnLLMStreamDelta);
end;

TLiteLLMProviderBase

Abstract base class implementing ILiteLLMProviderConfig. Concrete providers override the abstract methods and optionally override the virtual ones.

Abstract methods (must override): TransformRequest, TransformResponse, GetCompleteUrl, GetSupportedParams, GetProviderName

Virtual methods (optional override):

• GetAuthHeaders — default: Authorization: Bearer <key>
• MapParams — default: pass-through
• ProcessSSEEvent — default: empty (no streaming)

Class functions:

• MessagesToVariant(const Msgs: TLLMMessageDynArray): variant — OpenAI-format message array
• ToolsToVariant(const Tools: TLLMToolDynArray): variant — OpenAI-format tools array
• ExtractSystemMessages(var Messages: TLLMMessageDynArray): RawUtf8 — removes system/developer messages, returns concatenated content

Unit: mormot.ai.litellm.handler

TLiteLLMHTTPHandler

Central HTTP orchestrator. Manages provider registry, HTTP connections, and retry logic.

constructor Create; overload;
constructor Create(Callbacks: TLiteLLMCallbackManager); overload;
destructor Destroy; override;

Methods:

procedure RegisterProvider(const Name: RawUtf8;
  const Config: ILiteLLMProviderConfig);

Register or update a provider. Thread-safe (uses TLightLock).

function GetProvider(const Name: RawUtf8): ILiteLLMProviderConfig;

Returns the provider or raises ELiteLLM if not found.

function Chat(const ProviderName, ApiKey, ApiBase, Model: RawUtf8;
  const Messages: TLLMMessageDynArray;
  const Tools: TLLMToolDynArray;
  const Params: variant;
  TimeoutMs: Integer = 600000): TLLMResponse;

Synchronous chat completion. Creates a THttpClientSocket, sends POST, parses response.

function ChatStream(const ProviderName, ApiKey, ApiBase, Model: RawUtf8;
  const Messages: TLLMMessageDynArray;
  const Tools: TLLMToolDynArray;
  OnDelta: TOnLLMStreamDelta;
  const Params: variant;
  TimeoutMs: Integer = 600000): TLLMResponse;

SSE streaming. Creates TLiteLLMSSEStream as output stream, fires OnDelta for each text token.

function ChatWithRetry(const ProviderName, ApiKey, ApiBase, Model: RawUtf8;
  const Messages: TLLMMessageDynArray;
  const Tools: TLLMToolDynArray;
  const Params: variant;
  MaxRetries: Integer = 2;
  Stream: Boolean = False;
  OnDelta: TOnLLMStreamDelta = nil): TLLMResponse;

Wraps Chat or ChatStream with exponential backoff retry on retryable errors.

Properties:

• Callbacks: TLiteLLMCallbackManager — lifecycle event manager

Unit: mormot.ai.litellm.streaming

TLiteLLMSSEStream

TStream subclass for real-time SSE parsing.

constructor Create(OnDelta: TOnLLMStreamDelta;
  const Provider: ILiteLLMProviderConfig);

Key methods:

• Write(const Buffer; Count: Longint): Longint — receives HTTP chunks, parses SSE lines
• ProcessBuffer — process any remaining buffered data
• GetResponse: TLLMResponse — return the accumulated response

Properties:

• Done: Boolean — True when [DONE] marker received

Unit: mormot.ai.litellm.router

TLiteLLMRouter

Multi-deployment router with load balancing and failover.

constructor Create(AHandler: TLiteLLMHTTPHandler;
  ACostTracker: TLiteLLMCostTracker);

Methods:

procedure AddDeployment(const Deployment: TLLMDeployment);
procedure RemoveDeployments(const ModelName: RawUtf8);
procedure SetFallbacks(const Model: RawUtf8;
  const Fallbacks: TRawUtf8DynArray);

function Chat(const ModelName: RawUtf8;
  const Messages: TLLMMessageDynArray;
  const Tools: TLLMToolDynArray;
  const Params: variant;
  Stream: Boolean = False;
  OnDelta: TOnLLMStreamDelta = nil): TLLMResponse;

Route to best healthy deployment. Tries primary model, then fallbacks.

function DeploymentCount: Integer;

Properties:

• Strategy: TLLMRoutingStrategy — routing algorithm (default: rsSimpleShuffle)
• CooldownSeconds: Integer — cooldown after failure (default: 60)
• MaxRetries: Integer — max attempts per model (default: 3)

Unit: mormot.ai.litellm.cost

TLiteLLMCostTracker

Model pricing and cost calculation. Pre-loaded with ~20 models.

constructor Create;  // loads built-in prices

Methods:

procedure RegisterModel(const Info: TLLMModelInfo);
procedure LoadFromJson(const Json: RawUtf8);
function GetModelInfo(const ModelKey: RawUtf8;
  out Info: TLLMModelInfo): Boolean;
function CalculateCost(const ModelKey: RawUtf8;
  const Usage: TLLMUsage): Double;
function Count: Integer;

GetModelInfo tries exact match first, then strips provider prefix (openai/gpt-4o -> gpt-4o).

Built-in models: gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-4, gpt-3.5-turbo, o1, o1-mini, o3-mini, claude-3-5-sonnet-20241022, claude-3-5-haiku-20241022, claude-3-opus-20240229, claude-3-haiku-20240307, claude-sonnet-4-20250514, claude-opus-4-20250514, gemini-1.5-pro, gemini-1.5-flash, gemini-2.5-pro, gemini-2.5-flash, gemini-3.1-pro-preview.

Unit: mormot.ai.litellm.callbacks

ILiteLLMCallback

ILiteLLMCallback = interface
  ['{B1C2D3E4-F5A6-4B7C-8D9E-0F1A2B3C4D5E}']
  procedure OnStart(const Provider, Model: RawUtf8;
    const Messages: TLLMMessageDynArray; const Params: variant);
  procedure OnSuccess(const Provider, Model: RawUtf8;
    const Response: TLLMResponse; ElapsedMs: Int64);
  procedure OnFailure(const Provider, Model: RawUtf8;
    const Error: RawUtf8; StatusCode: Integer; ElapsedMs: Int64);
  procedure OnStreamDelta(const Provider, Model: RawUtf8;
    const Delta: RawUtf8);
end;

TLiteLLMCallbackManager

Thread-safe manager for multiple callbacks.

procedure Add(const Callback: ILiteLLMCallback);
procedure Remove(const Callback: ILiteLLMCallback);
procedure FireOnStart(const Provider, Model: RawUtf8;
  const Messages: TLLMMessageDynArray; const Params: variant);
procedure FireOnSuccess(const Provider, Model: RawUtf8;
  const Response: TLLMResponse; ElapsedMs: Int64);
procedure FireOnFailure(const Provider, Model: RawUtf8;
  const Error: RawUtf8; StatusCode: Integer; ElapsedMs: Int64);
procedure FireOnStreamDelta(const Provider, Model: RawUtf8;
  const Delta: RawUtf8);

Callback errors are swallowed to prevent breaking the main flow.

TLiteLLMLogCallback

ILiteLLMCallback implementation that logs to TSynLog:

• OnStart: sllInfo — provider, model, message count
• OnSuccess: sllInfo — provider, model, finish reason, usage, cost, elapsed
• OnFailure: sllWarning — provider, model, status, error, elapsed
• OnStreamDelta: sllTrace — provider, model, delta length

Unit: mormot.ai.litellm.client

TLiteLLM

Main SDK facade. Creates and manages all internal components.

constructor Create;
destructor Destroy; override;

Constructor reads .env from exe directory, creates handler/router/cost tracker/callbacks, registers OpenAI/Anthropic/Gemini providers.

Chat methods:

function Chat(const Model: RawUtf8;
  const Messages: TLLMMessageDynArray;
  const ApiKey: RawUtf8 = '';
  const ApiBase: RawUtf8 = '';
  MaxTokens: Integer = 4096;
  Temperature: Double = 1.0): TLLMResponse; overload;

Simple chat. No tools, no streaming.

function Chat(const Model: RawUtf8;
  const Messages: TLLMMessageDynArray;
  const Tools: TLLMToolDynArray;
  const ApiKey: RawUtf8 = '';
  const ApiBase: RawUtf8 = '';
  MaxTokens: Integer = 4096;
  Temperature: Double = 1.0;
  Stream: Boolean = False;
  OnDelta: TOnLLMStreamDelta = nil): TLLMResponse; overload;

Full chat with tools and optional streaming. Builds Params variant from MaxTokens and Temperature.

function ChatRaw(const Model: RawUtf8;
  const Messages: TLLMMessageDynArray;
  const Tools: TLLMToolDynArray;
  const Params: variant;
  const ApiKey: RawUtf8 = '';
  const ApiBase: RawUtf8 = '';
  Stream: Boolean = False;
  OnDelta: TOnLLMStreamDelta = nil): TLLMResponse;

Raw chat with pre-built params variant. Used by the proxy for pass-through.

function RouterChat(const Model: RawUtf8;
  const Messages: TLLMMessageDynArray;
  const Tools: TLLMToolDynArray;
  Stream: Boolean = False;
  OnDelta: TOnLLMStreamDelta = nil): TLLMResponse;

Route via the router (requires deployments to be registered).

Management:

procedure RegisterProvider(const Name: RawUtf8;
  const Config: ILiteLLMProviderConfig);
procedure AddCallback(const Callback: ILiteLLMCallback);
procedure AddDeployment(const Deployment: TLLMDeployment);

Properties:

• Handler: TLiteLLMHTTPHandler — for advanced HTTP configuration
• Router: TLiteLLMRouter — for routing configuration
• CostTracker: TLiteLLMCostTracker — for model info and pricing
• Callbacks: TLiteLLMCallbackManager — for lifecycle events

Unit: mormot.ai.litellm.proxy

TLiteLLMProxy

OpenAI-compatible HTTP proxy server.

constructor Create(aPort: Integer = 4000);
destructor Destroy; override;
procedure Start;
procedure Stop;
procedure AddModelMapping(const FromModel, ToModel: RawUtf8);

Properties:

• LiteLLM: TLiteLLM — underlying SDK client
• Port: Integer — listening port
• ApiKey: RawUtf8 — proxy authentication key (empty = no auth)
• DefaultModel: RawUtf8 — fallback model when request has no prefix

TProxyStreamAccumulator

Helper for buffered SSE in proxy mode.

constructor Create(const aId, aModel: RawUtf8; aCreated: Int64);
procedure OnDelta(const Delta: RawUtf8);  // accumulates SSE chunks
function GetSSEBody: RawUtf8;             // returns buffered SSE content

Unit: mormot.ai.litellm.provider.openai

TOpenAIProviderConfig

OpenAI provider. Quasi pass-through since OpenAI format is canonical.

• GetCompleteUrl returns /chat/completions
• GetAuthHeaders returns Authorization: Bearer <key> (via base class)
• Handles max_tokens vs max_completion_tokens for o1/o3 models
• Streaming: stream_options.include_usage = true for usage in final chunk

Unit: mormot.ai.litellm.provider.anthropic

TAnthropicProviderConfig

Anthropic Claude provider.

• System messages extracted to top-level system field
• Messages converted to content blocks (text, tool_use, tool_result)
• Tool parameters renamed to input_schema
• max_tokens always required (intelligent defaults per model)
• Auth: x-api-key + anthropic-version: 2023-06-01
• GetCompleteUrl returns /messages
• SSE state machine for event-typed streaming

Unit: mormot.ai.litellm.provider.gemini

TGeminiProviderConfig

Google Gemini provider.

• Messages converted to contents[].parts[] format
• Roles: assistant -> model, tool -> user
• System prompt to systemInstruction
• Tools wrapped in tools[].functionDeclarations[]
• Generation config: maxOutputTokens, topP, topK, stopSequences, responseMimeType
• Auth via URL query parameter (?key=)
• GetCompleteUrl returns /models/<model>:generateContent?key=<key> or :streamGenerateContent?alt=sse&key=<key>
• Tool call IDs are synthetic (call_0, call_1)
• thoughtSignature preserved in RawData for round-trip fidelity