Router

TLiteLLMRouter provides multi-deployment routing with load balancing, automatic failover, rate limiting, and fallback chains. Use it when you have multiple API keys, endpoints, or want automatic failover between providers.

Basic Setup

var
  LLM: TLiteLLM;
  D: TLLMDeployment;
begin
  LLM := TLiteLLM.Create;

  // Deployment 1: primary OpenAI endpoint
  FillCharFast(D, SizeOf(D), 0);
  D.ModelName := 'gpt-4o';           // logical name for routing
  D.ModelId := 'gpt4o-key1';         // unique deployment ID
  D.Provider := 'openai';
  D.ApiKey := 'sk-key-1';
  D.ApiBase := 'https://api.openai.com/v1';
  D.Model := 'gpt-4o';              // actual model name for API
  D.Rpm := 100;                      // requests per minute limit
  D.Weight := 2;                     // higher weight = more traffic
  LLM.AddDeployment(D);

  // Deployment 2: secondary OpenAI endpoint (different key)
  D.ModelId := 'gpt4o-key2';
  D.ApiKey := 'sk-key-2';
  D.Weight := 1;
  LLM.AddDeployment(D);

  // Route: picks best healthy deployment
  Response := LLM.RouterChat('gpt-4o', Messages, Tools);
end;

TLLMDeployment Record

TLLMDeployment = packed record
  // --- Configuration (set by you) ---
  ModelName: RawUtf8;    // logical model name (used for routing lookup)
  ModelId: RawUtf8;      // unique deployment identifier
  Provider: RawUtf8;     // 'openai', 'anthropic', 'gemini'
  ApiKey: RawUtf8;       // API key for this deployment
  ApiBase: RawUtf8;      // base URL (e.g. 'https://api.openai.com/v1')
  Model: RawUtf8;        // actual model name sent to the API
  Tpm: Integer;          // tokens per minute limit (0 = unlimited)
  Rpm: Integer;          // requests per minute limit (0 = unlimited)
  Weight: Integer;       // weight for shuffle strategy (default 1)

  // --- Runtime state (managed by router) ---
  CooldownUntil: Int64;  // UnixTimeUtc when cooldown expires
  FailCount: Integer;    // consecutive failures
  TotalLatencyMs: Int64; // total latency across all requests
  RequestCount: Int64;   // total request count
end;

Runtime fields are automatically managed by the router. Initialize them to 0 or use FillCharFast.

Routing Strategies

Set the strategy via LLM.Router.Strategy:

rsSimpleShuffle (default)

Weighted random selection. Each deployment's Weight determines its probability of being chosen. With weights 2 and 1, the first deployment gets ~67% of traffic.

rsLowestLatency

Selects the deployment with the lowest average latency (TotalLatencyMs / RequestCount). Only deployments with RequestCount > 0 are compared; if all deployments have zero requests, the first one is selected.

rsLowestCost

Selects the deployment whose model has the lowest InputCostPerToken (from TLiteLLMCostTracker). Useful when mixing expensive and cheap models under the same logical name.

rsLeastBusy

Selects the deployment with the lowest RequestCount. Distributes load evenly.

rsUsageBased

Selects the deployment with the lowest RequestCount among those with Tpm > 0. Useful for preferring deployments that have defined token limits. Note: this strategy does not perform actual TPM accounting — it uses RequestCount as a proxy for usage.

LLM.Router.Strategy := rsLowestLatency;

Cooldown

When a deployment fails with any ELiteLLM error, the router applies a cooldown (via MarkFailed). Only retryable errors (429, 500+) proceed to try the next deployment; non-retryable errors (400, 401, 404) are re-raised immediately after cooldown is applied:

LLM.Router.CooldownSeconds := 60;  // default

During cooldown, the deployment is excluded from selection. After cooldown expires, it becomes eligible again. The FailCount is reset to 0 on the next successful request.

Rate Limiting

The router tracks per-minute request counts and enforces RPM limits:

D.Rpm := 60;  // max 60 requests per minute for this deployment

The rate counter uses a fixed minute bucket (UnixTimeUtc div 60). Deployments that have reached their RPM limit are excluded from the healthy pool.

TPM (tokens per minute) limits are defined but only used as a filter for rsUsageBased strategy (deployments with Tpm > 0 are preferred). TPM is not actively tracked or enforced — only RPM is tracked per-minute.

Fallback Chains

Define fallback models to try when the primary model has no healthy deployments:

// If gpt-4o has no healthy deployments, try gpt-4o-mini, then claude
LLM.Router.SetFallbacks('gpt-4o',
  TRawUtf8DynArray.Create('gpt-4o-mini', 'claude-sonnet'));

The router tries models in order: primary first, then each fallback. For each model, it selects a healthy deployment and attempts the request. If all deployments for a model are in cooldown or rate-limited, it moves to the next fallback model.

Retry Logic

LLM.Router.MaxRetries := 3;  // default: 3 attempts per model

Within each model, the router retries up to MaxRetries times across different healthy deployments. A failed deployment is removed from the healthy pool for that request (and put on cooldown for future requests).

Non-retryable errors (400, 401, 404) are raised immediately without trying other deployments.

Full Example

var
  LLM: TLiteLLM;
  D: TLLMDeployment;
  Response: TLLMResponse;
begin
  LLM := TLiteLLM.Create;
  try
    // OpenAI deployments
    FillCharFast(D, SizeOf(D), 0);
    D.ModelName := 'smart-model';
    D.Provider := 'openai';
    D.ApiBase := 'https://api.openai.com/v1';
    D.Rpm := 60;

    D.ModelId := 'openai-1';
    D.ApiKey := 'sk-key1';
    D.Model := 'gpt-4o';
    D.Weight := 3;
    LLM.AddDeployment(D);

    D.ModelId := 'openai-2';
    D.ApiKey := 'sk-key2';
    D.Model := 'gpt-4o';
    D.Weight := 1;
    LLM.AddDeployment(D);

    // Gemini fallback
    D.ModelName := 'smart-model-fallback';
    D.ModelId := 'gemini-fb';
    D.Provider := 'gemini';
    D.ApiBase := 'https://generativelanguage.googleapis.com/v1beta';
    D.ApiKey := 'AIza...';
    D.Model := 'gemini-2.5-flash';
    D.Rpm := 100;
    D.Weight := 1;
    LLM.AddDeployment(D);

    // Fallback chain
    LLM.Router.SetFallbacks('smart-model',
      TRawUtf8DynArray.Create('smart-model-fallback'));

    // Configure
    LLM.Router.Strategy := rsLowestLatency;
    LLM.Router.CooldownSeconds := 30;
    LLM.Router.MaxRetries := 2;

    // Use
    Response := LLM.RouterChat('smart-model', Messages, Tools);
    WriteLn('Served by: ', Response.Model);
    WriteLn('Cost: $', FormatFloat('0.000000', Response.Cost));
  finally
    LLM.Free;
  end;
end;

Thread Safety

The router uses TLightLock for all deployment and rate counter operations. Multiple threads can safely call RouterChat concurrently. However, AddDeployment, RemoveDeployments, and SetFallbacks should be called during initialization, before concurrent access begins.

Router vs Direct Chat