Proxy Server

TLiteLLMProxy is an OpenAI-compatible HTTP proxy that accepts standard OpenAI API requests and routes them to any configured provider. Use it to give OpenAI-only applications access to Anthropic, Gemini, or multiple providers.

Quick Start

program MyProxy;
{$I mormot.defines.inc}
{$APPTYPE CONSOLE}

uses
  sysutils,
  mormot.core.base,
  mormot.core.log,
  mormot.ai.litellm.proxy;

var
  Proxy: TLiteLLMProxy;
begin
  with TSynLog.Family do
  begin
    Level := LOG_VERBOSE;
    EchoToConsole := LOG_VERBOSE;
  end;

  Proxy := TLiteLLMProxy.Create;  // reads port from .env, defaults to 4000
  try
    Proxy.Start;
    WriteLn('Proxy running on port ', Proxy.Port);
    WriteLn('Press ENTER to stop...');
    ReadLn;
    Proxy.Stop;
  finally
    Proxy.Free;
  end;
end.

Configuration

Configuration can be done via a .env file next to the executable, or programmatically via the Port, ApiKey, DefaultModel properties and AddModelMapping method:

# Proxy port (default: 4000)
PROXY_PORT=4000

# Optional: require Bearer token auth on proxy requests
PROXY_API_KEY=my-secret-proxy-key

# Optional: default model when request has no provider prefix
PROXY_DEFAULT_MODEL=gemini/gemini-2.5-flash

# Optional: model name mapping (JSON object)
MODEL_MAP={"gpt-4o":"gemini/gemini-2.5-flash","gpt-3.5-turbo":"gemini/gemini-2.5-flash","claude-3":"anthropic/claude-sonnet-4-20250514"}

# Provider API keys (used by the underlying TLiteLLM)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=AIza...

Endpoints

POST /v1/chat/completions

Accepts standard OpenAI chat completion requests:

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer my-secret-proxy-key" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 100,
    "temperature": 0.7
  }'

The proxy:

1. Validates Bearer token (if PROXY_API_KEY is set)
2. Maps the model name via MODEL_MAP or auto-detection
3. Extracts messages, tools, and parameters from the request
4. Routes to the appropriate provider via TLiteLLM.ChatRaw
5. Returns an OpenAI-format response

Also available as POST /chat/completions (without /v1 prefix).

GET /v1/models

Also available as GET /models (without /v1 prefix).

Returns the list of available models (from MODEL_MAP or default model):

curl http://localhost:4000/v1/models \
  -H "Authorization: Bearer my-secret-proxy-key"

{
  "object": "list",
  "data": [
    {"id": "gpt-4o", "object": "model", "owned_by": "litellm-proxy"},
    {"id": "gpt-3.5-turbo", "object": "model", "owned_by": "litellm-proxy"}
  ]
}

GET /health

Returns {"status":"ok"}. No authentication required.

Model Mapping

The proxy resolves incoming model names in this order:

1. Explicit MODEL_MAP — if the incoming model name is a key in the map, use the mapped value
2. Provider prefix — if the model already contains / (e.g. gemini/gemini-2.5-flash), pass through as-is
3. Default model — if PROXY_DEFAULT_MODEL is set, use it
4. Auto-detection — based on model name prefix:
5. gemini-* -> gemini/<model>
6. claude-* -> anthropic/<model>
7. Everything else -> openai (default, no prefix added)

Programmatic Mapping

Proxy := TLiteLLMProxy.Create;
Proxy.AddModelMapping('gpt-4o', 'gemini/gemini-2.5-flash');
Proxy.AddModelMapping('gpt-3.5-turbo', 'gemini/gemini-2.5-flash');
Proxy.AddModelMapping('claude-3', 'anthropic/claude-sonnet-4-20250514');
Proxy.Start;

Streaming

The proxy supports SSE streaming in buffered mode:

1. Client sends "stream": true in the request
2. Proxy calls TLiteLLM.ChatRaw with streaming enabled
3. TProxyStreamAccumulator.OnDelta receives plain text deltas and wraps each into an OpenAI-format SSE chunk
4. After streaming completes, the proxy sends all buffered chunks at once

This means the response is not truly real-time (tokens don't arrive one-by-one at the client), but the response format is correct SSE that any OpenAI-compatible client can parse.

Limitation: Buffered SSE only emits delta.content chunks. Tool calls are not streamed — they appear only in the non-streaming response format. Example streaming output:

data: {"id":"chatcmpl-proxy-...","object":"chat.completion.chunk","created":...,"model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-proxy-...","object":"chat.completion.chunk","created":...,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-proxy-...","object":"chat.completion.chunk","created":...,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":5,"total_tokens":15}}

data: [DONE]

Supported Request Parameters

The proxy extracts and forwards these OpenAI parameters:

Error Handling

Errors are returned in OpenAI format:

{
  "error": {
    "message": "Invalid API key",
    "type": "proxy_error",
    "code": 401
  }
}

HTTP status codes are passed through from the underlying provider:

All errors use "type": "proxy_error" regardless of origin. The status code and message from the underlying provider are preserved in the code and message fields.

CORS

The proxy includes CORS headers on all responses:

Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Authorization, Content-Type

OPTIONS preflight requests return 200 immediately.

Response Format

Non-streaming responses follow the OpenAI chat completion format:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1234567890,
  "model": "gpt-4o",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Hello!"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 5,
    "total_tokens": 15
  }
}

Note: tool_calls is omitted from the message when no tool calls are present (not set to null).

When tool calls are present and content is empty, content is set to null and tool_calls contains the array:

"message": {
  "role": "assistant",
  "content": null,
  "tool_calls": [{
    "id": "call_abc",
    "type": "function",
    "function": {
      "name": "get_weather",
      "arguments": "{\"location\":\"Paris\"}"
    }
  }]
}

Note: arguments in the response is a JSON string (matching OpenAI format), not an object.

Using with OpenAI-Compatible Clients

Any application that speaks OpenAI API can use this proxy. Just point it at http://localhost:4000:

# Python example with openai library
import openai
client = openai.OpenAI(
    base_url="http://localhost:4000/v1",
    api_key="my-secret-proxy-key"
)
response = client.chat.completions.create(
    model="gpt-4o",  # mapped to gemini/gemini-2.5-flash via MODEL_MAP
    messages=[{"role": "user", "content": "Hello"}]
)

// TypeScript with @anthropic-ai/sdk or any OpenAI-compatible library
const openai = new OpenAI({
  baseURL: 'http://localhost:4000/v1',
  apiKey: 'my-secret-proxy-key'
});