Every engineering team starts with the exact same architecture: one developer's credit card tied directly to an OpenAI API key pasted into three different .env files. Three months down the line, half the engineering department is experimenting with Anthropic's Claude 3.5 Sonnet, a couple of researchers are testing local models on an internal vLLM cluster, and finance is banging on your desk demanding to know which junior developer burnt through £4,000 running continuous integration loops over gpt-4o.
Enter LiteLLM Proxy by BerriAI (GitHub: BerriAI/litellm). It is an open-source, high-throughput API proxy that standardises over 100+ large language model backends behind a single, OpenAI-compatible interface, giving infrastructure teams centralised access control, virtual keys, rate limiting, and granular cost tracking.
What Is LiteLLM Proxy?
┌────────────────────────────────────────────────────────┐
│ Client Applications / SDKs │
│ (OpenAI Client, LangChain, Cursor, cURL) │
└───────────────────────────┬────────────────────────────┘
│ Standard OpenAI Request
▼
┌────────────────────────────────────────────────────────┐
│ LiteLLM Proxy Gateway │
│ ┌──────────────────┬─────────────────┬──────────────┐ │
│ │ Virtual Key Auth │ Spend Tracking │ Rate Limits │ │
│ └──────────────────┴─────────────────┴──────────────┘ │
│ ┌──────────────────┬─────────────────┬──────────────┐ │
│ │ Fallback Routing │ Model Aliasing │ Prometheus │ │
│ └──────────────────┴─────────────────┴──────────────┘ │
└───────┬───────────────────┼────────────────────┬───────┘
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ OpenAI API │ │ Anthropic │ │ vLLM/Ollama │
│ (gpt-4o) │ │ (Claude 3.5) │ │ (Local Llama)│
└──────────────┘ └──────────────┘ └──────────────┘
LiteLLM Proxy is an enterprise-grade reverse proxy designed specifically for LLM traffic. Rather than having application code talk directly to individual model providers using fragmented SDKs, every request targets the LiteLLM gateway.
The proxy intercepts the call, authenticates incoming requests using internal virtual keys, evaluates rate limits, checks spend allowances against a PostgreSQL database, logs latencies to Prometheus, and routes the payload to the desired provider. If an upstream vendor suffers a partial outage or throws a 429 Rate Limit Exceeded, LiteLLM automatically re-routes the query to a pre-configured fallback model.
Key Architectural Strengths
| Feature | Direct Provider Integration | LiteLLM Proxy Gateway |
|---|---|---|
| API Format | Fragmented per vendor SDK | 100% unified OpenAI schema |
| Credential Management | Keys scattered across microservices | Master keys stored in proxy only |
| Cost Attribution | Shared vendor invoices | Granular spend per team/key/user |
| Failover Routing | Manual application-level try/catch | Configurable automated fallbacks |
| Observability | Vendor-dependent dashboards | Native Prometheus, OpenTelemetry, Datadog |
1. Universal Drop-in Compatibility
Because LiteLLM exposes endpoints that mirror the official OpenAI /v1/chat/completions, /v1/embeddings, and /v1/models specifications, your downstream tools require zero rewrites. Point your LangChain scripts, Claude Code setups, or Cursor IDE configs to http://localhost:4000/v1 with a virtual key, and they operate identically.
2. Virtual Keys with Guardrails
Instead of distributing raw vendor tokens, administrators issue virtual keys (prefixed with sk-litellm-). Each virtual key can enforce:
- Hard monthly or daily spending caps (in USD or GBP).
- Model whitelists (e.g., allow
claude-3-5-haiku, block expensive reasoning models). - Requests-per-minute (RPM) and tokens-per-minute (TPM) ceilings.
3. Enterprise Observability Out-of-the-Box
The proxy natively exports operational metrics straight into Prometheus on /metrics. You get immediate visibility over total prompt tokens, completion tokens, upstream provider latency breakdowns, error rates, and key-specific spend without instrumenting application runtimes.
Step-by-Step Deployment Guide
Deploying a production-ready LiteLLM proxy requires Docker and a lightweight YAML configuration file.
Step 1: Create the Model Configuration (config.yaml)
Define your upstream providers, model aliases, and fallback policies.
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: "os.environ/OPENAI_API_KEY"
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: "os.environ/ANTHROPIC_API_KEY"
- model_name: local-llama
litellm_params:
model: openai/meta-llama/Llama-3.1-8B-Instruct
api_base: "http://host.docker.internal:8000/v1"
api_key: "none"
router_settings:
routing_strategy: latency-based-routing
fallbacks:
- gpt-4o: ["claude-sonnet"]
general_settings:
master_key: "sk-litellm-master-key-change-me"
Step 2: Spin Up the Container
Run the gateway using the official Docker image. Mount your configuration file and pass your vendor tokens as environment variables:
docker run -d \
--name litellm-proxy \
-p 4000:4000 \
-v $(pwd)/config.yaml:/app/config.yaml \
-e OPENAI_API_KEY="sk-proj-your-actual-key" \
-e ANTHROPIC_API_KEY="sk-ant-your-actual-key" \
ghcr.io/berriai/litellm:main-latest \
--config /app/config.yaml --port 4000 --num_workers 4
Step 3: Mint a Virtual Key with Spend Caps
Generate a scoped virtual API key for a specific team via the management API:
curl -X POST "http://localhost:4000/key/generate" \
-H "Authorization: Bearer sk-litellm-master-key-change-me" \
-H "Content-Type: application/json" \
-d '{
"models": ["gpt-4o", "claude-sonnet"],
"max_budget": 50.00,
"budget_duration": "30d",
"metadata": {"team": "frontend-dev"}
}'
The response returns a token formatted as sk-litellm-.... The team can now use this key until they reach the budget threshold, at which point the proxy cleanly rejects further calls with an informative HTTP 400 payload.
Step 4: Execute a Test Completion Call
Verify your setup using standard Python code:
from openai import OpenAI
client = OpenAI(
api_key="sk-litellm-your-generated-key",
base_url="http://localhost:4000/v1",
)
response = client.chat.completions.create(
model="claude-sonnet",
messages=[
{"role": "system", "content": "You are an infrastructure specialist."},
{"role": "user", "content": "Explain circuit breakers in microservices in two sentences."},
],
)
print(response.choices[0].message.content)
Prometheus Scrape Configuration
To capture telemetry, point your standard Prometheus scraper directly at the proxy's health endpoint:
scrape_configs:
- job_name: 'litellm'
scrape_interval: 10s
metrics_path: '/metrics'
static_configs:
- targets: ['localhost:4000']
Key exported metrics include litellm_requests_metric_total, litellm_deployment_latency_per_output_token, and litellm_spend_metric_total.
Why LiteLLM Proxy Stands Out
The common community debate on Reddit and developer forums usually boils down to: "Why not write a basic FastAPI wrapper ourselves?"
Writing a reverse proxy that parses incoming streaming tokens (text/event-stream), calculates token usage on the fly for multiple competing tokeniser formats (tiktoken vs Anthropic token counts), tracks concurrent requests, handles retries gracefully, and normalises response objects across vendors is hundreds of hours of undifferentiated heavy lifting. LiteLLM proxy solves the operational plumbing out of the box, letting your team treat model providers as interchangeable infrastructure commodities.