← Back to all spotlights

Deploy a Self-Hosted LLM Gateway with LiteLLM Proxy

Centralise 100+ LLMs behind a unified, OpenAI-compatible proxy with virtual API keys, automated fallbacks, and native Prometheus metrics.

P24
By Pickwise24 Editorial Team
Verified Open-Source Review

Every engineering team starts with the exact same architecture: one developer's credit card tied directly to an OpenAI API key pasted into three different .env files. Three months down the line, half the engineering department is experimenting with Anthropic's Claude 3.5 Sonnet, a couple of researchers are testing local models on an internal vLLM cluster, and finance is banging on your desk demanding to know which junior developer burnt through £4,000 running continuous integration loops over gpt-4o.

Enter LiteLLM Proxy by BerriAI (GitHub: BerriAI/litellm). It is an open-source, high-throughput API proxy that standardises over 100+ large language model backends behind a single, OpenAI-compatible interface, giving infrastructure teams centralised access control, virtual keys, rate limiting, and granular cost tracking.


What Is LiteLLM Proxy?


┌────────────────────────────────────────────────────────┐
│               Client Applications / SDKs               │
│        (OpenAI Client, LangChain, Cursor, cURL)        │
└───────────────────────────┬────────────────────────────┘
                            │ Standard OpenAI Request
                            ▼
┌────────────────────────────────────────────────────────┐
│                  LiteLLM Proxy Gateway                 │
│  ┌──────────────────┬─────────────────┬──────────────┐ │
│  │ Virtual Key Auth │ Spend Tracking  │ Rate Limits  │ │
│  └──────────────────┴─────────────────┴──────────────┘ │
│  ┌──────────────────┬─────────────────┬──────────────┐ │
│  │ Fallback Routing │ Model Aliasing  │ Prometheus   │ │
│  └──────────────────┴─────────────────┴──────────────┘ │
└───────┬───────────────────┼────────────────────┬───────┘
        │                   │                    │
        ▼                   ▼                    ▼
┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│ OpenAI API   │     │ Anthropic    │     │ vLLM/Ollama  │
│ (gpt-4o)     │     │ (Claude 3.5) │     │ (Local Llama)│
└──────────────┘     └──────────────┘     └──────────────┘

LiteLLM Proxy is an enterprise-grade reverse proxy designed specifically for LLM traffic. Rather than having application code talk directly to individual model providers using fragmented SDKs, every request targets the LiteLLM gateway.

The proxy intercepts the call, authenticates incoming requests using internal virtual keys, evaluates rate limits, checks spend allowances against a PostgreSQL database, logs latencies to Prometheus, and routes the payload to the desired provider. If an upstream vendor suffers a partial outage or throws a 429 Rate Limit Exceeded, LiteLLM automatically re-routes the query to a pre-configured fallback model.


Key Architectural Strengths

FeatureDirect Provider IntegrationLiteLLM Proxy Gateway
API FormatFragmented per vendor SDK100% unified OpenAI schema
Credential ManagementKeys scattered across microservicesMaster keys stored in proxy only
Cost AttributionShared vendor invoicesGranular spend per team/key/user
Failover RoutingManual application-level try/catchConfigurable automated fallbacks
ObservabilityVendor-dependent dashboardsNative Prometheus, OpenTelemetry, Datadog

1. Universal Drop-in Compatibility

Because LiteLLM exposes endpoints that mirror the official OpenAI /v1/chat/completions, /v1/embeddings, and /v1/models specifications, your downstream tools require zero rewrites. Point your LangChain scripts, Claude Code setups, or Cursor IDE configs to http://localhost:4000/v1 with a virtual key, and they operate identically.

2. Virtual Keys with Guardrails

Instead of distributing raw vendor tokens, administrators issue virtual keys (prefixed with sk-litellm-). Each virtual key can enforce:

  • Hard monthly or daily spending caps (in USD or GBP).
  • Model whitelists (e.g., allow claude-3-5-haiku, block expensive reasoning models).
  • Requests-per-minute (RPM) and tokens-per-minute (TPM) ceilings.

3. Enterprise Observability Out-of-the-Box

The proxy natively exports operational metrics straight into Prometheus on /metrics. You get immediate visibility over total prompt tokens, completion tokens, upstream provider latency breakdowns, error rates, and key-specific spend without instrumenting application runtimes.


Step-by-Step Deployment Guide

Deploying a production-ready LiteLLM proxy requires Docker and a lightweight YAML configuration file.

Step 1: Create the Model Configuration (config.yaml)

Define your upstream providers, model aliases, and fallback policies.


model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: "os.environ/OPENAI_API_KEY"

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: "os.environ/ANTHROPIC_API_KEY"

  - model_name: local-llama
    litellm_params:
      model: openai/meta-llama/Llama-3.1-8B-Instruct
      api_base: "http://host.docker.internal:8000/v1"
      api_key: "none"

router_settings:
  routing_strategy: latency-based-routing
  fallbacks:
    - gpt-4o: ["claude-sonnet"]

general_settings:
  master_key: "sk-litellm-master-key-change-me"

Step 2: Spin Up the Container

Run the gateway using the official Docker image. Mount your configuration file and pass your vendor tokens as environment variables:


docker run -d \
  --name litellm-proxy \
  -p 4000:4000 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY="sk-proj-your-actual-key" \
  -e ANTHROPIC_API_KEY="sk-ant-your-actual-key" \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml --port 4000 --num_workers 4

Step 3: Mint a Virtual Key with Spend Caps

Generate a scoped virtual API key for a specific team via the management API:


curl -X POST "http://localhost:4000/key/generate" \
  -H "Authorization: Bearer sk-litellm-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["gpt-4o", "claude-sonnet"],
    "max_budget": 50.00,
    "budget_duration": "30d",
    "metadata": {"team": "frontend-dev"}
  }'

The response returns a token formatted as sk-litellm-.... The team can now use this key until they reach the budget threshold, at which point the proxy cleanly rejects further calls with an informative HTTP 400 payload.

Step 4: Execute a Test Completion Call

Verify your setup using standard Python code:


from openai import OpenAI

client = OpenAI(
    api_key="sk-litellm-your-generated-key",
    base_url="http://localhost:4000/v1",
)

response = client.chat.completions.create(
    model="claude-sonnet",
    messages=[
        {"role": "system", "content": "You are an infrastructure specialist."},
        {"role": "user", "content": "Explain circuit breakers in microservices in two sentences."},
    ],
)

print(response.choices[0].message.content)

Prometheus Scrape Configuration

To capture telemetry, point your standard Prometheus scraper directly at the proxy's health endpoint:


scrape_configs:
  - job_name: 'litellm'
    scrape_interval: 10s
    metrics_path: '/metrics'
    static_configs:
      - targets: ['localhost:4000']

Key exported metrics include litellm_requests_metric_total, litellm_deployment_latency_per_output_token, and litellm_spend_metric_total.


Why LiteLLM Proxy Stands Out

The common community debate on Reddit and developer forums usually boils down to: "Why not write a basic FastAPI wrapper ourselves?"

Writing a reverse proxy that parses incoming streaming tokens (text/event-stream), calculates token usage on the fly for multiple competing tokeniser formats (tiktoken vs Anthropic token counts), tracks concurrent requests, handles retries gracefully, and normalises response objects across vendors is hundreds of hours of undifferentiated heavy lifting. LiteLLM proxy solves the operational plumbing out of the box, letting your team treat model providers as interchangeable infrastructure commodities.

🛡️ Editorial Standards & Methodology

Every repository featured on Pickwise24 undergoes testing on local workstation hardware before publication. We verify CLI installation steps, review open-source repository licensing, benchmark computational footprint, and evaluate architectural trade-offs to provide genuine, high-utility developer intelligence.