LiteLLM

LiteLLM functions as an open-source AI gateway and LLM proxy, designed to centralize and manage an organization's entire AI stack behind a single OpenAI-compatible API key. It enables platform teams to provide company-wide access to a vast array of AI models, agents, and Multi-Cloud Providers (MCPs).

Core Functionality

• Unified Access: Consolidates over 140 providers and 1,800+ models under one API, allowing model swapping without code changes and offering day-zero support for new models. It handles key management and integrates with existing secret managers, supporting SSO and scoped access.
• Visibility & Control: Provides detailed tracking of usage and spend per key, user, team, and model, facilitating enterprise chargeback. Users can set hard budgets and rate limits, with requests stopping once caps are hit. Features include leaked-key protection, model access control, guardrails, and a comprehensive audit log.
• Cost Optimization: Maximizes AI ROI through features like auto-routing, which directs prompts to the most appropriate (and often cheapest) model. It supports load balancing across providers and regions, response/semantic caching to avoid redundant payments, and prompt compression to reduce token usage.

Deployment & Performance

The gateway is self-hostable, supporting deployment via official Helm charts, Terraform modules, and Docker images. It runs on standard infrastructure like Postgres and Redis, scales with Kubernetes, and can operate in fully air-gapped environments. LiteLLM boasts sub-millisecond overhead, adding only 0.66 ms at p99, making it highly performant.

Security Features

Security is a core focus, with Cosign-signed images, vulnerability scanning, and robust identity management via SSO, JWT auth, and RBAC. It includes PII masking and prompt-injection guardrails, pulls secrets from secure vaults, and offers self-hosting without telemetry to ensure data residency.