Best open-source AI gateways in 2026

Every application that calls LLM providers directly accumulates technical debt over time. Different authentication schemes, inconsistent response formats, provider-specific rate limits, fragmented observability, and no centralized failover all increase operational complexity. An AI gateway sits between your application and the providers, normalizing all of that behind a single endpoint.

The open-source category has matured significantly. The tools below are not interchangeable: some are full gateways, one is a routing framework, and they differ meaningfully on performance, provider coverage, deployment complexity, and what you actually get for free versus what requires an enterprise license.

An AI gateway is a proxy that sits between your application code and one or more LLM providers. Instead of calling OpenAI, Anthropic, and Google directly from your application, you call the gateway. The gateway handles routing, failover, caching, rate limiting, cost tracking, and observability for all providers behind a single OpenAI-compatible endpoint.

A router is a component of a gateway, or a standalone tool, that decides which model receives each request. Routing logic ranges from simple round-robin to cost-based routing (send easy queries to a cheap model, hard queries to a strong model) to latency-based routing (send to the fastest provider right now).

Most teams need both. A gateway provides the operational layer, while a router determines which model or provider should handle each request.

LiteLLM is the most widely adopted open-source LLM gateway. MIT-licensed. Self-hostable via Docker. Supports 140+ providers and 1,800+ models through a unified OpenAI-compatible API.

LiteLLM has become the default open-source AI gateway for many teams because it usually adds support for new providers and models very quickly, making it one of the safest choices if provider compatibility is your highest priority.

What it provides: Unified OpenAI-format output across all providers. Virtual keys with team management and per-key budget limits. Latency-based, cost-based, and usage-based routing. A dashboard for spend tracking. Fallback routing when a provider is unavailable.

Where it has limits: No semantic caching (exact-match Redis only). The Python proxy still has throughput limitations at scale; the Rust gateway addresses this but is newer.

License: MIT. Fully open-source with no paywalled core features. Enterprise tier adds SSO and advanced governance.

Best for: Teams that need the broadest possible provider coverage and are comfortable managing self-hosted Python infrastructure. The community and documentation are the strongest in the category.

Deploy LiteLLM on Northflank: One-click template deploys LiteLLM with PostgreSQL, networking, and secrets management pre-configured. Or follow the step-by-step deployment guide for a fully customized setup.

Bifrost is a Go-based AI gateway built for high-throughput production workloads. Apache 2.0 licensed. Written in Go, Bifrost avoids Python's GIL constraint and delivers substantially lower overhead at scale.

What it provides: Multi-provider routing across 20+ providers. Semantic caching (embedding similarity match, not just exact match), which produces higher cache hit rates than Redis exact-match. Native MCP support for agentic workflows. Built-in governance with virtual keys and budget enforcement. Zero-config startup via npx -y @maximhq/bifrost.

Where it has limits: 20+ provider integrations versus LiteLLM's 140+. If you need a niche or emerging model provider, check Bifrost's provider list before committing. Smaller community than LiteLLM.

License: Apache 2.0. Fully open-source.

Best for: Teams hitting Python's performance ceiling, teams building agentic workflows that need MCP support, or teams that need semantic caching at the gateway layer.

Portkey Gateway combines an Apache 2.0 open-source gateway with a proprietary managed platform. The open-source gateway handles routing, retries, and fallbacks, while the managed platform adds observability, guardrails, and additional operational features.

What it provides: Routing across 250+ providers. Guardrails that run content policies on every request before it hits the model (block topics, detect PII, enforce output formats). Semantic caching using embedding similarity. Prompt versioning and management. SDK-first developer experience.

Where it has limits: The guardrails and semantic caching features run on Portkey's infrastructure when using the managed tier. The open-source core provides routing and fallbacks but not the full observability and guardrails stack without the cloud tier. The managed platform adds vendor dependency for teams that need those features.

License: Apache 2.0 (core gateway). Cloud platform is proprietary.

Best for: Teams that want guardrails and semantic caching out of the box without building middleware, and are comfortable with a split between open-source core and managed features.

Kong AI Gateway extends Kong Gateway, the widely deployed open-source API gateway built on NGINX and OpenResty, with LLM-specific routing plugins. If your organization already runs Kong across its API estate, the AI Gateway plugin slots into existing infrastructure rather than introducing a new system.

What it provides: Multi-LLM routing through the AI Proxy plugin. Support for OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Cohere, and Mistral. Rate limiting, authentication, and logging on Kong's proven infrastructure. OIDC integration with Okta, Entra ID, and any OIDC-compatible provider. A mature plugin ecosystem covering mTLS, OpenTelemetry, and more.

Where it has limits: The open-source base Kong Gateway is free. The AI-specific plugins that matter for LLM routing, including the AI Proxy plugin, PII redaction, and enterprise SSO, sit behind Kong Enterprise. This makes Kong the heaviest licensing consideration in this list. No semantic caching. No virtual key budgeting.

License: Kong Gateway is Apache 2.0. AI plugins require Kong Enterprise for production use.

Best for: Enterprises already operating a Kong API mesh that want to extend it to AI workloads without introducing a new gateway system. Not the right starting point for teams without existing Kong infrastructure.

RouteLLM is not an AI gateway. It is an open-source routing framework from LMSYS and Berkeley Research that decides which model should handle a request. Instead of managing authentication, observability, or provider integrations, it focuses entirely on optimizing model selection to reduce cost while maintaining response quality.

What it provides: Trained routing models that classify each incoming request by difficulty and route accordingly. On LMSYS's own benchmarks, RouteLLM achieves roughly 95% of GPT-4-level quality at 14% of the strong-model call volume. Multiple routing strategies, including matrix factorization, BERT classifiers, and causal LLM-based routers.

Where it has limits: RouteLLM is a router, not a gateway. It does not provide authentication, caching, observability, rate limiting, or provider normalization. It answers one question: which model should handle this request? You still need a gateway around it. The benchmark numbers are directional. Re-prove them on your own traffic before treating them as production targets.

License: Apache 2.0.

Best for: Teams that have already deployed a gateway and want to add cost-optimizing routing logic on top. Not a standalone replacement for a gateway.

Choosing an AI gateway is only part of the decision. The gateway itself still needs to run as production infrastructure with the same operational requirements as any other backend service.

For teams running LiteLLM, Bifrost, or another self-hosted gateway, this means:

• Secrets management for provider API keys, which should never appear in environment files or source code
• RBAC so different teams have scoped access to different virtual keys and provider configurations
• Audit logging of every request routed through the gateway for compliance and debugging
• Autoscaling as request volume grows, particularly for gateways handling thousands of requests per second
• Managed cloud or BYOC depending on whether workloads run on shared infrastructure or inside your own cloud account

Northflank deploys and manages AI gateways as production services. Deploy LiteLLM in minutes using the one-click template, or follow the step-by-step LiteLLM deployment guide for a fully customized setup.

Running an AI gateway is rarely just about the gateway itself. Most production deployments also include supporting services such as PostgreSQL or Redis, inference servers like vLLM, AI applications such as Open WebUI, and internal services that all need to communicate securely. Northflank lets you deploy and manage that entire AI stack on one platform with private networking, service discovery, integrated CI/CD, preview environments for every pull request, autoscaling, GPU workloads, and managed cloud or self-serve BYOC.

Secret groups keep provider API keys out of source code and inject them at runtime. RBAC at organisation, project, and environment level governs who can access gateway configuration. Audit logging integrates with your SIEM, while autoscaling automatically adjusts capacity as traffic changes.

For organisations with strict security or sovereignty requirements, Northflank supports self-serve BYOC and a forward-deployed control plane that runs entirely inside your own cloud or on-premises environment. This provides a consistent deployment experience while keeping AI traffic and operational data within your network boundary.

It depends on what matters most to your team.

• Choose LiteLLM if provider compatibility is your highest priority.
• Choose Bifrost if you need maximum throughput, semantic caching, or native MCP support.
• Choose Portkey if you want built-in guardrails and observability with the option of a managed platform.
• Choose Kong AI Gateway if your organization already standardizes on Kong for API management.
• Choose RouteLLM if you already have an AI gateway and want to optimize routing decisions between models to reduce cost.

An AI gateway is the full proxy layer: authentication, provider normalization, caching, rate limiting, cost tracking, observability, and routing. An LLM router is a component that decides which model or provider should handle a given request. Most gateways include routing logic. RouteLLM is an example of a standalone router that needs a gateway around it.

LiteLLM has the largest provider ecosystem (140+) and the strongest community. For teams that need maximum provider coverage and are comfortable with Python infrastructure, it remains the default choice. For teams that need higher throughput, semantic caching, or MCP support, Bifrost is the stronger option.

The gateway software itself is free. The cost is the infrastructure it runs on: compute, storage for logs and caches, and the operational time to manage it. LiteLLM requires PostgreSQL alongside the proxy. Bifrost is a single Go binary with minimal external dependencies. Both run on standard compute with no GPU requirement.

Semantic caching returns a cached response when a new request is semantically similar to a previous one, rather than requiring an exact text match. This produces higher cache hit rates than exact-match Redis caching. Bifrost supports semantic caching natively. Portkey supports it on the cloud tier. LiteLLM and Kong do not currently support semantic caching.

Yes. All five tools can be self-hosted. For teams that need the gateway and all AI traffic to stay inside their own cloud account, Northflank BYOC deploys the gateway and its supporting infrastructure into your own AWS, GCP, Azure, or on-premises environment.

Open-source AI gateways have reached the point where most teams no longer need to integrate every LLM provider individually. Whether your priority is provider compatibility, high throughput, semantic caching, enterprise governance, or intelligent routing, there is now a mature open-source option that fits those requirements.

The remaining challenge is running that gateway reliably in production. Secrets management, RBAC, audit logging, autoscaling, and deployment into your own cloud become just as important as the gateway itself. Northflank provides the production platform for running AI gateways and the rest of your AI stack, whether you're deploying LiteLLM, Bifrost, inference servers, or internal AI applications as managed services or inside your own AWS, Azure, GCP, or on-premises environment.

• What is BYOC in cloud computing?: How the BYOC deployment model works and when it applies to AI infrastructure decisions.
• Top cloud platforms for enterprise AI deployment: How Northflank, AWS, Google Cloud, Azure, and OCI compare for GPU support, BYOC, compliance, and production AI workloads.
• Best enterprise AI sandbox platforms: A comparison of enterprise sandbox platforms covering SOC 2, HIPAA, BYOC, and microVM isolation for AI agent execution.
• Best options for BYOC in cloud computing: How BYOC platforms compare on deployment targets, self-serve access, and enterprise controls.