LLMForge is an AI API gateway with hard budget caps, semantic circuit breakers, and local-first routing. Route between local GPUs, network machines, and cloud. Cap spend per API key. Strip secrets. Audit every call. Extend with built-in, JS/TS, or webhook plugins. OpenAI-compatible, one env var.
Hard budget caps kill runaway agents before they cost $30K. Route between local and cloud. Secure every prompt. Extend with plugins. All in a single Rust binary.
Per-API-key budget caps. Requests rejected when limit hit. Circuit breaker detects agent loops — exact hash + embedding similarity — and kills runaway sessions before they drain budget. 5 similar requests in 60s → 120s block. Cost tracking per key via /v1/budget.
Multi-tier failover: local GPU → machines on your network → cloud. Route profiles (/v1/code, /v1/fast, /v1/reasoning) with fixed model, pattern, plugin chain. Cloud is opt-in, not the default.
PII redaction strips API keys, tokens, emails before they reach any model. Audit logging records every call. 100% on your server by default. Bearer token auth per key.
Built-in plugins (Rust, zero overhead): PII redact, prompt harness, code formatter, audit log. JS/TS plugins via embedded V8 (registerPlugin() — no server, no network). HTTP webhooks for any language. Fusion is a plugin pattern — route to multiple models, vote, cascade with tests.
Write plugins in JS/TS — runs in-process via embedded V8, no server needed:
Agent loops burn $30K before billing alerts fire. Rate limits don't account for token density. Dashboards show the damage after it's done. LLMForge enforces hard caps and kills runaway agents — before the bill, not after.
LLMForge runs on a single GPU server you already own. Route, cache, and cap spend for every engineer. Cloud fallback only kicks in for the ~5% of hard problems.
| Team size | Requests / month | Current cost / mo | LLMForge / mo | Annual savings |
|---|---|---|---|---|
| 5 engineers | 2,000 | $20 | $0.10 | $239 |
| 50 engineers | 50,000 | $500 | $2.10 | $5,975 |
| 200 engineers | 500,000 | $6,750 | $11.70 | $80,860 |
Based on GPT-5.2 pricing ($1.62/1K calls) vs LLMForge on existing hardware. LLMForge cost = electricity + ~5% cloud fallback. Does not include GPU purchase — uses hardware you already have.
LiteLLM is the OSS distribution leader — 50K+ GitHub stars, Python, 100+ providers. But it's heavy: 500MB+ RAM, Postgres + Redis, 3-5s startup, 1.7-4x throughput drop in production. LLMForge is a 7.5MB Rust binary that starts in 8ms.
| Metric | LLMForge (Rust) | LiteLLM (Python) |
|---|---|---|
| Binary size | 7.5MB | ~200MB (pip + deps) |
| Startup time | ~8ms | ~3-5s |
| Memory (idle) | ~10MB | ~500MB+ (needs Postgres + Redis) |
| External deps | None | Postgres + Redis |
| Throughput overhead | Minimal | 1.7-4x drop reported in production |
| Deployment | Single binary | Docker Compose (3+ containers) |
LiteLLM has wider provider coverage (100+) and a larger community. LLMForge is purpose-built for teams that want a lightweight, local-first gateway with circuit breakers and plugins — not a full microservices stack.
Dashboards show you the damage after it's done. LLMForge stops it before it happens. Hard caps, circuit breakers, routing control, security, and plugins — all in one Rust binary.
Per-API-key spend limits. Requests rejected at the limit. No surprise bills, no postmortems.
Semantic loop detection — exact hash + embedding similarity. Kills runaway agents at iteration 3, not $30K later.
Multi-tier: local GPU → network machines → cloud. Per-request routing. Multi-endpoint profiles with fixed model and budget.
100% on your server by default. PII redaction strips secrets. Audit logging records every call. Cloud is opt-in.
PII redact, prompt harness, code formatter, audit log. Rust code — zero network overhead. Enable per route profile.
Write plugins in JavaScript or TypeScript via embedded V8. registerPlugin() with 3 hooks. No server, no network — runs in-process. Feature-gated.
HTTP hooks at 3 extension points: pre_request, prompt_filter, post_response. Any language, any service. Zero-config.
Plugin patterns: Validated Fallback, Validated Consensus, Self-Consistency, Stream Race. Route to multiple models, vote, cascade.
Exact match (instant) + semantic match (embedding similarity). $0 on cached calls. 54x faster on hits.
SSE passthrough + stream race. Race models, first response wins. Works with Cursor, Claude Code, OpenCode.
Fusion is a plugin pattern — route to multiple local models, vote on results. +10pp on HumanEval when you need it. Off by default for speed. The core product is flow control and security. Fusion is one thing the plugin system enables.
| Configuration | HumanEval Pass Rate | Cost / 1K calls |
|---|---|---|
| LLMForge — Local (single best model) | 81.7% | $0.00 |
| LLMForge — Local + Fusion (5 models) | 92.1% | $0.00 |
| LLMForge — Fusion (3 mixed models) | 84.1% | $0.00 |
| GPT-5.2 (cloud, for comparison) | ~95% | $1.62 |
3pp accuracy gap. 100% cost gap. Local handles the 95%, cloud fills the 5%. Full 164-problem HumanEval, pass@1. Run the benchmarks yourself: benchmarks/
You run Ollama on your Mac Studio for side projects but pay OpenAI for the hard stuff. There's no way to route easy prompts to local and hard prompts to cloud. Your local GPUs sit idle 90% of the time. You have no control over the flow.
/v1/fast hits local, /v1/reasoning cascades to cloud if local can't answer. Your local GPUs handle 95% of traffic. Cloud fills the gap. You control exactly where each request goes.
Every prompt your engineers send leaves your network — source code, internal docs, customer data. No PII redaction. No audit trail. No way to extend the pipeline. You're paying $6,750/mo for the privilege of losing control over your data.
Free, open source, MIT licensed. No signup, no cloud, no lock-in.