How to Self-Host a Private LLM Gateway (Stop Scattering API Keys Across Cloud AI Tools)

Bottom line up front: If your team uses AI tools, you almost certainly have API keys for OpenAI, Anthropic, or Google pasted into a dozen different places — a .env file on someone's laptop, a Notion doc, a Slack DM from six months ago, three different CI pipelines. Nobody can tell you who's using what, how much it costs, or what happens to that key when the person who created it leaves. A self-hosted LLM gateway fixes all of that with one box you control.

This guide walks through standing up LiteLLM — a free, open-source proxy — as the single door all your AI traffic passes through, whether it's headed to a local Ollama model or a cloud provider.

Last updated: 2026-07-05

The Problem: API Keys Are Your Real Attack Surface

Most privacy guides for AI tools focus on prompts — what you type into ChatGPT, whether it trains on your data. That matters, but there's a bigger, quieter problem: the credentials themselves.

A raw OpenAI or Anthropic API key is a bearer token. Whoever holds it can spend your budget, pull your usage logs, and in some misconfigurations, see prompt history tied to your account. Once that key is in a .env file, a CI secret, or a contractor's laptop, you've lost the ability to answer basic questions:

Which tool is actually using this key right now?
What happens when the contractor's engagement ends?
Can we see exactly what was sent to the model last Tuesday?
Can we cut off one integration without breaking three others?

Without a gateway, the answer to all of these is "not easily." With one, it's a single revoke command and a queryable log.

What a Private LLM Gateway Actually Does

A gateway sits between your tools (IDE plugins, internal scripts, chatbots, agents) and the model providers (OpenAI, Anthropic, Google, or your own Ollama instance). Every request goes through it. The gateway:

Holds the real provider API keys — nothing downstream ever sees them
Issues its own short-lived, scoped virtual keys to each tool or team member
Logs every request (model, tokens, cost, timestamp, requesting key) to storage you control
Enforces per-key budgets and rate limits
Routes requests to the cheapest or most private backend that satisfies the task — local Ollama for anything sensitive, cloud only when you explicitly allow it

This is the same architecture pattern banks use for internal API access to external services. Applied to AI, it turns "everyone has their own OpenAI key" into "everyone has a scoped, revocable, logged credential to your gateway."

What You Need Before You Start

A small always-on machine: a spare Mac Mini, a NUC, a cheap VPS, or a Raspberry Pi 5 (16 GB models will run fine; heavier local inference wants more RAM)
Docker installed, or Python 3.9+ if you'd rather run it natively
Your existing cloud provider API keys (OpenAI, Anthropic, etc.) — you're about to make these the last time anyone but the gateway sees them directly
Optionally, Ollama already running if you want local models in the routing mix

This guide assumes macOS or Linux for the host machine. A VPS on Railway or a home server both work well since the gateway itself is lightweight.

Step 1: Run LiteLLM Proxy

The fastest path is Docker:

```bash

docker run -d \

--name litellm-gateway \

-p 4000:4000 \

-e LITELLM_MASTER_KEY="sk-set-a-long-random-string-here" \

-v $(pwd)/litellm-config.yaml:/app/config.yaml \

ghcr.io/berriai/litellm:main-latest \

--config /app/config.yaml

```

The LITELLM_MASTER_KEY is the one credential that can create and revoke other keys — treat it like root. Generate it with:

```bash

openssl rand -hex 32

```

Never commit this value to a repo. Store it in your gateway host's environment, not in version control.

Step 2: Define Your Routing Config

This is where the privacy decisions actually live. Create litellm-config.yaml:

```yaml

model_list:

- model_name: local-default

litellm_params:

model: ollama/llama3.1

api_base: http://host.docker.internal:11434

- model_name: cloud-fallback

litellm_params:

model: gpt-4o-mini

api_key: os.environ/OPENAI_API_KEY

- model_name: cloud-research

litellm_params:

model: anthropic/claude-sonnet-5

api_key: os.environ/ANTHROPIC_API_KEY

general_settings:

database_url: os.environ/DATABASE_URL

store_prompts_in_spend_logs: false

```

Two lines matter most here. store_prompts_in_spend_logs: false keeps your usage logs to metadata only — model, tokens, cost, timestamp — without retaining the actual prompt text in the gateway's own database. And the model_name aliases are what your tools request, not the underlying provider name, so you can swap local-default from Llama 3.1 to Qwen 2.5 later without touching a single downstream config.

The real API keys (OPENAI_API_KEY, ANTHROPIC_API_KEY) live as environment variables on the gateway host only. They never touch a client machine again.

Step 3: Issue Scoped Virtual Keys

Instead of handing your real OpenAI key to a contractor's IDE plugin, issue a virtual key scoped to exactly what they need:

```bash

curl -X POST http://localhost:4000/key/generate \

-H "Authorization: Bearer sk-set-a-long-random-string-here" \

-H "Content-Type: application/json" \

-d '{

"models": ["local-default", "cloud-fallback"],

"max_budget": 20,

"duration": "30d",

"key_alias": "contractor-jane-ide"

```

This key can only reach the two models you named, can't spend more than $20, and expires in 30 days automatically. When the engagement ends early, one revoke call kills it instantly — no waiting for someone to remember to rotate a shared .env file.

Give each person, script, and CI pipeline its own key alias. When something looks wrong in the logs, you'll know exactly what to unplug.

Step 4: Point Your Tools at the Gateway

Any tool that speaks the OpenAI API format can be redirected with two settings: base URL and API key. In VS Code's Continue extension, a Python script, or an n8n workflow, that typically looks like:

```

OPENAI_BASE_URL=http://your-gateway-host:4000

OPENAI_API_KEY=

```

From the tool's perspective, nothing changed — it still calls an OpenAI-compatible endpoint. But now every request is logged, budgeted, and revocable, and the tool never held a real provider credential to begin with.

Step 5: Make Local the Default, Cloud the Exception

The biggest privacy win of a gateway isn't key management — it's that you can now enforce routing policy in one place instead of trusting every tool and teammate to remember it themselves.

Set local-default (your Ollama model) as the alias most tools point to by default. Reserve cloud-fallback and cloud-research for virtual keys explicitly granted access to them — reviewing client contracts, debugging proprietary code, or anything with an NDA attached should never have a cloud model in its routing path at all.

This is the same principle covered in our private AI research stack guide — local inference by default, cloud only when the task genuinely requires live web access or a larger model, and never for anything sensitive.

Encrypting the Gateway's Logs and Config at Rest

Your gateway's database now contains something valuable: a complete map of who used which model, when, and how much it cost. That's useful for auditing, but it's also exactly the kind of metadata you don't want sitting in a plaintext SQLite file on a shared server.

Back the gateway's Postgres database and config directory with an encrypted volume, then sync backups through Tresorit rather than a general-purpose cloud drive. Tresorit encrypts client-side before anything leaves your machine, so even a compromised backup destination never exposes the raw logs — only ciphertext Tresorit itself cannot read either.

Affiliate Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no extra cost to you. We only recommend products we genuinely believe in. This helps support our work and allows us to continue providing free content.

Proton Pass is free for basic use; Proton Unlimited bundles it with Drive, Mail, and VPN for $9.99/month, and PrivateAI readers get 100% back on the first month.

When You Still Need a Cloud Model

Local models handle most day-to-day work, but some tasks genuinely need live web results or a frontier model's reasoning — competitive research, fast-moving news, or anything requiring current information a local model's training cutoff can't provide. Route that traffic through a cloud-research alias pointed at Perplexity, with its Private Mode setting enabled so queries aren't retained for training.