How to Run AI Locally: The Complete Guide to Private LLMs That Never Phone Home

Every prompt you send to ChatGPT, Claude, or Gemini is logged, potentially used for training, and stored on servers you have zero control over. For developers, legal professionals, security researchers, and anyone handling sensitive data, that's not a trade-off — it's a liability.

The good news: you don't have to choose between capable AI and actual privacy. Modern open-weight models running on consumer hardware are now competitive with GPT-4 for most day-to-day tasks. This guide walks you through a complete local AI stack — from installing your first model to securing everything that touches it.

Last updated: 2026-03-22

Why "Private Mode" Isn't Actually Private

Before we build anything, let's be precise about the threat model. Most cloud AI providers offer some version of "privacy mode" or claim not to train on your data. Read the fine print:

API usage often has separate data retention policies from consumer products
Even with training opted out, your prompts still transit their infrastructure and are subject to their security posture
Legal hold, government requests, and data breaches are outside your control
Enterprise agreements don't protect you if you're a solo developer or small team

Running locally eliminates all of this. Your prompts go: keyboard → RAM → GPU → response. No network hop. No log entry. No terms of service to re-read every six months.

What You Actually Need to Run AI Locally

The barrier is lower than most people think. Here's the honest hardware breakdown:

Minimum viable (8B models, ~8GB VRAM or unified memory):

Apple M2/M3/M4 with 16GB RAM — excellent performance via Metal
Any NVIDIA GPU with 8GB VRAM (RTX 3070, 4060, etc.)
AMD RX 7900 XT+ with ROCm support (Linux only, more setup)

Comfortable (13B-34B models, real GPT-3.5 competition):

Apple M3 Pro/Max with 36GB unified memory
NVIDIA 3090/4090 (24GB VRAM) — the classic local AI workhorse

Overkill / serious (70B+ models):

Mac Studio or Mac Pro with 64-192GB unified memory
Multi-GPU setups or consumer AI boxes

If you're on Apple Silicon, you already have one of the best local AI platforms available. Unified memory means the GPU and CPU share the same pool, and llama.cpp has outstanding Metal support.

Installing Ollama: The Easiest On-Ramp

Ollama is the de facto standard for running open-weight models locally. It handles model downloading, quantization selection, and serves a local API compatible with the OpenAI spec — which means most AI tools that talk to ChatGPT can be redirected to your machine with one config change.

macOS:

```bash

brew install ollama

ollama serve # starts the local API server on localhost:11434

```

Linux:

```bash

curl -fsSL https://ollama.com/install.sh | sh

```

Windows: Download the installer from ollama.com. WSL2 with CUDA passthrough also works if you prefer a Linux environment.

Once Ollama is running, pull your first model:

```bash

ollama pull llama3.2 # Meta's 3B model — fast, surprisingly capable

ollama pull qwen2.5:14b # Excellent reasoning, strong code

ollama pull deepseek-r1:14b # Best-in-class for technical reasoning

ollama pull mistral-small # Fast, good for summarization

```

Test it immediately:

```bash

ollama run llama3.2 "Explain OAuth2 in plain English"

```

Everything that just happened: local only, zero network egress after the initial model download.

Choosing the Right Model for Your Work

Not all open-weight models are equal. Here's a practical guide for common use cases:

Code generation and review: qwen2.5-coder:14b or deepseek-coder-v2. Both outperform GPT-3.5-turbo on benchmarks for completions and bug finding.

Document analysis and summarization: llama3.2:3b for speed, llama3.1:8b for quality. Feed these your contracts, specs, and internal docs without the data ever leaving your machine.

Reasoning and analysis: deepseek-r1 series. Shows its chain-of-thought reasoning — useful when you want to audit how the model reached a conclusion.

Multilingual work: qwen2.5:7b — strong across 30+ languages.

Instruction following and chat: mistral-small or gemma3:9b — efficient and well-tuned for conversation.

A note on quantization: Ollama defaults to Q4_K_M quantization, which cuts model file size roughly in half with minimal quality loss. If you have headroom, Q6_K or Q8_0 give noticeably sharper outputs for technical work.

Connecting Local AI to Your Existing Tools

This is where local LLMs become genuinely useful rather than just a demo. Because Ollama exposes an OpenAI-compatible API, you can drop it into most AI-enabled tools with a URL change.

VS Code / Cursor:

Configure the API base URL to http://localhost:11434/v1 and use any Ollama model name as the model ID. You now have GitHub Copilot-equivalent code assistance with zero telemetry.

Continue.dev (VS Code extension):

Purpose-built for local AI coding assistance. Native Ollama integration, no config gymnastics.

Open WebUI:

A polished ChatGPT-style interface that connects to Ollama. Run it in Docker:

```bash

docker run -d -p 3000:8080 \

--add-host=host.docker.internal:host-gateway \

-v open-webui:/app/backend/data \

ghcr.io/open-webui/open-webui:main

```

Navigate to localhost:3000 — you have a full-featured AI chat interface with conversation history, model switching, and file uploads, all running locally.

n8n / Make / Zapier: Use Ollama's REST API (POST localhost:11434/api/generate) as a custom HTTP action to add local AI to your automations.

Securing Everything Around Your Local AI Stack

Running models locally handles the most sensitive part — your prompts and responses. But there are adjacent attack surfaces worth hardening.

Model downloads: Always pull from ollama.com or Hugging Face with verified checksums. Malicious fine-tunes exist. Check that the model card comes from the original developer organization.

Network exposure: By default, Ollama only listens on 127.0.0.1. Keep it that way unless you explicitly need LAN access. If you're sharing a machine or running a home server, add auth middleware (nginx + htpasswd at minimum).

Prompt/output storage: If you're logging conversations or feeding Ollama outputs into other workflows, those files can contain sensitive data. Don't sync them unencrypted.

For encrypted cloud backup of AI outputs, notes, and anything sensitive your local stack produces, Proton Drive gives you end-to-end encrypted storage with desktop sync. Unlike Dropbox or Google Drive, Proton cannot read your files — the encryption keys never leave your device. Free tier covers 1GB; paid plans start at €3.99/mo with a generous referral structure.

Affiliate Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no extra cost to you. We only recommend products we genuinely believe in. This helps support our work and allows us to continue providing free content.

The workflow: default to local for anything sensitive, use cloud AI for truly demanding tasks with non-sensitive inputs, and never paste credentials, PII, or confidential code into any cloud interface regardless of their privacy policy.

Handling Sensitive Files: Encrypted Sync for Your AI Workspace

If you're using local AI for work — drafting documents, analyzing data, reviewing contracts — those output files need the same protection as your prompts. Standard cloud sync (iCloud, Dropbox, Google Drive) scans file contents for compliance and other purposes.

Tresorit is the encrypted alternative used by law firms and healthcare providers precisely because they never hold your decryption keys. Files are encrypted client-side before upload, so even a subpoena to Tresorit returns ciphertext. The Business plan includes secure link sharing, which matters if you're collaborating on AI-generated documents with clients or teammates.

Have a question about your specific hardware or use case? Reply to any newsletter or reach us at the link below.