Two years ago, running a capable LLM on a laptop was a stunt. You could do it, but the model was dumb, the tokens were slow, and the fans were loud. In 2026 the math has flipped. A modern laptop — Apple Silicon, a Ryzen AI chip, or anything with a decent discrete GPU — will run Llama 4 at speeds that beat round-tripping to a cloud API for everything short of frontier-scale reasoning.

This is the complete setup guide. It covers installation on Mac, Linux, and Windows, how to choose a model size, how to read quantization labels without the folklore, performance tuning, and the errors you will actually hit. When you are done, you will have a private assistant that runs on your laptop, never talks to the internet unless you tell it to, and costs nothing per token.

Why run Llama locally at all?

Four reasons, in the order that actually matters for working professionals:

Privacy. Nothing you type leaves your machine. No vendor logs, no training data capture, no compliance surface area.
Cost. After the hardware is paid for, inference is electricity. A heavy daily user burns through $100-$300/month on hosted APIs. Local is zero marginal cost.
Latency. For short prompts on a local 8B-13B model, first-token latency beats any cloud provider.
Control. No deprecation, no rate limits, no unexpected behavior changes, no pricing changes.

The tradeoff is that very long-horizon agentic work and the absolute frontier of reasoning still favor closed models. For everything else — chat, summarization, coding help, RAG, document Q&A — local wins the practical argument.

What you need

A laptop with at least 16GB of RAM (32GB recommended, 64GB+ if you want 70B models).
About 30GB of free disk space per serious model.
Roughly 15 minutes.

That is it. No API keys, no accounts, no cloud billing.

Step 1: Install Ollama

Ollama is the simplest path to running Llama locally. It is one binary, one command to pull a model, and an OpenAI-compatible API on port 11434 for anything that needs to talk to it.

macOS

```bash

Homebrew is the cleanest install path

brew install ollama

Or, for a GUI-bundled version, download from https://ollama.com

```

Start the service:

```bash

brew services start ollama

Or run in foreground: ollama serve

```

Linux

```bash

curl -fsSL https://ollama.com/install.sh | sh

```

The install script sets up a systemd service. Check it is running:

```bash

systemctl status ollama

```

Windows

Download the installer from ollama.com and run it. Ollama runs as a background service and is accessible from PowerShell, WSL, or any app that can hit localhost.

Verify the install on any platform:

```bash

ollama --version

```

You should see something like ollama version 0.6.x or newer.

Step 2: Pick the right model size

Ollama ships with every major open-weight model. Pulling one is a single command. The hard part is picking which one.

| -------------------- | ------------ | ------------- | -------------- | ------------------------------- |

The short version:

16GB RAM: run llama4:8b comfortably. llama3.2:3b for speed.
32GB RAM: run llama4:13b comfortably, or squeeze llama4:70b with heavy quantization.
64GB+ RAM: run llama4:70b at Q4_K_M without breaking a sweat. This is the sweet spot.
96GB+ unified memory (Mac Studio, Framework Desktop): run llama4:70b at Q8_0 or go bigger.

Step 3: Pull and run a model

Pulling a model downloads the weights to your disk. Running it starts an interactive session.

```bash

Pull the model (one-time download)

ollama pull llama4:8b

Run it

ollama run llama4:8b

```

You are now in an interactive REPL talking to a 8B parameter Llama model on your own hardware. Nothing is being sent anywhere.

Try something:

```

>>> Summarize the CAP theorem in 3 sentences.

```

Exit with /bye or Ctrl-D.

Step 4: Quantization explained (without the folklore)

Every model on Ollama has a quantization tag. The default is usually Q4_K_M. Here is what the labels actually mean.

| -------- | ----- | ------------------- | ---------------------------------------- |

| Q2_K | ~2 | Severe | Only if you have no other choice |

| Q3_K_M | ~3 | Noticeable | Squeezing big models onto small hardware |

| Q4_K_M | ~4.5 | Minimal (1-3%) | Default. Use this. |

| Q5_K_M | ~5.5 | Barely measurable | If you have the VRAM headroom |

| Q6_K | ~6.5 | Essentially none | Overkill for most uses |

| Q8_0 | 8 | Lossless in practice| Reference quality on beefy hardware |

| F16/BF16 | 16 | None | Research / fine-tuning |

Q4_K_M is the default for a reason. It recovers 97-99% of full-precision quality while using a quarter of the memory. Going to Q8 doubles your memory use for maybe 1-2% more quality on general tasks.

The one exception: very small models (3B and under) are more sensitive to quantization. On a 3B model, go to Q6 or Q8 if you have the RAM. On a 70B model, Q4_K_M is all you need.

To pull a specific quantization:

```bash

ollama pull llama4:70b-q4_K_M

ollama pull llama4:70b-q8_0

```

Step 5: Performance tuning

Out of the box, Ollama is conservative. These settings squeeze 20-40% more tokens/second out of the same hardware.

Set context length deliberately

The default context window is often 2048 or 4096 tokens, well below what the model actually supports. Llama 4 handles 128K. But bigger context uses more memory and slows inference.

```bash

Set when you run the model

ollama run llama4:70b --ctx 8192

```

Or in a Modelfile:

```

FROM llama4:70b

PARAMETER num_ctx 8192

```

Rule of thumb: 8K context is enough for 95% of chat and coding. Go to 32K for document Q&A. Go to 128K only when you really need it.

Keep the model loaded in memory

Ollama unloads models after a default idle timeout. Reloading a 70B model costs 30+ seconds. Raise the timeout:

```bash

macOS/Linux

launchctl setenv OLLAMA_KEEP_ALIVE "24h"

Or in the shell session:

OLLAMA_KEEP_ALIVE=24h ollama serve

```

Use Metal / CUDA / ROCm acceleration

On Mac, Ollama uses Metal automatically. On Linux/Windows with an NVIDIA GPU, install the CUDA drivers and Ollama will detect them on restart. AMD GPUs use ROCm (Linux) and support is solid for RDNA 3 and newer.

Check that acceleration is active:

```bash

ollama ps

```

You should see the model with 100% GPU or similar. If it shows CPU, something is misconfigured.

Parallel requests

If you are running an agent setup that hits the model from multiple workers, raise the parallel request limit:

```bash

OLLAMA_NUM_PARALLEL=4 ollama serve

```

This improves throughput but eats more RAM.

Step 6: Talk to your model from code

Ollama exposes an OpenAI-compatible API on http://localhost:11434. Any library that supports OpenAI works.

```typescript

import OpenAI from "openai";

const client = new OpenAI({

baseURL: "http://localhost:11434/v1",

apiKey: "ollama", // required but unused

});

const response = await client.chat.completions.create({

model: "llama4:70b",

messages: [

{ role: "system", content: "You are a concise technical assistant." },

{ role: "user", content: "Explain eventual consistency." },

});

console.log(response.choices[0].message.content);

```

Python is identical with the openai package. Curl works too:

```bash

curl http://localhost:11434/api/chat -d '{

"model": "llama4:8b",

"messages": [{"role": "user", "content": "Hello"}]

```

Common errors and how to fix them

Error: model requires more memory than available

You picked a model too big for your RAM. Drop to a smaller model or a lower quantization. On Mac, close Chrome — it is almost certainly the culprit.

Error: CUDA out of memory

Same problem on NVIDIA hardware. Try num_ctx lower, or a smaller quantization, or smaller model.

Tokens per second drops off a cliff after 1000 tokens

You have exceeded VRAM and the model is paging through system memory. Lower context, lower quantization, or buy more RAM.

Model gives stupid answers on long inputs

Context window is set too low and earlier tokens are being truncated. Raise num_ctx.

connection refused on localhost:11434

The Ollama service is not running. ollama serve in a terminal, or start the service with brew services start ollama / systemctl start ollama.

Model takes forever to load every prompt

OLLAMA_KEEP_ALIVE is too short. Set it to 24h.

Output has Chinese characters mixed in (Qwen especially)

Add a system prompt enforcing English: "You must respond only in English."

Local vs cloud: the honest comparison

| Factor | Local (Ollama, Llama 4 70B) | Cloud API (frontier model) |

| --------------- | -------------------------------- | --------------------------------- |

| Privacy | Total — nothing leaves machine | Vendor logs, potential training |

| First-token lat | 50-200ms | 300-1500ms |

| Throughput | 10-60 tok/s | 50-200 tok/s |

| Monthly cost | $0 (after hardware) | $20-$500+ for heavy use |

| Quality ceiling | ~95% of frontier on most tasks | 100% frontier |

| Max context | Limited by RAM | 200K-1M tokens |

| Offline | Yes | No |

| Rate limits | None | Yes |

| Compliance | Full control | Vendor-dependent |

For a privacy-conscious engineer, the only categories where cloud still wins are raw throughput on a single request and access to the absolute largest context windows. Everything else — cost, privacy, control, latency — favors local.

A practical daily workflow

Here is what a realistic local-first setup looks like:

Laptop with 64GB RAM running Ollama + llama4:70b-q4_K_M as the daily driver.
llama3.2:3b kept loaded for instant autocomplete-style tasks.
Continue.dev plugged into VS Code, pointed at the local Ollama endpoint. (See the private AI coding tools guide.)
Open WebUI as a ChatGPT-style frontend on localhost:3000. (See the Open WebUI setup guide.)
Cloud models reserved for: frontier-only reasoning tasks, generating images, anything explicitly non-sensitive.

This is the setup I run. Sensitive work — client data, internal code, personal notes, legal and financial documents — touches the local stack only. Everything else is free to go to whichever cloud vendor is cheapest that week.

Lock down the rest of your stack

Running Llama locally closes the AI part of your privacy surface. It does not close your email, your calendar, your browser history, your cloud backups, or your VPN-less home network. A local LLM without the rest of the stack is a strong door on a house with open windows.

Proton is the privacy stack I pair with a local LLM setup: end-to-end encrypted email, calendar, VPN, password manager, and cloud drive, all zero-knowledge and under Swiss jurisdiction. It is the complement to running your own models — the non-AI parts of your digital life, handled with the same discipline.

Get Proton Unlimited →