If you've ever pasted a contract, a client brief, or a confidential internal doc into ChatGPT to get a quick summary — you've already handed that data to OpenAI's training pipeline. That's not hypothetical. It's in the terms of service.

The good news: you don't need to make that trade. You can build a fully local RAG (Retrieval-Augmented Generation) system that lets you ask natural-language questions of your own documents — PDFs, Markdown files, Word docs, meeting notes — with a model running entirely on your machine. No API calls. No data retention. No cloud.

This guide walks you through the full setup, from choosing a model to querying your first document, in plain language that assumes you're comfortable in a terminal but don't have an ML background.

Last updated: 2026-03-23

What Is RAG and Why Does It Matter for Privacy?

RAG stands for Retrieval-Augmented Generation. The name is academic jargon for a simple idea: instead of asking an AI to answer from memory, you give it relevant chunks of your actual documents at query time, then ask it to answer based on those chunks.

Here's why that matters for privacy specifically:

A base LLM — even a good local one — doesn't know what's in your files. RAG is the bridge between "I have a 200-page vendor contract" and "tell me every clause that references termination for convenience." Without RAG, you'd have to paste the whole document into the context window and hope it fits. With RAG, only the relevant passages get pulled, processed locally, and fed to the model.

The entire pipeline runs on your hardware. Your documents are chunked, embedded, and stored in a local vector database. The LLM runs locally via Ollama or LM Studio. Nothing touches the internet.

This setup is particularly valuable for:

Lawyers and paralegals querying case files
Engineers searching internal runbooks and architecture docs
Researchers working with embargoed papers or sensitive datasets
Anyone who handles client data under NDA or compliance obligations

What You'll Need

Before starting, make sure you have:

A machine with at least 16GB RAM (32GB recommended for larger models). Apple Silicon Macs, recent AMD laptops, and machines with NVIDIA GPUs all work well.
~20–30GB of free disk space for models and the vector store
Ollama installed — the easiest way to run local LLMs (ollama.com)
Python 3.10+ — for the RAG orchestration layer
Node.js 18+ — optional, if you want a web UI

If you haven't run a local LLM before, start with the Ollama beginner setup guide first, then come back here.

Step 1: Pull a Model Suited for Document Q&A

Not all local models are equally good at document comprehension. For RAG workloads, you want a model with a large context window and strong instruction-following. As of early 2026, these are solid choices:

```bash

Good balance of speed and quality (7B, runs on 16GB RAM)

ollama pull mistral

Better reasoning, slower (8B)

ollama pull llama3.1

Best document comprehension if you have 32GB+ RAM

ollama pull mixtral

```

For most users on 16GB machines, mistral or llama3.1:8b hits the right balance. Test with a simple prompt first:

```bash

ollama run mistral "Summarize what RAG stands for in one sentence."

```

If that responds in under 10 seconds, you're in good shape.

Step 2: Set Up Your Document Storage — Encrypted at Rest

Before indexing your documents, you should store them somewhere encrypted. If your laptop is stolen or your home directory is compromised, you don't want sensitive files sitting in plaintext.

Two strong options depending on your workflow:

If your documents live on-device only: Use Tresorit to maintain an encrypted, zero-knowledge backup of your source documents. Tresorit uses end-to-end encryption that even Tresorit's own servers can't read — meaning if you ever need to restore your doc library on a new machine, it arrives encrypted and intact with no cloud AI having ever touched it.

Affiliate Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no extra cost to you. We only recommend products we genuinely believe in. This helps support our work and allows us to continue providing free content.

For this guide, assume your documents are in a local folder: ~/private-docs/. Organize by subdirectory if you have multiple corpora (e.g., ~/private-docs/contracts/, ~/private-docs/research/).

Step 3: Install the RAG Stack

We'll use LlamaIndex — the most mature Python library for local RAG pipelines — along with ChromaDB as the local vector store. Both are open source, run entirely offline, and require no accounts.

```bash

pip install llama-index llama-index-llms-ollama llama-index-embeddings-ollama chromadb

```

You'll also need an embedding model. Embeddings convert text chunks into vectors that the similarity search runs against. Pull a dedicated embedding model via Ollama:

```bash

ollama pull nomic-embed-text

```

This is a small, fast model (~274MB) purpose-built for embeddings. It runs much faster than using your main LLM for embeddings, and the quality is excellent for English text.

Step 4: Index Your Documents

Create a file called index_docs.py in your working directory:

```python

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, StorageContext

from llama_index.llms.ollama import Ollama

from llama_index.embeddings.ollama import OllamaEmbedding

from llama_index.core import Settings

import chromadb

from llama_index.vector_stores.chroma import ChromaVectorStore

Configure local models — no API keys, no internet

Settings.llm = Ollama(model="mistral", request_timeout=120.0)

Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")

Point at your document folder

documents = SimpleDirectoryReader("~/private-docs/", recursive=True).load_data()

Set up local ChromaDB vector store

chroma_client = chromadb.PersistentClient(path="./chroma_db")

chroma_collection = chroma_client.get_or_create_collection("private_docs")

vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

storage_context = StorageContext.from_defaults(vector_store=vector_store)

Index and persist

index = VectorStoreIndex.from_documents(

documents,

storage_context=storage_context,

show_progress=True

)

print("Indexing complete. Vector store saved to ./chroma_db")

```

Run it:

```bash

python index_docs.py

```

Indexing time depends on document volume and your hardware. A few hundred pages typically takes 2–5 minutes on a modern laptop. The output is a persistent ChromaDB folder — you only need to re-index when you add new documents.

Step 5: Query Your Documents

Create query_docs.py:

```python

from llama_index.core import VectorStoreIndex, StorageContext

from llama_index.llms.ollama import Ollama

from llama_index.embeddings.ollama import OllamaEmbedding

from llama_index.core import Settings

import chromadb

from llama_index.vector_stores.chroma import ChromaVectorStore

Settings.llm = Ollama(model="mistral", request_timeout=120.0)

Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")

Load existing vector store

chroma_client = chromadb.PersistentClient(path="./chroma_db")

chroma_collection = chroma_client.get_or_create_collection("private_docs")

vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex.from_vector_store(

vector_store, storage_context=storage_context

)

query_engine = index.as_query_engine(similarity_top_k=5)

Interactive query loop

print("Private document assistant ready. Type 'quit' to exit.\n")

while True:

question = input("Your question: ").strip()

if question.lower() in ("quit", "exit"):

break

response = query_engine.query(question)

print(f"\nAnswer: {response}\n")

print(f"Sources: {[n.metadata.get('file_name', 'unknown') for n in response.source_nodes]}\n")

```

Run it:

```bash

python query_docs.py

```

You now have a local, private document assistant. Ask it anything — "What are the payment terms in the vendor contract?", "Summarize all action items from last month's meeting notes", "Which documents mention the word indemnification?"

Step 6: Add a Web UI (Optional but Recommended)

The terminal loop works, but if you want a chat interface, Open WebUI is the cleanest option. It wraps Ollama with a ChatGPT-style UI and supports RAG document uploads natively.

```bash

docker run -d -p 3000:8080 \

-v open-webui:/app/backend/data \

-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \

--name open-webui \

ghcr.io/open-webui/open-webui:main

```

Visit http://localhost:3000 — you'll get a full chat UI where you can drag-and-drop documents for per-session context, switch between local models, and maintain conversation history, all stored locally.

When Local Isn't Practical: A Privacy-Respecting Cloud Option

There will be times — traveling with a lightweight machine, querying from a phone, working on a borrowed computer — when running models locally isn't feasible. For those moments, Perplexity Pro is worth knowing about.

Perplexity's privacy policy is substantially more restrained than OpenAI's or Google's when it comes to using queries for model training. The Pro tier also gives you access to multiple model backends and supports file uploads. It's not zero-knowledge and it's not local — so don't use it for your most sensitive documents — but as a cloud fallback for non-sensitive research queries, it's a reasonable choice over the default alternatives.