Setting Up a Local AI Assistant: The No-Jargon Guide

You want an AI assistant that works like ChatGPT but runs entirely on your computer. No cloud servers. No subscriptions. No data leaving your machine. Maybe you are privacy-conscious. Maybe you want to use AI with sensitive documents. Maybe you just want to stop paying $20/month for a chat subscription.

Whatever the reason, this guide walks you through setting up a fully functional local AI assistant with a chat interface — from scratch, in under 30 minutes. No terminal expertise required. No jargon that assumes you already know what you are doing.

What You Are Building

By the end of this guide, you will have:

1. Ollama — a tool that downloads and runs AI models on your computer
2. Open WebUI — a browser-based chat interface that looks and feels like ChatGPT
3. A working AI assistant — that you can chat with, ask questions, analyze documents, and use for writing — with zero data leaving your machine

The entire stack is free and open source.

What You Need

Hardware minimums:

• 8GB RAM — runs smaller models (3B–7B parameters). Good for basic Q&A, writing, and summarization.
• 16GB RAM — runs mid-size models (7B–14B). Noticeably smarter. Handles coding, analysis, and longer conversations well.
• 32GB+ RAM — runs large models (30B–70B). Approaching GPT-4 quality for many tasks.

Operating system: macOS, Windows, or Linux. All three are supported.

Disk space: 4–20GB per model, depending on size. A typical setup with 2–3 models needs 15–30GB of free space.

Apple Silicon note: If you have a Mac with an M1, M2, M3, or M4 chip, you are in an excellent position. Apple Silicon shares memory between CPU and GPU, making it unusually efficient for running AI models. A MacBook Air M2 with 16GB RAM runs local AI faster than many desktop PCs.

Step 1: Install Ollama (5 Minutes)

Ollama is the engine that downloads, manages, and runs AI models on your computer. Think of it as the foundation everything else sits on.

macOS:

Visit ollama.com and download the Mac app. Open the downloaded file and drag it to your Applications folder. Launch it once — Ollama runs in the background as a menu bar item.

Alternatively, if you are comfortable with Terminal:

```

curl -fsSL https://ollama.com/install.sh | sh

```

Windows:

Visit ollama.com and download the Windows installer. Run the installer. It takes about 60 seconds. Ollama runs as a system tray application.

Linux:

Open a terminal and run:

```

curl -fsSL https://ollama.com/install.sh | sh

```

Verify it works: Open your terminal (Terminal on Mac, Command Prompt or PowerShell on Windows) and type:

```

ollama --version

```

If it prints a version number, you are good.

Step 2: Download Your First Model (5 Minutes)

Now you need an AI model — the actual "brain" of your assistant. Open your terminal and run:

```

ollama pull llama3.2

```

This downloads Meta's Llama 3.2 model (the 8B parameter version). It is approximately 4.7GB and takes 2–10 minutes depending on your internet speed. This is a general-purpose model that handles conversation, writing, coding, and analysis well.

Quick test: After the download finishes, type:

```

ollama run llama3.2

```

You are now chatting with a local AI model in your terminal. Type a question, press Enter, and watch it respond. Everything is happening on your computer — no internet required after the initial download.

Type /bye to exit.

Other models worth downloading:

```

ollama pull mistral # 7B, fast and capable

ollama pull codellama # Optimized for coding tasks

ollama pull llama3.2:3b # Smaller, faster, for quick tasks

ollama pull qwen2.5:14b # Larger, smarter (needs 16GB+ RAM)

```

Step 3: Install Open WebUI (10 Minutes)

Chatting in the terminal works, but it is not a great experience for daily use. Open WebUI gives you a browser-based chat interface that looks and feels like ChatGPT — with conversation history, multiple models, file uploads, and a clean design.

The easiest method (Docker):

If you have Docker installed (or are willing to install it), Open WebUI is one command:

```

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

```

Then open your browser and go to http://localhost:3000. Create an account (stored locally — not sent anywhere), and you are done.

If you do not have Docker:

Install Docker Desktop first:

• macOS: Download from docker.com, drag to Applications, launch it
• Windows: Download from docker.com, run the installer, restart if prompted
• Linux: Follow the Docker install guide for your distribution

Then run the Docker command above.

Alternative (without Docker):

If you prefer not to use Docker, Open WebUI can be installed via pip (Python package manager):

```

pip install open-webui

open-webui serve

```

This requires Python 3.11+ installed on your system.

Step 4: Connect and Configure (5 Minutes)

When you open Open WebUI in your browser, it automatically detects Ollama running on your machine and shows all the models you have downloaded. You can:

• Select a model from the dropdown at the top of the chat
• Start a conversation by typing in the message box
• Upload files (PDFs, text files, code) for the model to analyze
• Switch between models mid-conversation to compare responses
• Search conversation history to find previous chats
• Create custom system prompts to tailor the AI's behavior

Recommended settings to configure:

1. Go to Settings and set your default model to whichever you prefer
2. Enable "Stream responses" for a more natural chat feel
3. Set a system prompt if you want the AI to behave in a specific way (e.g., "You are a helpful assistant that gives concise, direct answers")

What You Can Do With Your Local AI

Now that everything is running, here is what your local assistant handles well:

Writing and editing: Drafting emails, blog posts, reports, and documentation. Rewriting text for tone or clarity. Summarizing long documents.

Coding assistance: Writing code, debugging, explaining error messages, generating boilerplate. CodeLlama is specifically optimized for this.

Analysis: Summarizing PDFs, extracting key points from documents, analyzing data patterns. Upload files directly into Open WebUI.

Brainstorming: Generating ideas, outlining projects, thinking through problems. No risk of proprietary ideas leaking to a cloud provider.

Learning: Explaining concepts, answering questions, acting as a study partner. Run it offline on a plane or in a location without internet.

What Local AI Does Not Do Well (Yet)

Be realistic about limitations:

It is not GPT-4 or Claude. The best local models (70B parameters) approach but do not match the reasoning quality of frontier cloud models. For most daily tasks this does not matter, but for complex multi-step reasoning or creative writing, cloud models still have an edge.

Image generation and multimodal tasks are limited. Some local models support vision (analyzing images), but generating images requires separate tools (Stable Diffusion).

Real-time information is not available. Local models have a knowledge cutoff from their training data. They cannot browse the internet or access current information.

Very long documents may exceed the model's context window. Most local models handle 4,000–8,000 tokens of context, with some supporting 32,000–128,000. For very long documents, you may need to break them into sections.

Key Takeaways

• A fully functional local AI assistant takes under 30 minutes to set up with Ollama + Open WebUI
• 16GB RAM is the sweet spot for capable local AI — Apple Silicon Macs perform exceptionally well
• Your data never leaves your computer — no cloud, no training, no third-party access
• Local models handle 70–80% of daily AI tasks (writing, coding, analysis, Q&A)
• The entire stack is free and open source

Related articles:

• How to Run AI on Your Laptop With No Data Leaving Your Device
• Ollama Setup Guide for Complete Beginners
• ChatGPT vs Claude vs Gemini: Which One Actually Respects Your Privacy?

Affiliate Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no extra cost to you. We only recommend products we genuinely believe in. This helps support our work and allows us to continue providing free content.