How to Run a Private AI Assistant on Android (No Cloud, No Data Collection)

You can run a genuinely useful AI assistant on a modern Android phone with the airplane mode switch on. No API calls, no server logs, no "your conversation may be reviewed to improve our models" disclaimer. The model runs on your phone's own chip, and nothing you type leaves the device unless you tell it to.

This wasn't practical two years ago — phone hardware couldn't hold a model good enough to be worth using. That changed with small, efficient models like Gemma 2 (2B), Phi-3.5-mini, and Llama 3.2 (1B/3B), paired with Android apps that quantize and run them entirely on-device via llama.cpp or MLC-LLM. If your phone has 6GB+ of RAM and shipped in the last three years, you can do this right now.

Here's exactly how.

Why Your Phone's Built-In AI Isn't Private

Gemini on Android, Samsung's Galaxy AI, and most assistant apps route your requests to a cloud server by default — even for tasks that don't need it, like rewriting a text message or summarizing a screenshot. Google's own privacy documentation confirms Gemini conversations can be reviewed by human raters and retained for product improvement unless you manually change your activity settings, and some on-device features still phone home for anything beyond basic autocomplete.

That's a real problem if you're drafting anything sensitive on your phone: legal correspondence, medical questions, financial details, or notes about a competitor's product. A cloud assistant doesn't know the difference between "help me caption a vacation photo" and "help me word this termination letter" — it processes and potentially logs both the same way.

An on-device model has no such distinction to make. It can't leak what it never transmits.

What You Actually Need

• A phone with 6GB+ RAM (8GB+ is comfortable). Most flagship and mid-range Android phones from 2023 onward qualify.
• 2-4GB of free storage per model you install — small models are 1.5-2.5GB, quantized.
• No internet connection required after the initial model download.
• Realistic expectations: a 2-4B parameter model on a phone is not GPT-4. It's very good at drafting, rewriting, summarizing short text, brainstorming, and answering general-knowledge questions. It will struggle with complex multi-step reasoning or long-context tasks — for those, you'll want a hybrid workflow (more on that below).

Step 1: Pick an App

Three apps cover most use cases well:

PocketPal AI (free, open source) is the easiest starting point. Clean interface, built-in model browser, no account required. Good defaults for people who don't want to think about quantization formats.

ChatterUI (free, open source) is closer to a full chat client — it supports character/persona presets, longer context windows, and more granular control over sampling parameters (temperature, top-p, etc.) if you want to tune output quality.

Google AI Edge Gallery (free, from Google itself) is worth knowing about specifically because it runs Gemma models fully on-device and is transparent about doing so — a rare case of a first-party Google app that doesn't default to cloud processing. Good option if you want something backed by an official app store listing.

All three are available on the Google Play Store or as direct APKs from their GitHub releases (F-Droid also carries PocketPal). Avoid app-store listings that promise "AI chat" with vague privacy policies — check that the app explicitly states inference happens on-device before installing.

Step 2: Choose Your Model

Model choice matters more than app choice for how useful this actually is. Three solid picks as of mid-2026:

| Model | Size (quantized) | Best for | RAM needed |

|-------|-------------------|----------|------------|

| Llama 3.2 1B Instruct | ~1.3GB | Fast replies, low-RAM phones | 4GB+ |

| Gemma 2 2B Instruct | ~1.6GB | Best balance of speed and quality | 6GB+ |

| Phi-3.5-mini Instruct (3.8B) | ~2.3GB | Strongest reasoning of the three | 8GB+ |

Start with Gemma 2 2B — it's the best default for general use. If replies feel too shallow for what you need, step up to Phi-3.5-mini. If your phone is older or you want snappier responses for quick tasks, drop to Llama 3.2 1B.

Download the model directly inside the app (PocketPal and ChatterUI both have a built-in downloader that pulls GGUF-format files from Hugging Face). This is the one step that needs an internet connection — everything after is offline.

Step 3: Verify It's Actually Offline

Don't just trust the marketing copy. Confirm it yourself:

1. Fully download your chosen model.
2. Turn on airplane mode.
3. Open the app and send a few prompts.

If you get coherent responses with zero connectivity, you've confirmed the model is running locally and isn't silently falling back to a cloud API. This takes 30 seconds and removes any doubt.

Step 4: Lock Down the App's Permissions

Once it's working, check Android Settings → Apps → [your app] → Permissions. A pure on-device inference app has no legitimate reason to request contacts, location, or background network access. If it asks for network access at all beyond the initial model download, that's worth understanding before you grant it — some apps include optional cloud fallback features that are opt-in but pre-toggled. Turn those off explicitly in the app's own settings.

The Hybrid Workflow: When You Need More Than On-Device Can Give

A 2-4B model on your phone won't out-reason a frontier cloud model, and it has no access to current information — it can't check today's news or look up a fact from last week. For that gap, don't default back to whatever cloud assistant came preinstalled on your phone. Use a tool that's explicit about what it does with your data.

Perplexity is a reasonable middle ground for the research half of this workflow: it's built around citing live sources rather than silently training on your queries, and its Focus modes let you scope a search instead of broadcasting your full question history into a general-purpose profile. Keep the sensitive drafting and rewriting on-device, and reserve Perplexity for the "what's the current status of X" queries your local model can't answer.

Affiliate Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no extra cost to you. We only recommend products we genuinely believe in. This helps support our work and allows us to continue providing free content.

Either one beats the default: an unencrypted folder synced to a Google account that's already indexing everything else you do.

Realistic Limits — Set These Expectations Now

• Battery drain is real. Local inference is CPU/NPU-intensive. Expect noticeably faster battery drain during active use, though idle drain is unaffected since nothing runs in the background.
• Long conversations slow down. Context windows on phone-class models are smaller than their desktop counterparts. Past a few thousand tokens, expect a noticeable speed drop or the app truncating earlier messages.
• It won't browse the web, run code, or use tools unless the specific app you chose has built-in function-calling support — most don't yet. Treat it as a text-in, text-out assistant.
• Model updates are manual. Unlike a cloud assistant, nobody pushes silent capability upgrades to your on-device model. Check back every few months for newer small-model releases — the space is moving fast.

Frequently Asked Questions

Does this work on iPhone too?

Apple's more locked-down runtime makes third-party on-device LLM apps harder to ship and slower to update, and Apple Intelligence itself still routes complex requests to Apple's cloud via Private Cloud Compute. iOS options exist (MLC Chat, some PocketPal builds) but the Android ecosystem is currently ahead on app choice and model support.

Can I use this instead of ChatGPT entirely?

For drafting, rewriting, brainstorming, and general Q&A, yes. For anything requiring current information, web browsing, code execution, or very long documents, no — use the hybrid workflow above instead of forcing a small on-device model to do a job it isn't sized for.

Will a local model drain my battery even when I'm not using it?

No. Inference only runs while you're actively generating a response. There's no background process polling a server or running inference when the app is idle, which is part of what makes this more battery-friendly overall than you'd expect — the cost is concentrated entirely in active use.

Is a quantized model less accurate than the full-size version?

Slightly, but for the 4-bit and 5-bit quantization levels these apps use by default, the quality loss is small enough that most people can't tell the difference in everyday use. It's a reasonable trade for a model that fits on a phone.

What happens to my data if I uninstall the app?

Everything — the model file and any local chat history — is deleted with the app, since none of it was ever synced anywhere. If you want to keep your conversation history, export it first and back it up using the encrypted storage options above.

The Bottom Line

Running AI on your Android phone without sending a single token to a server isn't a compromise anymore — it's a legitimate daily-driver setup for drafting, summarizing, and brainstorming, with a clear, explicit fallback for the research tasks it can't handle. The setup takes about 15 minutes. The privacy difference is permanent.

---

Want the model and app picks kept current as new releases ship?

Last updated: 2026-07-07