Last updated: 2026-06-05

The Short Version: Your RAG Pipeline Is Probably Phoning Home

If you stood up a RAG pipeline this week using LangChain or LlamaIndex tutorials, your documents, user queries, and chain execution traces are very likely leaving your infrastructure right now. Not because of a breach. Because that is the default.

Both frameworks are optimized for developer velocity, which means they default to cloud APIs, managed observability platforms, and hosted vector stores. Privacy requires configuration, and configuration requires knowing where to look. Most tutorials never tell you.

This audit covers six concrete leakage vectors, where each one lives in the code, and a drop-in replacement for each.

Leakage Vector 1: LangSmith Tracing (LangChain)

This is the most aggressive default and the one most developers miss entirely.

When the environment variable LANGCHAIN_API_KEY is present, LangChain v0.2+ automatically enables LangSmith tracing. Every chain invocation, every LLM call, every tool call, and every prompt template rendered — including the full text of those prompts — is shipped to LangSmith's cloud at api.smith.langchain.com. This happens silently in the background.

If you are building a RAG system over sensitive documents (legal, medical, financial), and you set LANGCHAIN_API_KEY to test something once, you are now continuously streaming query+context pairs to a third-party service.

The fix:

```python

Explicitly disable tracing regardless of env vars

import os

os.environ["LANGCHAIN_TRACING_V2"] = "false"

os.environ["LANGCHAIN_API_KEY"] = "" # belt-and-suspenders

Or use the SDK-level toggle

from langchain_core.callbacks import CallbackManager

Pass an empty callback manager to individual chains

chain = your_chain.with_config({"callbacks": []})

```

For teams that want observability without cloud egress, self-host LangSmith's open-source langsmith-server Docker image or use Langfuse — a privacy-respecting LangSmith alternative you can run on your own hardware.

Affiliate Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no extra cost to you. We only recommend products we genuinely believe in. This helps support our work and allows us to continue providing free content.

Leakage Vector 2: Embedding API Calls Over Your Source Documents

Every document you index in a RAG pipeline must be converted to a vector embedding. The default embedding model in both LangChain and LlamaIndex is OpenAI's text-embedding-ada-002 (or text-embedding-3-small in newer defaults).

That means every chunk of every document you index is sent to OpenAI's API. Your proprietary product docs, customer emails, internal knowledge base articles — all of it hits OpenAI's servers as plain text during the embedding step. Even if your final LLM is local, the indexing step bleeds data.

The same applies to Cohere Embeddings, Voyage AI, and any other hosted embedding provider used as a default.

The fix — local embeddings with sentence-transformers:

```python

LangChain

from langchain_community.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(

model_name="nomic-ai/nomic-embed-text-v1.5",

model_kwargs={"device": "cpu"}, # or "cuda" / "mps"

encode_kwargs={"normalize_embeddings": True},

)

```

```python

LlamaIndex

from llama_index.embeddings.huggingface import HuggingFaceEmbedding

from llama_index.core import Settings

Settings.embed_model = HuggingFaceEmbedding(

model_name="nomic-ai/nomic-embed-text-v1.5"

)

```

Strong local embedding model options ranked by privacy-performance balance:

nomic-ai/nomic-embed-text-v1.5 — 768 dims, excellent retrieval quality, Apache 2.0 license
BAAI/bge-m3 — multilingual, good for mixed-language corpora
sentence-transformers/all-MiniLM-L6-v2 — lighter, faster, lower quality ceiling

If you want Ollama-managed embeddings (simpler dependency stack):

```python

from langchain_ollama import OllamaEmbeddings

embeddings = OllamaEmbeddings(model="nomic-embed-text")

```

Leakage Vector 3: Cloud Vector Database Defaults

The retrieval half of RAG depends on a vector store. The problem is that popular managed vector databases — Pinecone, Weaviate Cloud, Qdrant Cloud — store your vectors (and often the associated document chunks they're derived from) on their infrastructure.

This is not a theoretical risk. When you push embeddings to Pinecone, the metadata payload typically includes the raw text chunk. You are storing your indexed content on Pinecone's servers. Their data processing agreements and retention policies govern what happens to it.

LlamaIndex's quickstart docs historically pointed at Pinecone as the example store. LangChain's cookbook examples use Pinecone and Chroma Cloud. Neither flags the privacy implications inline.

The fix — local vector stores:

```python

Chroma local (in-process or persistent)

from langchain_chroma import Chroma

vectorstore = Chroma(

collection_name="my_docs",

embedding_function=embeddings,

persist_directory="./chroma_db", # local disk, no cloud

)

```

```python

Qdrant local via Docker

docker run -p 6333:6333 qdrant/qdrant

from langchain_qdrant import QdrantVectorStore

from qdrant_client import QdrantClient

client = QdrantClient(host="localhost", port=6333)

vectorstore = QdrantVectorStore(

client=client,

collection_name="my_docs",

embedding=embeddings,

)

```

```python

FAISS — fully in-memory, no server required

from langchain_community.vectorstores import FAISS

vectorstore = FAISS.from_documents(docs, embeddings)

vectorstore.save_local("./faiss_index") # serializes to disk

```

For LlamaIndex:

```python

from llama_index.vector_stores.chroma import ChromaVectorStore

from llama_index.core import StorageContext

import chromadb

chroma_client = chromadb.PersistentClient(path="./chroma_db")

chroma_collection = chroma_client.get_or_create_collection("my_docs")

vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

storage_context = StorageContext.from_defaults(vector_store=vector_store)

```

Leakage Vector 4: LlamaIndex Framework Telemetry

LlamaIndex has historically collected anonymous usage telemetry to track which components developers use. This is disabled by an environment variable, but it is not off by default in older versions, and the setting location is not obvious.

```python

Disable before any llama_index imports

import os

os.environ["LLAMA_INDEX_DISABLE_TELEMETRY"] = "1"

Or in code after import

from llama_index.core import Settings

In some versions:

from llama_index.core.callbacks import CallbackManager

Settings.callback_manager = CallbackManager([])

```

Check your current configuration by monitoring outbound network traffic with sudo lsof -i -n | grep python or a proxy tool like mitmproxy during pipeline execution. Any unexpected connections to llamaindex.ai or analytics endpoints should be investigated.

Leakage Vector 5: Callback and Observability Integrations

Both frameworks expose a callback/event system that third-party integrations hook into. The problem: some popular integrations — Weights & Biases, Arize, Helicone, Traceloop — are imported and registered in tutorials without the reader understanding they are activating cloud data pipelines.

A common pattern that silently enables telemetry:

```python

This import alone may register a global callback handler

from traceloop.sdk import Traceloop

Traceloop.init(app_name="my_rag_app") # now streaming traces to Traceloop cloud

```

Audit your callback stack:

```python

LangChain — inspect active callbacks

from langchain_core.callbacks import get_callback_manager

print(get_callback_manager().handlers)

LlamaIndex — inspect registered handlers

from llama_index.core import Settings

print(Settings.callback_manager.handlers)

```

Remove any handlers you did not explicitly add. For LangChain, pass callbacks=[] explicitly to chains, LLMs, and retrievers during instantiation rather than relying on globals.

Leakage Vector 6: The LLM Call Itself

If your pipeline makes it through the above hardening but still routes LLM inference to OpenAI, Anthropic, or another hosted API, every query plus its retrieved context chunks are sent to that provider. This is the most obvious vector but worth stating explicitly.

The fix — local inference via Ollama:

```python

LangChain

from langchain_ollama import ChatOllama

llm = ChatOllama(

model="llama3.2:3b", # or mistral, qwen2.5, gemma3, etc.

temperature=0,

)

```

```python

LlamaIndex

from llama_index.llms.ollama import Ollama

from llama_index.core import Settings

Settings.llm = Ollama(model="llama3.2:3b", request_timeout=120.0)

```

For production deployments requiring GPU inference at scale, consider Ollama on dedicated hardware or a self-hosted vLLM instance behind your own network perimeter.

The Privacy-Hardened Baseline Configuration

Here is a minimal, fully local RAG pipeline that eliminates all six leakage vectors. No data leaves your machine.

```python

import os

Block LangSmith before any imports

os.environ["LANGCHAIN_TRACING_V2"] = "false"

os.environ["LANGCHAIN_API_KEY"] = ""

os.environ["LLAMA_INDEX_DISABLE_TELEMETRY"] = "1"

from langchain_ollama import ChatOllama, OllamaEmbeddings

from langchain_chroma import Chroma

from langchain.chains import RetrievalQA

from langchain.callbacks import CallbackManager

Local LLM — no API calls

llm = ChatOllama(model="llama3.2:3b", temperature=0)

Local embeddings — no API calls

embeddings = OllamaEmbeddings(model="nomic-embed-text")

Local vector store — no cloud

vectorstore = Chroma(

collection_name="private_docs",

embedding_function=embeddings,

persist_directory="./chroma_db",

)

No callback handlers registered

qa_chain = RetrievalQA.from_chain_type(

llm=llm,

retriever=vectorstore.as_retriever(),

callbacks=[], # explicit empty — no globals

)

```

Run python -m mitmproxy in a companion terminal and verify zero outbound HTTPS connections during a query. That is your acceptance test.

Verification Checklist

Before deploying any RAG pipeline handling sensitive data, run through this list:

[ ] LANGCHAIN_TRACING_V2=false confirmed in environment
[ ] LLAMA_INDEX_DISABLE_TELEMETRY=1 confirmed in environment
[ ] Embedding model is local (HuggingFace or Ollama) — no OpenAI/Cohere calls during indexing
[ ] Vector store is local (Chroma persistent, Qdrant local, FAISS) — no Pinecone/Weaviate Cloud
[ ] Callback handler list inspected and contains only explicitly added handlers
[ ] LLM is local (Ollama) or is a provider with a DPA and data retention policy you have reviewed
[ ] Network traffic audited with mitmproxy or equivalent during a test query

The defaults in these frameworks are not adversarial — they exist because cloud services are easier to demo. But easy defaults and private defaults are not the same thing. Audit once, harden at initialization, and your pipeline stays yours.

Stay Ahead of the Next Leak

New integrations ship weekly in both ecosystems. The fastest way to catch a new default sending data somewhere unexpected is to subscribe to dependency changelogs and re-run your network audit after every pip install --upgrade.

Get our RAG Privacy Hardening Checklist — updated whenever a new leakage vector surfaces in either framework:

Get the RAG Privacy Checklist →

No noise. One email when something important changes.