Stop Paying for OpenAI: How to Self-Host an Enterprise-Grade RAG on Your Laptop with Ollama

Retrieval-augmented generation (RAG) has become the default way to point a language model at your own documents. Most tutorials still assume you will send every chunk of your data, and every user question, to a hosted API. That is a fine way to prototype. It is a harder sell when the documents are contracts, incident reports, or source code, and when the monthly bill grows with every query your users make.

The open-weight model ecosystem has changed that calculation. Quantized 7–8B parameter models run comfortably on a modern laptop, open embedding models are good enough for most retrieval workloads, and the tooling to serve them is now a single install. You can build a RAG pipeline where nothing leaves your machine or your network, and you can do it in an afternoon.

This article walks through that stack end to end: the four components, how they fit together, working Python code for ingestion and querying, a small FastAPI service in front of it, and the tuning decisions that separate a demo from something you would let colleagues use. It also covers where local RAG falls short, because "stop paying for OpenAI" is a good default for some workloads and a bad one for others.

The architecture in four pieces

Every RAG system, hosted or local, has the same moving parts. Self-hosting just means you choose an open component for each one.

• The model server (the engine). Ollama downloads, quantizes, and serves open-weight models behind a local HTTP API on port 11434. The "Docker for LLMs" comparison is apt: ollama pull fetches a model, ollama run starts it, and the server manages loading and unloading models from memory.
• The generator model (the brain). An instruction-tuned model in the 7–8B range, such as Llama 3.1 8B, Mistral 7B, or Qwen3 8B. RAG is a forgiving task for small models: the answer is supposed to come from the retrieved context, so you need good instruction following and summarization more than broad world knowledge.
• The embedding model (the glue). A separate, much smaller model that turns text into vectors. nomic-embed-text and mxbai-embed-large are both available through Ollama; sentence-transformers models from Hugging Face are another option if you prefer to embed in-process.
• The vector store (the memory). An embedded database such as Chroma or LanceDB that runs inside your Python process and persists to a local directory. No separate server, no managed service, no network hop.

On top of those sits an orchestrator. That can be LangChain or LlamaIndex, both of which have first-class Ollama integrations. For a pipeline this small I prefer plain Python: the code is short, every step is visible, and there is no framework abstraction to debug when retrieval quality is poor.

Setting up the model server

Install Ollama from the official site (or with Homebrew on macOS), then pull one generator and one embedding model:

ollama pull llama3.1:8b
ollama pull nomic-embed-text

# Sanity check: the server is listening and both models are present
curl http://localhost:11434/api/tags

The default tags are 4-bit quantized. An 8B model at that quantization needs roughly 5 GB of memory, which fits on a 16 GB laptop alongside everything else you have open. On Apple Silicon, Ollama uses the GPU through Metal automatically; on Linux and Windows it uses an NVIDIA or AMD GPU when one is available and falls back to CPU otherwise. CPU-only inference works, but expect noticeably slower generation.

Install the Python dependencies:

pip install ollama chromadb fastapi uvicorn

Ingestion: chunk, embed, store

Ingestion is where most RAG quality is won or lost. The model can only answer from what retrieval hands it, and retrieval can only find what you chunked sensibly.

The script below reads Markdown and text files from a directory, splits them into overlapping chunks, embeds each chunk with nomic-embed-text, and stores the result in a persistent Chroma collection.

# ingest.py
import hashlib
from pathlib import Path

import chromadb
import ollama

EMBED_MODEL = "nomic-embed-text"
CHUNK_SIZE = 1200      # characters
CHUNK_OVERLAP = 200

client = chromadb.PersistentClient(path="./rag_db")
collection = client.get_or_create_collection(
    name="docs",
    metadata={"hnsw:space": "cosine"},
)


def chunk_text(text: str, size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list[str]:
    """Split on paragraphs, then pack paragraphs into chunks of roughly `size` chars."""
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks, current = [], ""
    for para in paragraphs:
        if current and len(current) + len(para) > size:
            chunks.append(current)
            current = current[-overlap:]  # carry some context forward
        current = f"{current}\n\n{para}" if current else para
    if current:
        chunks.append(current)
    return chunks


def embed(texts: list[str], prefix: str) -> list[list[float]]:
    # nomic-embed-text expects a task prefix on every input
    response = ollama.embed(model=EMBED_MODEL, input=[prefix + t for t in texts])
    return response["embeddings"]


def ingest(folder: str) -> None:
    for path in Path(folder).rglob("*"):
        if path.suffix not in {".md", ".txt"}:
            continue
        chunks = chunk_text(path.read_text(encoding="utf-8"))
        if not chunks:
            continue
        ids = [
            hashlib.sha256(f"{path}:{i}:{c}".encode()).hexdigest()
            for i, c in enumerate(chunks)
        ]
        collection.upsert(
            ids=ids,
            documents=chunks,
            embeddings=embed(chunks, prefix="search_document: "),
            metadatas=[{"source": str(path), "chunk": i} for i in range(len(chunks))],
        )
        print(f"Indexed {len(chunks):>4} chunks from {path}")


if __name__ == "__main__":
    ingest("./docs")

A few details in that script matter more than they look:

• Task prefixes. nomic-embed-text is trained with search_document: and search_query: prefixes, as described on its model card. Leaving them off still produces vectors, just worse ones. Other embedding models have their own conventions; read the card for whichever one you choose.
• Deterministic IDs and upsert. Hashing the path, position, and content means re-running ingestion is idempotent. Unchanged chunks overwrite themselves instead of piling up as duplicates.
• Source metadata. Storing the file path with every chunk is what lets you show citations later. In an internal tool, "here is the answer and here is the document it came from" is the difference between something people trust and something they double-check by hand.
• Paragraph-aware chunking. Splitting on blank lines keeps sentences and short sections intact. Fixed-size character splits that cut mid-sentence are a common source of confusing retrieval results.

One rule is non-negotiable: the same embedding model must be used at ingestion time and query time. If you switch models, re-embed the whole corpus into a new collection. Vectors from different models live in different spaces and cannot be compared.

Querying: retrieve, then generate

The query path embeds the question, pulls the closest chunks from Chroma, and asks the generator to answer using only that context.

# rag.py
import chromadb
import ollama

EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL = "llama3.1:8b"
TOP_K = 5

client = chromadb.PersistentClient(path="./rag_db")
collection = client.get_collection("docs")

SYSTEM_PROMPT = """You answer questions using only the provided context.
If the context does not contain the answer, say you don't know.
Cite sources using the [n] markers that precede each context block."""


def retrieve(question: str, k: int = TOP_K) -> list[dict]:
    query_vec = ollama.embed(
        model=EMBED_MODEL, input="search_query: " + question
    )["embeddings"][0]
    result = collection.query(query_embeddings=[query_vec], n_results=k)
    return [
        {"text": doc, "source": meta["source"], "distance": dist}
        for doc, meta, dist in zip(
            result["documents"][0], result["metadatas"][0], result["distances"][0]
        )
    ]


def answer(question: str) -> dict:
    hits = retrieve(question)
    context = "\n\n".join(
        f"[{i + 1}] (source: {h['source']})\n{h['text']}" for i, h in enumerate(hits)
    )
    response = ollama.chat(
        model=CHAT_MODEL,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ],
        options={"temperature": 0.1, "num_ctx": 8192},
    )
    return {
        "answer": response["message"]["content"],
        "sources": [h["source"] for h in hits],
    }


if __name__ == "__main__":
    print(answer("How do we rotate the database credentials?"))

Two settings in the options dictionary deserve attention.

num_ctx sets the context window Ollama allocates for the request. Ollama picks a default based on available VRAM, and on a machine with less than 24 GiB that default is 4k tokens, far below what models like Llama 3.1 support. When a prompt exceeds the window, the input is truncated rather than rejected, so retrieved context can quietly disappear. Five chunks of 1,200 characters plus a system prompt and question fits in 8,192 tokens with room to spare. If you raise TOP_K or chunk size, raise num_ctx with it, and remember that a larger context costs memory and slows the first token.

temperature is set low because the goal is faithful summarization of retrieved text, not creative writing.

Exposing it as a service

A pipeline that only runs from a terminal does not help your team. Wrapping it in FastAPI takes a few lines:

# api.py
from fastapi import FastAPI
from pydantic import BaseModel

from rag import answer

app = FastAPI(title="Local RAG")


class Query(BaseModel):
    question: str


@app.post("/ask")
def ask(query: Query) -> dict:
    return answer(query.question)

uvicorn api:app --host 127.0.0.1 --port 8000

curl -s http://127.0.0.1:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "What is our on-call escalation policy?"}'

The handler is a plain def rather than async def on purpose. The Ollama and Chroma calls are blocking, and FastAPI runs sync handlers in a thread pool so they do not stall the event loop.

If you already have code written against the OpenAI SDK, you may not need to rewrite it at all. Ollama exposes an OpenAI-compatible endpoint at /v1, so pointing the client at your local server is often enough:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key is ignored
reply = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Summarize our deployment runbook."}],
)
print(reply.choices[0].message.content)

That compatibility layer is what makes "OpenAI is optional" literal: the switch can be a configuration change rather than a migration.

From weekend project to something people rely on

The code above works. Whether it works well depends on decisions that have nothing to do with which model you run.

Retrieval quality beats model size

When answers are wrong, inspect what was retrieved before you reach for a bigger model. Print the chunks and their distances for a handful of real questions. Most failures trace back to chunks that are too large (the relevant sentence is diluted), too small (the answer is split across two chunks that are never retrieved together), or missing entirely because a PDF was extracted as garbage. Build a small set of question-and-expected-source pairs and check that retrieval finds the right document; that is a cheap, repeatable evaluation you can run after every change.

Pure vector search also struggles with exact identifiers such as error codes, ticket numbers, and function names. If your users search for those, add keyword search alongside it. LanceDB supports full-text and hybrid search natively; with Chroma you can add a BM25 index next to it and merge the results.

Security does not come for free

"Local" means the data does not go to a vendor. It does not mean the system is secure. Ollama's API has no authentication, so keep it bound to localhost (the default) or behind a reverse proxy with auth if you expose it on a network. Your FastAPI layer also needs authentication and, in most organizations, document-level access control: if a user cannot open a file in the source system, retrieval should not hand its contents to them through the model. Store an access label in each chunk's metadata and filter on it with Chroma's where argument at query time.

Concurrency is the real limit

A laptop serves one person well. A shared team service is a different workload: a single GPU can only process a limited number of generations at once, and Ollama queues the rest. If you need many concurrent users, move the model to a dedicated GPU server and consider a throughput-oriented server such as vLLM, which also speaks the OpenAI API. The rest of the pipeline does not change.

When local RAG is the wrong choice

Self-hosting is a trade, not a free upgrade. Be honest about the cases where it loses.

• Hard reasoning over long context. A quantized 8B model is good at "find and summarize." It is noticeably weaker than frontier hosted models at multi-step reasoning, synthesizing conflicting sources, or working across very long documents. If your users ask analytical questions, test both before committing.
• Low or spiky volume. If a team sends a few hundred queries a month, a hosted API is likely cheaper than the engineering time to run and maintain your own stack. The cost argument for self-hosting gets stronger as volume grows and stays steady.
• No one to own it. Models, embeddings, and indexes need upgrades, re-indexing, and monitoring. A hosted API outsources that work. A self-hosted stack makes it yours.

The strongest case for local RAG is data that legally or contractually cannot leave your infrastructure, steady internal usage, and a team willing to treat it as a real service. Many organizations land on a hybrid: local models for sensitive corpora, hosted models where data sensitivity is low and reasoning quality matters most.

A practical checklist

1. Install Ollama, pull an 8B instruction model and an embedding model, and confirm both appear in /api/tags.
2. Chunk on document structure, store source metadata, and use deterministic IDs so re-ingestion is idempotent.
3. Use the embedding model's required prefixes, and never mix embedding models in one collection.
4. Set num_ctx explicitly so retrieved context is not silently truncated.
5. Instruct the model to answer only from context, to admit when it does not know, and to cite sources.
6. Build a small retrieval evaluation set before tuning prompts or swapping models.
7. Keep Ollama on localhost, put authentication in front of your API, and filter retrieval by user permissions.
8. Plan the move to a dedicated GPU server before the laptop becomes a shared dependency.

Running RAG entirely on your own hardware is no longer a research project. The hard parts are the same ones you would face with a hosted model, retrieval quality, access control, and evaluation, and none of them require sending your data anywhere.