Lab 8 — Local Models with Ollama (RAG, fully offline)¶

⏱️ ~40 min (incl. model download)

Everything you built in Lab 4, but with no cloud API and no API key — the embedding model and the chat model both run on your own machine via Ollama.

This lab is local-only — it does NOT run in Google Colab (Ollama must be installed on the machine running the notebook). Use VS Code or a local Jupyter.

⚠️ First run downloads ~2–4 GB of model weights (llama3.2 + nomic-embed) — allow several minutes on first execution; it's instant afterwards.

Setup¶

1. Install Ollama from https://ollama.com
2. Pull the two models this lab uses:

   ollama pull llama3.2
   ollama pull nomic-embed-text
   

3. Make sure the Ollama app/server is running (it usually starts automatically).

In [ ]:

!pip install -q ollama chromadb
import ollama

# Connectivity check — fail fast with a clear message if Ollama isn't running.
try:
    ollama.list()
    print("Ollama is running.")
except Exception:
    raise RuntimeError(
        "Could not reach Ollama. Install it from https://ollama.com, run "
        "`ollama pull llama3.2` and `ollama pull nomic-embed-text`, and ensure "
        "the Ollama server is running.")

!pip install -q ollama chromadb import ollama # Connectivity check — fail fast with a clear message if Ollama isn't running. try: ollama.list() print("Ollama is running.") except Exception: raise RuntimeError( "Could not reach Ollama. Install it from https://ollama.com, run " "`ollama pull llama3.2` and `ollama pull nomic-embed-text`, and ensure " "the Ollama server is running.")

1. The same documents as Lab 4¶

In [ ]:

documents = [
    "The Helios API rate limit is 100 requests per minute per API key.",
    "Helios refunds are processed within 5 business days to the original payment method.",
    "The Helios SDK supports Python 3.9 and above; Python 3.8 reached end of life.",
    "To rotate a Helios API key, go to Settings > Security > Rotate Key. Old keys expire in 24h.",
    "Helios stores data in the EU (Frankfurt) region by default for all new accounts.",
]

documents = [ "The Helios API rate limit is 100 requests per minute per API key.", "Helios refunds are processed within 5 business days to the original payment method.", "The Helios SDK supports Python 3.9 and above; Python 3.8 reached end of life.", "To rotate a Helios API key, go to Settings > Security > Rotate Key. Old keys expire in 24h.", "Helios stores data in the EU (Frankfurt) region by default for all new accounts.", ]

2. Embed locally with nomic-embed-text (runs on your machine)¶

In [ ]:

def embed(texts):
    return [ollama.embeddings(model="nomic-embed-text", prompt=t)["embedding"] for t in texts]

doc_embeddings = embed(documents)
print(f"Embedded {len(doc_embeddings)} chunks locally, dim={len(doc_embeddings[0])}")

def embed(texts): return [ollama.embeddings(model="nomic-embed-text", prompt=t)["embedding"] for t in texts] doc_embeddings = embed(documents) print(f"Embedded {len(doc_embeddings)} chunks locally, dim={len(doc_embeddings[0])}")

3. Store in Chroma (identical to Lab 4)¶

In [ ]:

import chromadb
chroma = chromadb.Client()
collection = chroma.create_collection("helios-docs-local")
collection.add(
    ids=[f"doc-{i}" for i in range(len(documents))],
    documents=documents,
    embeddings=doc_embeddings,
)
print("Stored in Chroma.")

import chromadb chroma = chromadb.Client() collection = chroma.create_collection("helios-docs-local") collection.add( ids=[f"doc-{i}" for i in range(len(documents))], documents=documents, embeddings=doc_embeddings, ) print("Stored in Chroma.")

4. Retrieve the most relevant chunks¶

In [ ]:

def retrieve(question, k=2):
    q_emb = embed([question])[0]
    res = collection.query(query_embeddings=[q_emb], n_results=k)
    return res["documents"][0]

def retrieve(question, k=2): q_emb = embed([question])[0] res = collection.query(query_embeddings=[q_emb], n_results=k) return res["documents"][0]

5. Generate the answer with a local llama3.2¶

In [ ]:

def answer(question):
    chunks = retrieve(question)
    context = "\n".join(f"- {c}" for c in chunks)
    resp = ollama.chat(model="llama3.2", messages=[
        {"role": "system", "content":
         "Answer ONLY from the context. If it's not in the context, say you don't know."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
    ])
    return resp["message"]["content"]

print(answer("How long until my old API key stops working after I rotate it?"))

def answer(question): chunks = retrieve(question) context = "\n".join(f"- {c}" for c in chunks) resp = ollama.chat(model="llama3.2", messages=[ {"role": "system", "content": "Answer ONLY from the context. If it's not in the context, say you don't know."}, {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}, ]) return resp["message"]["content"] print(answer("How long until my old API key stops working after I rotate it?"))

6. Grounding still works — ask something NOT in the docs¶

In [ ]:

print(answer("What is the Helios phone support number?"))

print(answer("What is the Helios phone support number?"))

Takeaway: this is the exact RAG pipeline from Lab 4 — chunk, embed, store, retrieve, augment, generate — but the embeddings and the LLM both run on your laptop. No API key, no per-token cost, no data leaving your machine. That is the case for local/open models: privacy, offline use, and zero marginal cost, traded against the convenience and raw capability of the big hosted models.

✅ Did it work?¶

• The connectivity check printed "Ollama is running" and the embed step reported chunks embedded locally with a dimension.
• The grounded answer came from local llama3.2 citing the 24h key-expiry doc, and the phone-support question was declined — all with no API key and no network calls.