Module 6 — Ecosystem, Frameworks & Developer Practice

In this module

• Know the major AI frameworks — LangChain, LlamaIndex, CrewAI, AutoGen — and when to reach for a framework vs. raw API calls.
• Identify the AI-native developer tools (Copilot, Cursor, Claude Code) and how they slot into a normal dev workflow.
• Understand why evals are the test suite for LLM quality, and why "looks right" is not a passing criterion.
• Reason about cost, latency, and reliability as first-class engineering concerns when calling LLMs in production.
• Apply responsible-AI basics — bias, PII exposure, audit trails, data residency — before shipping anything to users.

Time: ~35 min · Labs: Lab 6 · Lab 7

Why this matters

Modules 1–5 taught you the primitives: how LLMs work, how to prompt them, how to build RAG pipelines, and how to wire up agents. This module zooms out to the full production picture — the frameworks that package those primitives, the tools that make you faster as an AI developer, and the operational concerns (cost, reliability, safety) that separate a proof-of-concept from something you can actually ship. For a senior developer the shift is familiar: you have moved from understanding TCP to choosing a networking library and thinking about SLAs.

Frameworks overview — LangChain, LlamaIndex, CrewAI, AutoGen

You can call every LLM API by hand — craft the JSON, parse the response, manage retries. Frameworks wrap that boilerplate, provide pre-built chains of common patterns, and add observability. The tradeoff is the same as with any library: less boilerplate, more magic you need to understand when something breaks.

LangChain is the general-purpose orchestration framework. Its core abstraction is the composable chain: prompt → model → output parser → next step. It covers the widest surface area — tool use, RAG pipelines, memory, multi-agent coordination via LangGraph — at the cost of a large API surface and frequent breaking changes. Reach for LangChain when you need flexibility across many providers and are willing to read the changelog.

LlamaIndex starts from data: it handles document ingestion, chunking, embedding generation, vector database integration, and retrieval with re-ranking in one cohesive API. For RAG-heavy applications — document Q&A, knowledge bases, search — LlamaIndex is typically faster to production-quality than building the same pipeline from scratch with LangChain.

CrewAI takes a role-based approach to multi-agent systems: you define agents with named roles (Researcher, Analyst, Writer), assign them tasks, and let them collaborate. The mental model is close to how you'd staff a small team, which makes it easy to reason about. CrewAI handles the orchestrator logic; you define the specialists.

AutoGen (from Microsoft) is a conversation-centric multi-agent framework. Agents exchange messages in structured dialogues; any agent can be a human proxy, an LLM, or a function-executing bot. It shines for workflows where human-in-the-loop review is part of the process.

When to skip frameworks entirely: if your use case is a single LLM call or a fixed two-step pipeline, the framework overhead is noise. Vanilla API calls are easier to debug, cheaper to maintain, and have no transitive dependency risk. The rule of thumb: reach for a framework when you need more than three LLM calls chained with conditional logic, or when you need the built-in integrations (vector stores, observability, memory).

At a glance:

Framework	Best for	Core abstraction
LangChain	general orchestration across many providers	composable chains (prompt → model → parser)
LlamaIndex	RAG / document Q&A	data ingestion + retrieval pipeline
CrewAI	role-based multi-agent teams	named agents with roles + tasks
AutoGen	human-in-the-loop multi-agent dialogue	agents exchanging messages

Which one? — a quick decision tree:

flowchart TD
  START["What's your problem shape?"] --> Q1{"Mostly searching<br/>your own documents?"}
  Q1 -->|yes| LI["LlamaIndex"]
  Q1 -->|no| Q2{"Multiple agents<br/>collaborating?"}
  Q2 -->|"yes, role-based team"| CR["CrewAI"]
  Q2 -->|"yes, human in the loop"| AG["AutoGen"]
  Q2 -->|no| Q3{"More than ~3 calls<br/>with branching logic?"}
  Q3 -->|yes| LC["LangChain"]
  Q3 -->|no| RAW["Just call the API directly"]

AI developer tools — Copilot, Cursor, Claude Code

The same LLMs you are building with are also accelerating the act of building. Three tools have become standard in AI-augmented development:

GitHub Copilot integrates into VS Code and JetBrains IDEs as an autocomplete layer. It suggests the next line (or next function) based on your file context. The mental model is an extremely fast junior pair-programmer who has read every public repository — useful for boilerplate, test stubs, and pattern-matching on familiar code shapes, less useful for novel architectural decisions.

Cursor is an IDE built around chat-driven editing. You describe a change in natural language; Cursor applies it across multiple files, respects your project's structure, and lets you review diffs before accepting. The distinguishing feature is multi-file context: it can refactor a module end-to-end rather than one function at a time.

Claude Code — a terminal-first agentic coding assistant that can read, edit, and run code across your whole project. It sits closer to the agent model from Module 5 than to the autocomplete model of Copilot — you give it a goal, it forms a plan, and it executes.

None of these tools changes the fundamentals: you still need to understand what good code looks like, review everything before it ships, and take responsibility for correctness. They reduce the cost of the first draft, not the cost of thinking.

The AI stack — model to application

Seeing the layers in one diagram helps clarify where each framework and tool sits:

The AI stack — from application down to foundation model:

Your application code sits at the top. Frameworks occupy the orchestration layer, abstracting the model API and the infrastructure layer beneath. RAG and agents are horizontal concerns that span orchestration and model — both need the model layer but also depend on infrastructure (vector stores, external tools). The model layer is an external API dependency you do not own, which is why the operational concerns in the next sections matter so much.

Evals — the test suite for fuzzy correctness

Traditional unit tests assert exact equality: assert result == expected. LLM outputs are non-deterministic and open-ended, so exact equality never works. Evals are the LLM equivalent of a test suite: a structured set of inputs with criteria for what a good output looks like, run automatically to detect regressions.

Four common eval patterns, cheapest and broadest first:

Eval pattern	What it checks	Cost
Heuristic assertions	contains a keyword, matches a regex, parses as valid JSON	cheapest
Reference comparison	semantic similarity to a gold-standard answer (via embedding)	low
LLM-as-judge	a second LLM rates relevance, accuracy, safety	medium (extra call)
Human review	the ground truth	highest

In plain words

A normal unit test is a multiple-choice key — the answer is either exactly "B" or it's wrong. An eval is grading an essay with a rubric: there's no single correct string, so you score the answer against criteria ("did it cite the document? stay on topic? avoid leaking PII?"). Same discipline as testing, just scored against a rubric instead of an exact match.

The development analogy: writing evals feels like writing tests for a service with a probabilistic contract rather than a deterministic one. Start with a small golden dataset (20–50 representative inputs), automate the heuristic checks, and add LLM-as-judge for the outputs that matter most. Run evals in CI so that prompt changes or model upgrades don't silently degrade quality.

"Looks right" is not a passing criterion. Without evals you are flying blind every time you tweak a prompt or upgrade a model.

Cost & latency — tokens are money

Every token sent to and received from an LLM has a price. At small scale the numbers are trivial; at production scale — millions of requests per day — prompt design becomes a cost engineering problem.

Key levers:

• Context window size matters. Sending a 100 k-token system prompt on every request costs roughly 100× more than a 1 k-token prompt. RAG's entire value proposition includes cost: you retrieve the three most relevant chunks rather than stuffing the whole knowledge base into context.
• Model routing. Not every task needs the most capable (most expensive) model. Route classification and extraction tasks to a smaller, cheaper model; reserve the large model for synthesis and generation that actually needs it.
• Output length. Set max_tokens intentionally. Unbounded responses on high-traffic endpoints will spike your bill and your P99 latency in lock-step.
• Caching. Provider-side prompt caching (Anthropic, OpenAI) re-uses the KV cache for a repeated prefix at a fraction of the normal input price. Structure prompts so the static part (system prompt, RAG chunks) comes first and the variable part (user message) comes last.
• Latency. Tokens/second is a function of model size and provider infrastructure. Streaming (stream=True) reduces perceived latency for chat interfaces even when total generation time is the same. For batch workloads, use async calls or the provider's batch API.

The mental model: treat the model API like a database query. You would never issue a SELECT * on a million-row table in a hot path — don't send a hundred-page document in a prompt when a targeted retrieval would do.

Rate limits & reliability — LLMs are external APIs

LLMs are network services. Everything that applies to any third-party API applies here, amplified by the fact that LLM calls are expensive and slow:

• Rate limits. Providers enforce tokens-per-minute and requests-per-minute ceilings. Design for throttling from day one: use an exponential-backoff retry library (e.g. tenacity in Python), catch 429 responses, and surface the retry delay to callers rather than silently blocking.
• Timeouts. A model generation can take 30–60 seconds for a long response. Set explicit read timeouts; never let a stalled API call hold a thread indefinitely.
• Fallbacks. Consider a secondary provider or a smaller local model as a fallback when the primary provider is degraded. This is operational insurance, not just a nice-to-have.
• Observability. Log every LLM call: model name, token counts, latency, cost estimate, and a trace ID linking the call to the user request. You cannot debug production issues without this data.

The developer analogy: treat the model provider the way you treat a third-party payment API — with circuit breakers, retries, timeouts, and a mock for local development.

Responsible AI basics — bias, PII, audit trails, data residency

Shipping AI to users creates obligations that don't exist with traditional software, because the failure modes are harder to predict and the harms can be diffuse.

Bias and fairness. LLMs reflect the biases in their training data. If your application makes recommendations, generates content about people, or is used in high-stakes decisions, test explicitly across demographic groups and document the results. Bias audits are not a one-time gate; they are a recurring process.

PII exposure. User data sent to a model API leaves your infrastructure. Audit what you send: strip or pseudonymise personally identifiable information before it goes into a prompt. Check your provider's data-use and retention policies — "your prompts are not used for training" is not the same as "your prompts are not stored."

Audit trails. Log inputs and outputs for AI-assisted decisions, especially in regulated domains. The question "why did the system say this?" will be asked. Without logs, you cannot answer it.

Data residency. Many providers route requests through data centres in multiple jurisdictions. If your data is subject to GDPR, HIPAA, or local data-sovereignty rules, verify that the provider offers a compliant deployment option and configure it explicitly — the default endpoint is usually the globally load-balanced one.

Output safety. Prompt injection is a real threat: malicious content in user input or retrieved documents can hijack model behaviour. Apply input validation, output filtering, and the principle of least privilege to tool permissions.

Responsible AI is not a separate ethics workstream bolted on at the end — it is a set of engineering requirements that should be in the design doc alongside performance and reliability.

Pre-ship checklist — run through this before any AI feature reaches users:

• [ ] PII — stripped or pseudonymised before anything leaves your network; provider's data-use & retention policy checked
• [ ] Bias — outputs tested across demographic groups for any people-affecting or high-stakes use; results documented
• [ ] Audit trail — inputs, outputs, model/version, and a trace ID logged for every AI-assisted decision
• [ ] Data residency — provider region/endpoint configured for GDPR/HIPAA/sovereignty needs (not the default global one)
• [ ] Prompt injection — untrusted input/retrieved content delimited and validated; tool permissions least-privilege
• [ ] Output safety — model output filtered/validated before it's shown or acted on
• [ ] Human-in-the-loop — a person reviews before any consequential or irreversible action

Mental model

Think of an AI framework the way you think of a web framework. You could hand-write raw HTTP socket code for every endpoint, but Express or FastAPI exists because the patterns are common enough to standardise. LangChain, LlamaIndex, and CrewAI do the same for LLM orchestration, RAG pipelines, and multi-agent systems respectively. Choose the framework whose abstractions match your problem shape — and be prepared to drop to raw API calls whenever the abstraction gets in the way.

Hands-on

Open Lab 6 — RAG with LangChain and Lab 7 — Agent with Frameworks. In Lab 6 you will rebuild the vanilla RAG pipeline from Module 4 using LangChain, observing how the framework changes the code shape; in Lab 7 you will wire up a multi-step agent using CrewAI and compare it to the hand-rolled agent loop from Module 5.

Optional: Lab 8 — Local Models (Ollama) rebuilds the RAG pipeline with a model running entirely on your own machine — no API key, no cost.

Go deeper

• AI Agent Frameworks Compared — Atlan — Side-by-side breakdown of LangChain, LlamaIndex, CrewAI, and AutoGen with use-case fit tables. Open this to make a framework choice for your next project without guessing.

• LLM Evals: Everything You Need to Know — Hamel Husain — Practitioner FAQ format, written by an ML engineer who has run evals at scale. Read this before writing your first eval — it explains the four evaluator types and when to use each one.

• LLM Price Calculator — Artificial Analysis — Interactive calculator that compares token cost and throughput across every major model and provider. Bookmark it; open it whenever you are choosing a model or estimating a monthly bill.

• AI Governance Checklist — OpenRouter Blog — Concise, stack-mapped responsible-AI checklist covering bias, PII, audit logging, and data residency. Use it as a pre-ship review gate for any AI feature.

Jargon recap

RAG · chunking · embedding · vector database · retrieval · multi-agent · orchestrator · evals · context window · token · hallucination · prompt injection · foundation model

Check yourself

Try to answer before expanding — active recall is what makes it stick.

When should you reach for a framework versus raw API calls?

Use raw API calls for a single call or a fixed two-step pipeline — they are easier to debug and have no dependency risk. Reach for a framework when you need more than ~3 chained calls with conditional logic, or want built-in integrations (vector stores, memory, observability). Match the framework to your problem shape: LlamaIndex for RAG, CrewAI/AutoGen for multi-agent, LangChain for general orchestration.

Why can't you test LLM output with normal assert result == expected, and what do you use instead?

LLM output is non-deterministic and open-ended, so exact equality never holds. You use evals — a golden dataset scored against criteria, like grading an essay with a rubric: heuristic checks (regex, valid JSON), reference comparison, LLM-as-judge, and human review, run in CI.

What are the main levers for controlling LLM cost in production?

Right-size the context (RAG retrieves a few chunks instead of stuffing everything), route easy tasks to cheaper/smaller models, set max_tokens deliberately, and cache repeated prompt prefixes. Tokens are money — treat a model call like a database query, never SELECT * in a hot path.

What goes on the responsible-AI pre-ship checklist?

Strip or pseudonymise PII before it leaves your network, test for bias on people-affecting decisions, log inputs/outputs with a trace ID for audit, configure data-residency for GDPR/HIPAA, guard against prompt injection with least-privilege tools, filter outputs, and keep a human in the loop for consequential actions.

Key takeaways

• Frameworks (LangChain, LlamaIndex, CrewAI, AutoGen) package common LLM patterns — choose based on whether your problem is orchestration-heavy, retrieval-heavy, or multi-agent-heavy; use raw API calls when the pattern is simple.
• Evals are the test suite for LLM output quality: automate heuristic checks and LLM-as-judge criteria, run them in CI, and treat prompt changes like code changes.
• Tokens are money and latency: right-size prompts, route tasks to cheaper models where possible, cache repeated prefixes, and set explicit output limits.
• LLMs are external APIs — design for rate limits, timeouts, and provider outages from day one, with retries, fallbacks, and full observability on every call.
• Responsible AI is engineering, not ethics theatre: strip PII before it leaves your infrastructure, log every AI-assisted decision, verify data-residency compliance, and guard against prompt injection.

You've finished the course. Revisit the Glossary and Resources anytime.