Building with LLMs:
A Practical Study Guide

Understanding the constraints shapes every design decision. All techniques in this guide exist to solve one or more of these four problems.

Zero-Shot vs Few-Shot

• Zero-shot: give the LLM a task description only, no examples. Always start here.
• Few-shot: provide 2–5 input/output examples before the actual task.
• Few-shot helps when the model needs to understand output format, tone, or domain-specific patterns.
• Rule of thumb: try zero-shot first; add examples only when output quality is inconsistent.

Chain-of-Thought (CoT)

• Instruct the model to "think step by step" before giving a final answer.
• Dramatically improves performance on reasoning, math, and multi-step logic.
• Works because intermediate tokens give the model space to work through the problem.
• Can be explicit ("let's think step by step") or via few-shot examples that show reasoning.

Prompt Chaining

• Break a complex task into sequential prompts, each with a focused sub-task.
• Output of step N becomes input to step N+1.
• Benefits: easier to debug, can inject validation between steps, shorter prompts perform better.
• Use cases: research → draft → edit pipelines; extract → classify → respond flows.

Prompt Templates

• Parameterize prompts so the structure is stable — only the inputs change.
• Makes prompts testable, versionable, and reusable across similar tasks.
• Store templates separately from business logic.

LLM Judges / Evals with Promptfoo

Use an LLM to evaluate whether another LLM's output meets quality criteria. Define rubrics: what does "good" look like for this task? Useful for subjective quality (tone, helpfulness, completeness) where rule-based checks fail.

• Promptfoo: open-source tool for running LLM evals at scale.
• Define test cases with expected behaviors; run across model versions to detect regressions.
• Generates quality reports automatically.

Why to Avoid It

• Overfitting risk: fine-tuning on small datasets leads to narrow behavior that fails on edge cases.
• Stale quickly: a fine-tuned model becomes outdated as base models improve; you're constantly re-tuning.
• Data collection is hard: requires high-quality labeled examples that are expensive to produce.
• Prompt engineering often wins: a well-crafted prompt frequently achieves what fine-tuning would.

The Slack Fine-Tuning Failure (Case Study)

When Fine-Tuning Is Justified

• Very specific output format that prompts can't reliably produce.
• Latency/cost requirements that need a smaller model tuned to reach quality bar.
• You have thousands of high-quality labeled examples.
• Task is stable — won't change as base models improve.

RAG solves the domain knowledge problem: instead of baking proprietary knowledge into the model, you retrieve it at query time.

Vector Databases

• Store documents as embeddings — dense numerical representations of meaning.
• Semantic search: find documents similar in meaning, not just keyword matches.
• Common options: Pinecone, Weaviate, pgvector (Postgres), Chroma.
• Embedding model choice matters — different models have different tradeoffs for your domain.

Chunking Strategy

• Split source documents into chunks before embedding.
• Chunk size is critical: too small loses context; too large dilutes relevance signal.
• Common approaches: fixed-size with overlap, sentence-boundary aware, paragraph-level.
• Experiment for your specific domain — no universal right answer.

HyDE — Hypothetical Document Embeddings

Problem: query embeddings and document embeddings live in different semantic spaces. A user asking "why is the sky blue?" doesn't match well against a physics textbook paragraph.

1. Ask the LLM to generate a hypothetical answer to the query.
2. Embed that hypothetical answer.
3. Use that embedding for retrieval instead of the original query.

The hypothetical answer matches the document's "answer language" better than the question. Measurably improves retrieval quality on knowledge-intensive tasks.

RAG vs Fine-Tuning Comparison

Memory Types

• Working memory (short-term): the current context window. What the agent knows "right now." Limited by context window size.
• Archival memory (long-term): external storage (databases, vector stores, files) the agent can read/write. Enables persistence across sessions. This is what lets an agent "remember" past interactions.

Tools

Tools transform the LLM from a question-answerer into an actor.

• Web search: access current information.
• Code execution: run Python, test logic.
• Database read/write: query and update persistent state.
• API calls: interact with external services (calendar, email, CRM).
• File system: read documents, save outputs.

MCP — Model Context Protocol

• Standardized protocol for agents to call tools and other services.
• Treats tools as typed interfaces: here's what this tool accepts, here's what it returns.
• Enables tool reuse across different agents and systems.
• In multi-agent systems: agents communicate with each other via MCP — an agent is just another tool.
• Anthropic-backed standard gaining adoption across the ecosystem.

Autonomy Spectrum

The lecture walked through building evals for a customer support AI agent handling email inquiries: order tracking, address changes, refund requests.

Step 1: Task Decomposition

Break the agent's workflow into discrete, testable components. Each step is a potential failure point — evals should exist at each one.

1. Read and parse the customer email.
2. Identify the intent (what is the customer asking?).
3. Extract entities (order ID, address, email address).
4. Look up relevant information (order status, account data).
5. Draft a response.
6. Update database if needed.
7. Send the email.

LLM Judges with Rubrics

• Write a rubric: "A good response is polite, addresses all questions, provides the order status, and includes a next step."
• Ask an LLM to grade the output 1–5 on each rubric criterion.
• Use multiple LLM judges and average for reliability.
• Works better than rule-based checks for open-ended text.

Objective vs Subjective Evals

• Objective (code/rules): did the agent extract the correct order ID? Did it look up the right account? Was the database updated correctly? These are pure Python assertions.
• Subjective (LLM judges): is the response tone appropriate? Did the email address all the customer's questions? Is the response concise without being curt?

Component-Based vs End-to-End

• Component-based: test each step in isolation. Faster, easier to pinpoint failures. Use during development. "The intent classifier is wrong 30% of the time on refund requests."
• End-to-end: test the full workflow with real inputs. Catches emergent failures from step interactions. Slower, more expensive. Use before deploys.
• Use both. Component evals for development; end-to-end for regression testing.

LLM Traces Are Non-Negotiable

Good trace tooling: LangSmith, Braintrust, Helicone, Arize.

• Full chain of every prompt sent.
• Every LLM response at each step.
• Tool call inputs/outputs.
• Timing data per component.

Why Multi-Agent?

• Parallelism: run independent tasks simultaneously — climate, security, and energy agents all run at once.
• Specialization: each agent is optimized for a narrow domain — easier to tune, test, and debug.
• Reusability: a design agent built for the marketing team can also serve the product team.

Interaction Patterns

Smart Home Case Study (In-Class Exercise)

• Biometric/location agent: tracks where residents are in the home.
• Climate agent: controls temperature, access to room sensors and thermostats.
• Energy agent: monitors usage, can cut power to non-essential systems.
• Security agent: manages entry, identifies who's entering, sets permissions per person (parent vs child access rules).
• Fridge/grocery agent: monitors contents via camera, has access to grocery delivery API.
• Weather agent: pulls external APIs, adjusts blinds/temperature based on conditions.
• Orchestrator: single user-facing interface, delegates to all other agents.

Are We Plateauing?

Ilya Sutskever (OpenAI co-founder) raised this question publicly. Recent GPT releases showed less capability jump than previous generations; marginal returns on raw compute may be diminishing. Counter-argument: scaling laws still hold if compute and energy continue to scale. The question is unresolved.

Architecture Search as the Unlock

Most current LLMs are transformer-based. The transformer was a breakthrough — but not the end state. The human brain doesn't use transformers or backpropagation, yet learns faster, from less data, with far lower energy. This gap represents massive opportunity in architecture search. Labs are hiring thousands of engineers to run this search. The next "transformer discovery" — a new architecture that reduces compute by 10x — could restart exponential improvement.

Multi-Modality as a Source of Gain

• Adding image understanding made text models better at text.
• Adding audio will make image+text models better at everything.
• Adding video will push further gains across all modalities.
• Modalities reinforce each other: knowing what a cat looks like makes you better at writing about cats.
• The endpoint: robotics, where all modalities fuse into physical action.

Multiple Learning Methods in Harmony

Current AI uses individual methods in isolation. Human learning uses all simultaneously:

The Velocity Problem: Skills Have Short Half-Lives

Each topic in this guide has a standalone deep dive with extended examples, implementation patterns, and decision frameworks. New guides publish as the space develops.

• 01Start simple. Prompt engineering before RAG, RAG before fine-tuning, single agent before multi-agent.
• 02Measure everything. LLM traces + evals from day one. You can't debug what you can't observe.
• 03Decompose tasks. Break workflows into testable components; eval each one separately.
• 04Use LLM judges. For subjective quality at scale, LLMs grading LLMs works surprisingly well with rubrics.
• 05Prefer RAG over fine-tuning. RAG is debuggable, updatable, and cheaper in almost every scenario.
• 06MCP is the agent interface standard. Treat agents as typed tools; reuse them across systems.
• 07Multi-agent = parallelism + specialization. Not magic — just good engineering decomposition.
• 08Stay broad. AI techniques have short half-lives. Pull depth when you need it, don't accumulate in advance.