Free study guide

50 Advanced GenAI Concepts

I've interviewed candidates for AI roles at top Y Combinator startups. Most of them can tell me what these things are, but not how they work, what they cost, or what the downsides are. That's the difference between a mid-level and a senior answer.

Each concept below has the slide from the video, how it works, the trade-off, and a link to the moment I cover it.

01Synthetic Data & Model Collapse

Watch at 0:45
Slide: Generate, filter, train

Synthetic data is training data generated by a model instead of collected from real people or real events.

You point a strong model at a task, have it produce examples by the million, throw out the bad ones, and train on what survives.

Why it matters: the internet is finite, and most of the good data is already used up by today's models. If you need a million examples of some narrow format, nobody has collected that for you.

How it works:

  • A teacher model generates candidates
  • A verifier scores them and throws out the bad ones
  • Only the survivors reach the training set

That filter is doing most of the real work.

The trade-off: model collapse. Train a model on its own unfiltered output for long enough and it drifts toward its own average, smoothing away the rare cases you cared about.

So use synthetic data where the answer can be checked automatically:

  • Code that has to run and return the expected output
  • Math where the equation has to balance

And never train on synthetic data alone. Keep real data in the mix.

02RLHF (Reinforcement Learning from Human Feedback)

Watch at 2:07
Slide: Two steps, and humans only touch the first

RLHF is the training stage that turns a model that predicts plausible text into one that's actually helpful.

It happens in two steps, and humans only touch the first:

  1. Collect preferences. Show a person two responses to the same prompt and ask which is better. Those rankings train a separate reward model, whose only job is to score a response the way a person would.
  2. Optimize against the reward model. The language model is trained to produce answers the reward model rates highly, usually with PPO (proximal policy optimization).

PPO caps how far each update can move the model. Without that cap, the model could take big steps in the wrong direction chasing the reward. Many small, careful steps is the idea.

The trade-off: reward hacking. The reward model is a proxy, never the real thing. The model will find responses the reward model loves and real people don't, like hedging, flattery and padding.

Reach for RLHF when you need behavior you can rank but can't write a rule for.

In an interview: this comes up constantly. Be ready to explain both steps and why the reward model is only a guess at what people like.

03Constitutional AI & RLAIF

Watch at 3:12
Slide: Answer, criticise, rewrite

Constitutional AI is Anthropic's alignment method, where a model critiques and revises its own answers against a written set of principles.

The constitution is just plain-language rules, like "prefer the response that is least harmful". It replaces most of the human labelling with self-supervision.

The loop:

  1. Give the model a risky prompt. It answers.
  2. Ask the same model to criticise its answer against one principle from the constitution.
  3. Ask it to rewrite the answer.

Do that across a pile of prompts and you have a dataset of revised responses to fine-tune on. The same principles then generate preference pairs for the reinforcement learning stage. That's RLAIF: reinforcement learning from AI feedback.

What you gain: scale and transparency. The rules are written down, so you can read them and argue with them.

The trade-off: you give up an outside check, because the model is grading itself. If it can't recognize a harmful response, the constitution won't catch it.

04Transformers & Self-Attention (Query, Key, Value)

Watch at 4:50
Slide: Three vectors per token, four operations

This is probably the question I've been asked most in AI engineer interviews, and one I love to ask as an interviewer.

The transformer is the architecture behind essentially every modern language model. It revolutionized machine learning because it processes a whole sequence at once instead of token by token. What makes that work is self-attention: every token looks at every other token and decides which ones matter to it.

Each token gets three vectors:

  • Query: what this token is looking for
  • Key: what this token can offer others
  • Value: what it actually contributes

Four operations: dot a token's query against every key to get scores, softmax the scores into weights that add up to one, and take the weighted sum of the values. That's the whole operation, repeated across heads and layers.

Example: in "The trophy didn't fit in the case because it was too big", the word "it" puts almost all of its attention on "trophy", with "big" as the runner-up. Nobody programmed that rule. It fell out of training.

The trade-off: cost. Every token attends to every other token, so compute and memory grow with the square of the sequence length, O(n²). That's why long context is expensive, why long prompts generate slower, and why tricks like flash attention and sliding windows exist.

In an interview: if you get one architecture question, this is it.

05Multimodal LLMs & Vision Language Models

Watch at 7:18
Slide: Patches join the same stream as words

Multimodal LLMs take in more than text. A vision language model reads images alongside words and reasons about both inside the same context window.

The interesting part: the language model never actually learns to see.

The pipeline:

  1. Slice the image into small patches
  2. Push them through a separate vision encoder, usually a vision transformer, which turns each patch into an embedding
  3. A small projection layer maps those embeddings into the same space as the text tokens

After that, the model just sees a stream of ordinary tokens. Image patches join the same stream as the words.

The trade-off: resolution against cost. A high-resolution image becomes hundreds or thousands of tokens, so a few screenshots can eat your context window and pull the model's focus away from the text.

These models also still hallucinate on fine detail: small text, exact counts, precise positions. They're great for things like OCR, but verify anything numeric you pull out of an image.

06Mixture of Experts (MoE)

Watch at 8:56
Slide: The router picks two experts per token

Mixture of Experts is an architecture where a model's feed-forward layer is split into many smaller networks called experts, and a router picks only a couple of them for each token.

It's how you get a huge model that behaves like a small one at inference time.

  • A dense model runs every parameter for every token
  • A Mixture of Experts model might hold hundreds of billions of parameters but only activate a fraction per token, because the router sends each token to, say, two experts out of 128

You pay the compute of a small model and get the capacity of a big one. Same layer, different token, and a different pair of experts lights up.

The trade-offs:

  • Memory. Every expert sits in GPU memory whether you use it or not, so you don't escape the memory bill of a large model.
  • Balance. Routers collapse onto a few favorite experts unless you add a load-balancing loss during training, which adds complexity.

07Knowledge Distillation

Watch at 10:11
Slide: Teacher and student

Knowledge distillation is training a small model to imitate a large one. The big model is the teacher, the small model is the student, and the student learns from the teacher's outputs rather than from the original labels.

Why it works: soft targets. The teacher's full probability distribution carries far more information than a hard label. If the teacher says an image is 70% cat, 25% fox and 5% dog, it has just told the student that cats and foxes look alike, without the student needing any extra parameters to work that out. Raising the temperature spreads those probabilities out further.

The student is trained on two losses: one against the true label, one against the teacher.

The trade-offs:

  • A ceiling. A student rarely beats its teacher, and it inherits the teacher's mistakes wholesale.
  • Narrowness. The student only copies the teacher on the data you distilled over. It looks great on your eval set and falls off a cliff on the long tail, usually while staying just as confident. Broad coverage in the distillation data is most of the work.

Use it when you have a model that works and you need that behavior to run cheaper, faster, or on a phone.

08LoRA & QLoRA Fine-Tuning

Watch at 11:57
Slide: How LoRA works

LoRA stands for Low-Rank Adaptation. QLoRA is the quantized version.

The problem: you want a big open model to write in your company's voice. The obvious way is to fine-tune it, nudging all 70 billion parameters a little. That takes a room full of GPUs, and you end up with a whole second copy of the model. Ten customers, ten copies.

LoRA's idea: don't touch the model at all.

  • Freeze the base model
  • Add a small side path next to each layer, made of two small matrices
  • The first shrinks the signal down to something like 16 numbers wide; the second blows it back up to full size
  • Only those two matrices get trained, and their output is added onto the original layer's output

It works because you're not teaching the model new facts, you're teaching it a style or a format, and that's a small nudge.

The payoff: your fine-tune is megabytes, not gigabytes. Load the big model once and snap on a different adapter per customer, like plugins.

QLoRA squeezes harder: it stores the frozen model at 4 bits instead of 16, roughly a quarter of the resolution. That's what lets a 70-billion-parameter fine-tune fit on one GPU, at a small cost in quality.

The catch: LoRA changes how the model behaves, not what it knows. If the base model never saw your data, no adapter will teach it. That's a job for retrieval.

In an interview: asked how you'd fine-tune on a single GPU? QLoRA is the answer.

09KV Cache (Key-Value Cache)

Watch at 14:31
Slide: Writing word 500, with and without the cache

A model writes one token at a time, and to pick the next one it has to look back at everything so far.

On token 500, attention needs the keys and values for tokens 1 through 499. Recomputing all of them every step means each new token costs more than the last, and the whole answer costs roughly the square of its length.

The insight: those keys and values never change. The entry for token 12 depends only on token 12 and the tokens before it. Nothing generated later can touch it.

So you compute each one once, keep it, and reuse it. That stored pile is the KV cache. Each generation step now does work on exactly one new token instead of the whole sequence.

The trade-off: memory. The cache grows with every token and with every user you serve at once. On long conversations it can take up more GPU memory than the model's own weights, which caps how many people you can serve.

Keep the scope straight: the KV cache belongs to one request. It isn't shared between users, and it's thrown away when the request finishes. Sharing it across requests takes extra machinery, which is where the next few concepts come in.

10vLLM & Continuous Batching

Watch at 16:06
Slide: Continuous batching

vLLM is the software you actually run when you serve an open model to real users. It's not the model, it's the machinery around it. Think of it as the operating system for a model you host.

It:

  • Accepts requests over an HTTP API that looks like OpenAI's
  • Decides what order to run them in
  • Splits the model across GPUs if it has to
  • Streams tokens back as they appear

Why it exists: serving a model badly wastes most of your hardware.

Continuous batching is the part worth understanding. The old way was to gather, say, 16 requests, run them together, and wait. The batch isn't done until its slowest member is done, so 15 users who asked short questions sit there finished, holding their seats, while one person's essay generates. The GPU runs at a fraction of what it could.

With continuous batching, the moment any request finishes, it leaves and a queued request takes its seat immediately. The batch is never idle and never waiting on its worst case. That's where the throughput comes from.

In an interview: asked how you'd serve an open model in production? vLLM is the expected answer, and continuous batching is why.

Builds on #9 KV Cache (Key-Value Cache).

11PagedAttention

Watch at 17:47
Slide: How PagedAttention works

PagedAttention is the memory trick that makes vLLM's batching possible. It builds straight on the KV cache.

The old way: for each request, book one continuous stretch of GPU memory big enough for the longest answer it might ever produce. If the answer stops after 20 tokens instead of 2,000, the rest of that booking sits empty. And because each booking has to be continuous, you end up with gaps between them too small to use, like a car park full of spaces no car fits into.

The fix, borrowed from operating systems:

  • Chop the KV cache into small fixed-size blocks
  • Let them sit anywhere in memory
  • Keep a small index of where each piece went

Nothing has to be next to anything, so almost nothing is wasted. And because blocks are independent, two requests that start with the same prompt can point at the same blocks instead of each storing its own copy.

Why you care: freed memory means more requests resident at once, which is exactly what continuous batching needs. Same GPU, several times the users.

The trade-off: a little bookkeeping on every attention step. It's worth it every time.

Builds on #9 KV Cache (Key-Value Cache) and #10 vLLM & Continuous Batching.

12Speculative Decoding

Watch at 19:21
Slide: One round of draft and verify

Speculative decoding makes a large model generate text faster without changing a single token of what it produces.

How it works:

  • A small draft model guesses the next few tokens
  • The big model checks all of those guesses in one forward pass
  • Every token that matches what the big model would have produced is kept; everything from the first mismatch onward is thrown away

Why that helps: generation is memory-bound, not compute-bound. Producing one token means hauling the entire model's weights through the GPU, so checking five tokens costs almost the same as checking one.

The output is identical to normal decoding. It's exact, not an approximation, so you can turn it on in production without a quality review. Typical speedups are two to three times.

The trade-off: you're running two models, and a draft model that disagrees often is pure overhead.

Slide: Two timelines, streamed and not

LLM streaming sends tokens to the client as the model generates them, instead of waiting for the whole response. The connection stays open, the server pushes each chunk down, and the client renders it as it arrives.

Why it matters: perception. Total generation time doesn't change at all. What changes is that the user sees the first word after a few hundred milliseconds instead of staring at a spinner for eight seconds. Once text starts appearing, people feel like the answer is already there.

The trade-off: everything downstream gets harder.

  • You can't validate JSON you haven't finished receiving
  • You can't moderate a half-written answer
  • Error handling gets messy, because the response already returned a success status before it failed mid-stream

A trick for streaming JSON: close the object yourself after each complete key-value pair and render what you have, then keep filling it in as more arrives. Chat apps do this a lot.

Stream anything a person reads live. Skip it for batch jobs and for structured output you need to parse before you can trust it.

14LLM Quantization

Watch at 21:30
Slide: The same weight, at four precisions

Quantization means storing a model's weights at lower numerical precision: taking 16-bit numbers and squeezing them down to 8 bits, or 4.

Why bother? Size. A 7-billion-parameter model needs roughly 14 GB at 16-bit. At 4-bit it fits in under 4 GB. That's the difference between renting a data-center GPU and running it on your laptop.

It's usually faster too, because you're moving a quarter of the bytes, and memory bandwidth is what actually caps generation speed.

The trade-off: you lose information. The same weight drifts a little every time you drop a precision level.

  • 8-bit is close to free
  • 4-bit is measurable but usually fine
  • Below that, quality falls off a cliff

Have evals in place so you know how far you can quantize before it gets too bad.

Reach for quantization when you're self-hosting and memory is your bottleneck. If you're calling a hosted API, it isn't your problem.

Slide: The router

LLM routing sends each request to a different model depending on what it needs, instead of pushing everything through one big expensive model.

The logic: most production traffic is easy. Classification, short rewrites, formatting. A small, fast model handles those at a fraction of the cost. The genuinely hard requests go to the frontier model.

The router is whatever makes that call:

  • A keyword rule or regex
  • A small classifier
  • A cheap model asked to grade the difficulty first

I'd be surprised if the big providers don't do some version of this behind their own APIs.

The trade-off: the router becomes its own failure point. Misroute a hard question to the small model and you ship a bad answer, and now you're debugging two systems.

In an interview: this usually comes up as "how would you cut inference costs?" Routing is one of the answers they're looking for.

16Time to First Token (TTFT)

Watch at 23:57
Slide: One request, on a timeline

Time to first token is the delay between sending a request and getting the very first token back. It's the latency your users actually feel.

It's separate from throughput, which is how fast tokens come out once they've started. You can have excellent throughput and still feel sluggish, because the user was watching a blank screen for two seconds before anything appeared.

Three things go into it:

  • Network round trip
  • Queueing on the server
  • Prefill: the model reading your entire prompt and computing attention across it before it can produce anything

Prefill scales with prompt length, which is why TTFT gets worse the more context you stuff in. Everything after the first token is inter-token latency.

To improve it: shorten the prompt, cache the prefix, or use a smaller model. Streaming doesn't fix TTFT, it just makes it the only latency anyone notices.

In an interview: a great metric to bring up when asked how you'd make a chatbot feel faster.

Builds on #13 LLM Streaming.

17Prefix Caching (Prompt Caching)

Watch at 24:47
Slide: Prefix caching across requests

Prefix caching means the server keeps the processed state of a prompt's opening tokens, so any later request that starts with the exact same tokens skips prefill on that part.

Where the KV cache lives inside one generation, prefix caching works across requests. Your system prompt, tool definitions and reference documents are the cached prefix. Only the user's message changes. The first call pays for everything and stores it; every call after pays for the tail.

Two names, same mechanism:

  • Prefix caching on your own server (vLLM, SGLang). On by default, automatic, and it even works across users who share a prefix. You gain time to first token and throughput.
  • Prompt caching on a provider API. You mark a breakpoint, the provider holds the state for a few minutes, and cached tokens are billed at a fraction of the normal rate. You gain money.

This is one of the techniques that saved a startup I worked with over a million dollars a year in token costs.

Two gotchas:

  • It's prefix-only. Change one token near the top and the whole cache is cold. Put all static content first and anything that varies, like a timestamp or a user name, as late as possible.
  • It expires in minutes, so a low-traffic endpoint may never hit.

Builds on #9 KV Cache (Key-Value Cache) and #11 PagedAttention.

18Semantic Caching

Watch at 26:05
Slide: Matching on meaning, not tokens

Semantic caching matches meaning instead of tokens. It embeds the incoming question, compares it against questions you've already answered, and if one is close enough, returns the stored answer without calling the model at all.

That's the distinction: the other caches make the model cheaper to run. Semantic caching skips it entirely.

"How do I reset my password?" and "I forgot my password, what now?" are different strings but the same question. Semantic caching catches that where a prefix match never would.

The threshold is the entire design:

  • Too loose, and you confidently answer a question nobody asked
  • Too tight, and you almost never hit

Use it for high-volume, repetitive traffic like support FAQs and docs lookups, where the cost of a wrong answer is low. Keep it away from anything personalized, anything with numbers (like financial data), and anything where a near-miss does damage.

Slide: Extract, cluster, answer

GraphRAG is Microsoft's approach to retrieval-augmented generation. Instead of only chunking and embedding your documents, you first run a language model over the whole corpus to pull out entities and the relationships between them, and build a graph.

Why bother? Normal RAG can't answer global questions. Ask it for the main themes across 2,000 support tickets and it falls over, because it fetches five chunks and no five chunks hold that answer.

Extract, cluster, answer:

  1. A language model reads the corpus and extracts entities and how they relate
  2. The graph is clustered into communities, tightly connected groups, each with a written summary
  3. A broad question is answered from those summaries, not raw chunks

Summaries can reference the chunks they came from, so you still get citations. You might discover that most login errors trace back to authentication in one region, which makes debugging a lot faster.

The trade-off: you pay for a language model pass over your entire corpus before anyone asks anything, and re-indexing when documents change hurts. Think of it like an index in a SQL database: slower writes, faster reads.

Reach for GraphRAG when questions are broad and connective. Stay with plain vector search for narrow lookups.

23HNSW Vector Index

Watch at 35:04
Slide: One search through the HNSW layers

HNSW (Hierarchical Navigable Small World) is the graph index that makes approximate nearest-neighbour search fast enough to use at scale.

The problem: comparing your query against 10 million stored vectors one at a time is exact, and far too slow.

The solution: a multi-layer graph.

  • The top layer is sparse, with long hops across the whole space
  • Every layer below gets denser
  • A search starts at the top, greedily moves toward the closest node it can find, then drops a layer and repeats, until it's walking a fine-grained neighbourhood at the bottom

Millions of comparisons become a few hundred.

When does it drop a layer? At each node it checks the neighbours and moves to whichever is closer to the query. When no neighbour is closer than where it's standing, that layer can't take it further, so it drops.

How are the layers built? Each vector, as it's inserted, is randomly assigned a top layer. Most land only on the bottom layer; a few make it higher. Then it's linked to its nearest neighbours on each layer it reached.

The trade-off: "approximate" is the key word. HNSW can miss the true nearest neighbour, so you tune recall against speed and memory. That dial is why almost every vector database ships HNSW as its default index.

24Cosine Similarity vs Dot Product vs Jaccard

Watch at 37:21
Slide: Three metrics, and when to reach for each

Three ways to measure how alike two things are. Picking the wrong one quietly wrecks your retrieval.

Cosine similarity measures the angle between two vectors and ignores their length. It asks which direction each one points, not how far.

That's what you want for text, because length mostly tracks how wordy something is, not what it's about. A two-line tweet and a 40-page whitepaper on the same subject should score equally. A customer who writes three paragraphs shouldn't match an FAQ better than one who writes six words.

Dot product is cosine with length left in, so longer vectors score higher. Use it when length carries information.

Say you're Spotify. Song vectors are trained on listening data, so a track with millions of plays ends up with a longer vector than one nobody played. A listener wants more lo-fi: with cosine, an unheard bedroom demo scores the same as a hit. With dot product, the popular one wins. (If your vectors are normalized to length one, dot product and cosine rank identically.)

Jaccard similarity compares sets: how much they share, divided by everything between them. Use it for things that really are sets:

  • Deduplicating a scraped corpus before you pay to embed it
  • Plagiarism and near-duplicate detection
  • Comparing tag sets or permission lists

Summary: cosine for ordinary semantic search, dot product when magnitude means something, Jaccard when you're comparing sets.

25Re-ranking & Cross-Encoders

Watch at 39:27
Slide: The second pass

Re-ranking is a second pass over your search results: a slower, smarter model re-scores the top candidates and reorders them before they reach the language model.

Why you need it: first-stage retrieval is built for speed. An embedding model encodes the query and the document separately, then compares vectors. It never looks at the two together.

A cross-encoder does exactly that. The query and document go in as one input, and the model reads them jointly, which catches relevance a vector distance misses.

Retrieve wide, rerank narrow: e.g. 50 candidates out of retrieval, and the reranker keeps the best five.

The trade-off: latency, because you're running a model over every candidate instead of one vector lookup. In practice I've used very small models for this, not even large language models, and you barely notice the latency. But it is one more system that can break, so it needs test cases and error handling.

In an interview: asked how you'd fix a RAG system that finds the right document but ranks it fourth? Re-ranking.

Builds on #20 Hybrid Search.

26Knowledge Graphs & Ontology

Watch at 40:52
Slide: A factory fire traced through three facts

A knowledge graph stores facts as triples: entity, relationship, entity. The ontology is the schema on top that says which entity types and relationships are allowed.

Example: a factory in Shenzhen burns down overnight. Which of your products are affected? Nobody has that written down. What you have are three boring facts, in different systems:

  • The Shenzhen plant supplies the M4 capacitor
  • The M4 capacitor is part of the audio board
  • The audio board is part of the Helio speaker

Every fact has the same three-part shape, so they link up into a graph. Follow the arrows and the answer falls out: the fire hits the Helio speaker. Nobody stored that fact directly; the graph chained it.

The ontology is the rulebook: a supplier supplies a component, a component is part of an assembly, an assembly is part of a product. It also stops bad data at the door. If a broken integration tries to write that a product supplies a factory, the write fails.

Why not just use vector search? Vector search finds documents that read like the question. It can't follow exact relationships across several hops, and it can't show you which facts produced the answer. A knowledge graph gives you an exact list, not a "probably", and agents can traverse it easily.

The trade-off: labour. Someone has to design the ontology and keep it current as the business changes. That's a permanent job.

Use a knowledge graph when your domain has real structure and the questions are about how things connect.

27Query Expansion & HyDE

Watch at 43:43
Slide: Expand and merge

Query expansion rewrites or broadens a user's query before you search with it, so you retrieve documents the raw wording would have missed.

People type badly. "k8s pod stuck" and "Kubernetes pod in pending state" are the same question, but keyword search hands back nothing for the first one.

Expand and merge:

  • Add synonyms or related terms (this needs a glossary that knows k8s means Kubernetes)
  • Or have a language model write three or four paraphrases of the query
  • Search with all of them and merge the results

HyDE (hypothetical document embeddings) is a neat variant: the model drafts a fake answer to the question, and you embed and search with that instead. An answer looks a lot more like other answers than a question does, and research shows this can noticeably improve accuracy.

The trade-off: drift. Expand too aggressively and you drag in loosely related junk that buries the one good hit. Keep the original query in the mix, cap the number of variants, and put a re-ranker behind it.

Builds on #25 Re-ranking & Cross-Encoders.

28RAG Chunking Strategies

Watch at 45:39
Slide: The chunking ladder

Chunking is how you cut documents into pieces before you embed them, and it's the highest-leverage decision in a retrieval pipeline.

A chunk is your unit of retrieval, so getting the size wrong hurts either way:

  • Too small: a sentence with no context. "This cut latency by 40%" is useless when you can't tell what "this" is.
  • Too large: you dilute the embedding, because one vector now has to stand for five unrelated ideas, and it becomes hard to match anything.

Get chunking wrong and no reranker, fancy index or bigger model will save you.

The usual ladder:

  1. Fixed-size chunks with some token overlap
  2. Recursive splitting that respects headings and paragraphs
  3. Semantic chunking that cuts where the topic actually shifts

The trade-off: there's no universal right size, and each step up the ladder costs more work at ingestion time. Measure retrieval quality on your own data before and after you change it.

29MCP (Model Context Protocol)

Watch at 46:45
Slide: What an MCP server declares

MCP is an open standard for connecting language models to tools and data, so any model can reach any tool through one shared interface.

"Why not just call the API?" You still do. The MCP server calls it for you. A model can't call an API itself; at best it writes out an instruction to call a function. So you write that function, a tool, and the glue around it: turning the API's schema into tool definitions the model understands, handling auth, pagination and rate limits, shaping errors into something the model can recover from, and truncating big responses.

The problem is you write that glue again inside every application that wants the same tool. Five apps and ten tools is fifty integrations. Put it behind an MCP server and you write it once. Every client gets it, including ones you didn't build. And because discovery is built in, you can add a tool on the server without anyone redeploying.

An MCP server declares three things:

  • Tools: functions the model can call
  • Resources: data at a URI the model can read
  • Prompts: reusable templates the user can invoke

Each one comes with a JSON schema and a description written for a model, so it's plug and play.

When it's not worth it: one application, two endpoints, you own both ends. Just call the API. MCP earns its keep when a capability needs to exist in more than one place, or in places you don't control.

Slide: Two kinds of agent memory

Agent memory is everything an agent keeps outside its context window, so it can carry knowledge across turns and sessions instead of forgetting you when a conversation ends.

The hard part isn't storing memories. It's forgetting them.

Two kinds:

  • Short-term: the conversation in the current context window. Cheap, exact, capped, and gone when the session closes.
  • Long-term: a store you deliberately write to and read from. It splits again into:
    • Episodic: things that happened. "The user deployed on Friday and it broke."
    • Semantic: durable facts. "The user's stack is Postgres and Go." (Nothing to do with semantic search.)

The trade-off: stale facts poison later runs, so you need decay, updates and conflict resolution. And memory you load back in takes up context, which can crowd out the instructions that matter.

In an interview: asked how you'd stop an agent acting on outdated information? That's your answer.

Slide: Cheap to know, expensive to load

An agent skill is a packaged folder of instructions, scripts and reference files that an agent loads on demand, only when a task calls for it.

The problem it solves: prompt bloat. You want your agent to handle PDFs, query BigQuery and follow your brand guidelines. You could stuff all of that into one giant system prompt, but every token competes for the model's attention, and most of it is irrelevant on any given turn.

Cheap to know, expensive to load:

  • A short description is visible to the agent all the time, just enough to know what the skill does and when to use it
  • The body is only read when the agent decides the skill matches the job in front of it

You only pay for a skill when it's relevant.

The trade-off: discoverability. If the description is vague, the agent never loads the skill, and you'll swear it's broken when it just never got picked. Write the description for the agent doing the choosing, not for yourself.

Use observability tools like LangSmith or PostHog to check that your agents actually load the skills they're supposed to.

32Context Engineering

Watch at 51:39
Slide: The context budget

Context engineering is deciding what occupies the model's context window at every step of an agent's run. It took over from prompt engineering once agents started running for hours instead of answering one question.

Why: a context window is finite, and it degrades as it fills. The model gets worse at recalling things buried in the middle, and worse at following instructions when they compete with a hundred thousand tokens of tool output.

The work is a budget:

  • Retrieval: pull in only what matters right now. Grep the three relevant pages out of a 200-page manual instead of loading all of it.
  • Compaction: replace a long history, like forty tool calls, with a short summary of what they found.
  • Knowing what to throw away.

How badly recall drops in the middle of a long window should guide when you compact and what you keep.

In an interview: asked how you'd keep an agent coherent over a long task? If you answer "a bigger context window", that's a red flag.

33Sub-Agents & Multi-Agent Orchestration

Watch at 52:47
Slide: Static vs dynamic orchestration

An orchestrator agent takes a goal, breaks it into sub-tasks, and hands each to a separate sub-agent with its own clean context window, then works with what comes back.

Why: if your main agent reads 50 files to answer one question, those 50 files live in its window for the rest of the run. Dispatch a sub-agent instead and you get back a two-paragraph answer. All the noise dies with the sub-agent, and the orchestrator never sees it.

Two flavors:

  • Static orchestration: sub-agents and roles fixed up front, like a researcher, a coder and a reviewer
  • Dynamic orchestration: the orchestrator spawns sub-agents at runtime based on what the task needs, and manages them through the run. I think this is the more common and useful one.

The trade-offs:

  • You pay for several context windows at once
  • Sub-agents can't see each other's work, by design, so they can duplicate effort or contradict each other
  • A lot more complexity

Get as far as you can with a single agent. Reach for sub-agents when one agent is doing too much and you're seeing context rot.

Slide: One turn of the agent loop

The harness is the program you write around the model. On its own, a model can't do anything: text in, text out. It can't open a file, hit an API, or remember what happened last turn. The harness makes all of that real.

One turn of the loop:

  1. The harness builds the prompt: system rules, the list of tools, and everything that's happened so far
  2. The model returns text, like "read file config.json"
  3. The harness parses it, checks it's a tool it can call, and checks the agent is allowed to read that file
  4. It opens the file and pastes the contents back into the context window
  5. It sends the whole conversation back, and the model takes its next step

The model never touches your file system; the harness does.

Why it matters: same model, two harnesses. One hands the tool error straight back, so the model reads "file not found", tries a different path, and recovers. The other swallows the error and returns an empty string, so the model assumes the file is empty and keeps building on nothing. Neither is a model problem.

There's a lot you can do to improve a harness before paying for a more expensive model. LangChain has published some great posts on harness engineering.

The trade-off: harness bugs look exactly like model failures. Before you swap models, log the raw tool results going back in.

35Chain of Thought Prompting

Watch at 56:52
Slide: Before and after chain of thought

Chain of thought is prompting a model to write out its intermediate reasoning before it commits to an answer.

Those reasoning tokens aren't decoration. Each one is fed back in as context, so generating them gives the model more computation to spend before the final answer.

Example: "A cafe has 23 apples. It uses 20, then buys 6 more. How many does it have?"

  • Without it: the model predicts the most likely-looking number based on its training data, and can easily get it wrong.
  • With it: it works through 23 − 20 = 3, then 3 + 6 = 9, and each step is easy.

Two caveats:

  • The written chain isn't an accurate account of what the model actually did internally. It can reach the right answer with visibly wrong reasoning.
  • Modern reasoning models do this built in, so adding "think step by step" to them adds tokens and buys you little.
Slide: Same question, two approaches

A ReAct agent alternates between thinking and doing. The name is reasoning + acting, and has nothing to do with the JavaScript framework.

Same question, two approaches: "Can I still get a refund on my last order?"

  • Chain of thought reasons in its head: most stores allow 30 days, the order was probably recent, so yes. Every step sounds sensible and none of it was checked.
  • A ReAct agent turns uncertainty into tool calls. It calls a tool to get the order (delivered 41 days ago), then looks up the refund policy (30 days from delivery), and answers no, which is correct.

The difference is whether each step gets checked against the real world.

Most agents today are ReAct agents under the hood, so you rarely build this yourself. But it's a likely interview question, so understand the architecture.

Builds on #35 Chain of Thought Prompting.

37Agentic Workflows

Watch at 59:29
Slide: Processing documents, two ways

Two ways to get from a goal to a finished task:

  • Fixed workflow: a predefined graph of steps you wrote in advance
  • Dynamic (agentic) workflow: a plan the model generates at runtime, then executes

Example, document processing:

  • Fixed: extract, classify, route, summarize, save. Every document walks the same path, whatever its format.
  • Dynamic: the model gets the goal and the tools and decides. A PDF goes extract, classify, save. An image needs OCR first. Different paths, same shape of output.

The benefit is obvious: the dynamic version handles the odd file you never anticipated.

The trade-off is the whole concept: predictability vs flexibility. Fixed workflows are testable, debuggable and cheap. Dynamic workflows handle new inputs, but two identical runs can take different paths, which makes them hard to test, eval and bill for, and they can introduce new error cases.

Start fixed, and only go dynamic where you need it.

Slide: What gets written

An LLM wiki is an internal knowledge base that the model itself writes and maintains. RAG searches the documents you already have; an LLM wiki is the agent writing the ones you don't.

The problem it solves: amnesia. An agent spends an hour learning that the staging database needs a VPN or that one test is flaky. Then the session ends and tomorrow it learns it all again.

What gets written:

  • Entity pages: one per person, company or customer, with the short facts about them
  • Concept pages: built on top, explaining how entities connect. A page for a product launch links to the people who lead it and the customers rolling it out.

Before the next task, the agent reads the relevant concept page and follows the links to what it needs. It ends up working like a graph, written at ingestion time, which makes retrieval fast.

The trade-off: maintenance and rot. Ingesting a new document means updating every page it touches, and a model-maintained wiki quietly accumulates stale and contradictory entries. Date entries, keep them short, and prune them.

39Agent Sandboxes

Watch at 1:02:14
Slide: Isolated by default

An agent sandbox is an isolated environment (a container, VM or micro-VM) where code the agent writes actually runs, so nothing it does can touch your real machine.

Why: the model isn't malicious, but it will confidently run a recursive delete in the wrong directory, or execute a command it read on a web page that was really a prompt injection. Inside a sandbox, the worst case is you throw the box away and start again.

Isolated by default:

  • Each agent session gets its own pod or micro-VM, with its own file system
  • Copy the files it needs in when the sandbox starts
  • A network policy, often no network at all
  • A hard timeout
  • Nothing shared and nothing persistent unless you deliberately mount it

This matters most when the agent works on customer data. You don't want it dropping tables it shouldn't touch.

The trade-off: latency and friction. Spinning up an environment per session costs compute, it's more to maintain, and the agent loses access to things it legitimately needs, so grant those back one at a time.

40A2A (Agent-to-Agent Protocol)

Watch at 1:03:48
Slide: The agent card

A2A is an open standard, started by Google, for agents to discover each other and hand off work.

A2A vs MCP, the comparison you'll get asked about: MCP connects a model to tools. A2A connects agents to other agents, as peers, across company boundaries. Complementary, not competing.

How it works:

  • Every agent publishes an agent card: a small JSON document saying what it is, what it can do, and how to authenticate with it
  • A client agent reads the card and decides whether to hand off a task
  • Tasks have a lifecycle (submitted, working, completed), so a long job doesn't have to finish in one request

Example: your travel-planning agent needs a flight booked. It reads the airline agent's card, sees it can book flights, and hands it the task. Neither side built a custom integration.

The trade-off: every hand-off adds discovery, authentication and a task to track, and it can be hard to follow a task across agents. Adoption is still early. Inside one team, a plain sub-agent call is simpler; A2A earns its keep across organizational boundaries.

Builds on #29 MCP (Model Context Protocol).

41Structured Outputs & Constrained Decoding

Watch at 1:05:16
Slide: How the schema is enforced

Structured outputs force a model to return data in a shape you define, usually a JSON schema, instead of free text.

With constrained decoding, invalid JSON is never a possible output. No parser, retry loop or repair step for when the model wraps the JSON in a code fence or invents a field.

How the schema is enforced:

  1. The schema is compiled into a grammar, a state machine that knows which tokens are allowed next
  2. The model scores every token as normal
  3. Every token that would break the schema is masked out
  4. The model picks the most likely token that's left, the grammar moves one step, repeat

If the output so far is {"name": "Mug", "price":, only number tokens are allowed next.

Worth knowing:

  • It's different from JSON mode, which only promises some valid JSON, not your schema
  • It guarantees structure, not truth: the price is always a number, but it can be the wrong number

The trade-off: a tight schema can make the model dumber, because it commits before it thinks. "I was charged twice and now the app won't open", with category as the first field, might just get "bug". Fix: put a reasoning field first. The model fills fields top to bottom, so it reasons ("double charge plus a crash, money first") before it picks "billing". Your code ignores the reasoning.

Use structured outputs whenever model output feeds another program.

42Tree of Thoughts

Watch at 1:07:19
Slide: Branch and search

Tree of thoughts is a prompting technique where the model explores several reasoning paths, scores them, and keeps only the promising ones.

Chain of thought is a straight line: if step two is wrong, everything after it is wrong too.

Tree of thoughts branches instead:

  • At each step, generate a few candidate next steps
  • Evaluate them, usually by asking the model how promising each one looks
  • Keep the best, prune the rest, and backtrack out of dead ends

Example, the Game of 24 from the paper: use 4, 9, 10 and 13 to make 24. The model proposes three first moves: 13 − 9 looks sure, 10 − 4 is a maybe, 4 + 9 is impossible and gets pruned. It expands the best branch, hits a dead end, backs up, tries 10 − 4, and finds (13 − 9) × (10 − 4) = 24.

The trade-off: cost. Many more calls, much more latency and many more tokens for one answer.

In an interview: asked what separates this from chain of thought? The word they want is search.

Builds on #35 Chain of Thought Prompting.

43Prompt Compression

Watch at 1:08:43
Slide: Three compression methods

Prompt compression means shrinking what you send to the model before you send it, so you pay for fewer input tokens while keeping what the model needs.

Three methods:

  • Token pruning: a small, cheap model scores every token in the input and drops the ones carrying little information
  • Summarization: replace older conversation history with a short recap
  • Retrieval: never build the big prompt in the first place. Instead of pasting a 200-page policy manual, retrieve the three or four chunks about refunds

Done well, a 4,000-token prompt drops to about 800 and the answer doesn't change. It's cheaper and faster.

The trade-off: it's lossy, and you often can't tell what you threw away until one specific question starts failing. With retrieval the risk moves: you can fetch the wrong chunks.

Compress the stable, repetitive parts, like chat history and boilerplate instructions. Leave the user's actual question untouched.

44LLM Observability & Tracing (OpenTelemetry)

Watch at 1:09:53
Slide: One trace

Observability for an LLM app means being able to answer "why did this response happen?" after the fact. Open the trace for a bad response and you immediately see whether retrieval missed, or the model ignored good context.

How a trace is built: every step of a request is recorded as a span, and spans nest to form one trace. For one user question:

  • The root span is the request itself
  • A retrieval span holds the query and the chunks it returned
  • An LLM span has the exact prompt, model, token counts, latency and cost
  • A tool call span
  • A second LLM span for the final answer

OpenTelemetry (OTel) is the open standard these spans are emitted in. Platforms like LangSmith and PostHog speak it, so you can switch backends easily.

The trade-off: traces contain full prompts and user data, so storage and privacy need thought. Add tracing on day one anyway; retrofitting it once you have unhappy users is miserable.

45Precision vs Recall

Watch at 1:10:53
Slide: The pull between precision and recall

Two numbers that tell you how good a retrieval or classification system really is. (I mix them up all the time, so do as I say, not as I do.)

  • Precision: of everything you returned, how much was correct?
  • Recall: of everything that was correct, how much did you return?

They pull against each other. Return more documents and recall climbs while precision drops. Return only what you're certain about and it goes the other way.

Which one to optimize depends on what being wrong costs. In RAG you usually chase recall first, because a reranker can throw out junk but it can't rescue a document you never retrieved.

In an interview: mixing these two up is an instant red flag.

Builds on #25 Re-ranking & Cross-Encoders.

46Golden Datasets

Watch at 1:11:48
Slide: What's in a golden dataset

A golden dataset is a curated set of example inputs with outputs a human has signed off as correct. It's the ground truth your system is measured against every time you change something.

You don't need thousands. A hundred well-chosen rows will catch more regressions than ten thousand random ones.

What's in it:

  • Real examples, pulled from production logs or assembled by a person
  • Edge cases: ambiguous questions, and questions where saying no is the correct behavior
  • Columns: the question, the expected answer (or expected documents), and a short note on why it's right

The trade-offs:

  • Cost. Good labels need people, sometimes full-time. When I had dinner with the team at OpenAI and with Tandem Health, both told me they pay in-house industry experts to label data and build golden datasets. We started doing the same at my company and it's given us great results.
  • Maintenance. It goes stale as your product changes, and if you tune against the same rows forever you'll overfit. Refresh it, and hold back a slice you never tune on.

47LLM Evals & LLM-as-a-Judge

Watch at 1:13:01
Slide: Three layers of evals

LLM evals systematically score model output, so you know whether a change made your system better or worse instead of guessing from a few prompts you tried by hand.

Three layers:

  1. Deterministic checks: valid JSON? Called the right tool? Contains the required order number? These are just unit tests.
  2. Similarity scoring against a reference answer
  3. LLM-as-a-judge: a second model grades the output against a written rubric, for things code can't check, like tone or whether the answer is grounded in the retrieved context. It gets the question, context, answer and rubric, and returns a score plus a reason.

Without a reference answer, the judge decides on its own whether the answer is right.

Watch out: judges are biased. They favor longer answers and models from their own family, and scores shift when you swap the judge model.

The trade-off: a judge is fast, cheap and scales, but it's often less reliable than a human labeller. Check your judge against human labels before you trust its numbers. That's why companies like OpenAI keep hiring industry experts to label data and build evals.

Builds on #46 Golden Datasets.

48Forward Deployed Engineer (FDE)

Watch at 1:14:38
Slide: From demo to deployment

A forward deployed engineer works embedded with a customer, in their environment and on their data, solving their specific problem. The role was made famous by Palantir, and it's one of the fastest-growing roles in tech right now. I'm the lead FDE at my company, and I think it's a really fun job.

Why the role exists: an AI demo and an AI deployment are very different things. The demo runs on clean sample data, follows one happy path, and looks finished. At a real customer, the data is a mess, the systems are 20 years old, every team works a bit differently, and nobody agrees on what a correct answer is. The model isn't the hard part anymore. Fitting it into the company is.

What an FDE does:

  • Sits with the people doing the work and watches how it's actually done
  • Wires the model into their data, tools and old systems, usually most of the engineering
  • Defines what correct means, turning their real cases into evals, so you can tell whether you're improving or regressing
  • Reports back: when the same fix keeps coming up across customers, it becomes a feature in the core product

This matters more in the AI era, where building is cheap and fast but adoption is harder than ever.

The trade-off: you can end up with a lot of code serving exactly one customer and drift into being a services shop. Knowing when to push something into the core product, and when to say no, is the skill.

49Claude Artifacts

Watch at 1:17:03
Slide: The artifact loop

An artifact is a self-contained generated output, like a working app, a document, a chart or a diagram, rendered in its own window beside the chat instead of in a code block you have to copy out. (Gemini has something similar called Canvas.)

The loop: chat on the left, artifact on the right, running. Ask for a change and you get a new version, with the previous one still there to roll back to. You can even comment on a specific part of it, and that part is used as context. The feedback loop collapses: look at it, "make the button smaller", look again.

I use it all the time at work, to understand complex, abstract things visually and to share things with the rest of my team.

The limits: it runs in a sandbox, so no arbitrary network access. And a three-line answer doesn't belong in an artifact. Use it when the output is a thing, not an answer.

In an interview: a great example for "how do you use AI in your development?" or "how do you share knowledge with your team?"

50Git Worktrees for AI Coding Agents

Watch at 1:18:27
Slide: One history, many trees

A git worktree lets you check out several branches of the same repository at once, each in its own directory, all sharing one git folder and one commit history.

Normally one clone gives you exactly one working tree, so moving between branches means stashing, switching and waiting.

git worktree add points at a new directory and a branch, and you get a second checkout next to the first. Nothing is copied twice, and both share the same history.

Why it belongs in an AI video: parallel agents. Give each coding agent its own worktree on its own branch and they can edit, test and commit at the same time without stomping on each other's files. Run the big task in one, and ship small bug fixes and UI tweaks in others while you wait.

I take it further with what I call dev trees: each worktree spins up its own Docker containers for the backend and frontend, so I can test every branch in the UI side by side.

The trade-offs:

  • Parallel agents don't know what the others are doing, so you can get double work
  • Every worktree needs its own dependencies and build output, so disk use adds up
  • Stale worktrees (and their running containers) need cleaning up, or your machine will suffer

Know the concepts, but not sure how you come across in the interview?

I run mock interviews the way I ran them as a hiring engineer, then tell you exactly what would get you a yes.

Book a mock interview 

The Human Loop is my newsletter: what I learn day to day as a senior AI engineer. You'll also be first to know when the learning hub launches.