Hands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set — model APIs, structured output, tool calling, RAG, evals-as-the-spine, agents (loop from scratch, tool design, guardrails, MCP, Skills), fine-tuning vs LoRA, prompt-injection/security, LLMOps, and customer craft. Runs on the free Groq API.
620
stars
86
commits
Jupyter Notebook
primary language
Aug 28, 2026
updated
README
AI Engineer Notebooks
Learn the applied-LLM stack the way you'll actually be interviewed on it: framework-free, on a free API, from prompting all the way to serving, fine-tuning, and a red-team benchmark.
Runnable Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set. You build working systems on top of foundation models (model APIs, RAG, evals, agents, adaptation, serving) using raw APIs, not frameworks.
What makes this different
- Framework-free, on purpose. You write the agent loop, RAG, and evals from raw API calls first, so you understand what LangChain/LlamaIndex actually do before you reach for them (and can judge when not to). Patterns are durable; wrappers churn.
- Evals are the spine. "Measure before you tune" is installed early and returns in every section. It's the habit that separates an engineer who shipped a system from one who built a demo.
- Free to run, end to end. Everything runs on the free Groq API (no credit card). The two topics Groq can't host, LoRA fine-tuning (06) and self-hosted serving (09), are concept-first with optional, fenced Colab-GPU appendices that were verified on a real Colab T4.
- Real case studies, not toy demos. Three end-to-end case studies show the skills combined under real constraints: a support assistant debugged in production, a pipeline-vs-agent cost showdown, and a red-team robustness benchmark.
- OpenAI-compatible throughout, so every pattern transfers directly to OpenAI and (with small changes) Anthropic. Swap the base URL and the skills carry over.
Built as the hands-on companion to Plan: Transitioning to Forward Deployed Engineer / AI Engineer. The plan explains what to learn and why; these notebooks are where you run it.
Who this is for
Backend or full-stack engineers moving into AI Engineer, FDE, Applied AI, or Solutions Engineer (AI) roles. Different titles, largely the same job. You can ship production code; you want the applied-model layer on top.
Learning order
Work top to bottom. Each notebook is self-contained (installs its own dependencies, reads API keys from Colab secrets) and ends with exercises.
00 — Setup
| Notebook | What you'll learn |
|---|---|
| Environment & cost hygiene | API keys via Colab secrets, spend guards, model picking |
01 — Model APIs
| Notebook | What you'll learn |
|---|---|
| Prompting fundamentals | Clear instructions, few-shot, output-format specs, step-by-step reasoning: the cheapest lever, each shown moving a number |
| Structured output | Getting reliable JSON out of a model, and where it breaks |
| Tool calling | Function/tool calling end to end, error paths included |
| Streaming | Streaming responses and what UIs need from them |
| Context & caching | Context-window budgeting, prompt caching, batch vs real-time pricing |
02 — Evals I: measuring outputs
| Notebook | What you'll learn |
|---|---|
| Measuring outputs | Golden sets and metrics on the section-01 task; install the "measure before you tune" habit before building anything you'd need to tune. Evals is the spine; it returns in every section after this |
03 — RAG
| Notebook | What you'll learn |
|---|---|
| What is RAG? | The retrieve → augment → generate loop, why RAG beats a plain LLM, and why RAG isn't the same as embeddings, with a 15-line working demo |
| Embeddings & retrieval | Embedding choice, vector search, similarity pitfalls. Get retrieval working first |
| Hybrid & reranking | Keyword + vector hybrid retrieval, rerankers, when each earns its cost |
| Chunking | Chunking strategies on a real messy corpus, revisited last once you can judge them against retrieval |
| Why RAG fails | Diagnosing bad answers: retrieval quality, not generation, is usually the bottleneck |
04 — Evals II: the differentiator
| Notebook | What you'll learn |
|---|---|
| Golden sets | Building a golden set for the RAG system from section 03 |
| LLM as judge | Judge prompts, agreement with humans, and the judge's own failure modes |
| Regression evals | Evals as CI: catching quality regressions when you change a prompt or model |
05 — Agents
| Notebook | What you'll learn |
|---|---|
| Agent loop from scratch | A working agent loop in raw API calls, no framework |
| Tool design | Designing tools the model can actually use well |
| Guardrails & budgets | Stopping conditions, cost/latency budgets, when a pipeline beats an agent |
| MCP & the tool ecosystem | Concept: what the Model Context Protocol standardizes, how it maps to the raw tool loop, and when to reach for it |
| Skills & progressive disclosure | Concept: packaging reusable know-how an agent loads on demand. The SKILL.md pattern, the context-budget payoff, and Tools/MCP/Skills as one story |
| Harness engineering | Synthesis: the scaffold around the call: context assembly & compaction, tool-result shaping, and verification loops. Names the discipline the section has been teaching piece by piece |
06 — Adapting the model
| Notebook | What you'll learn |
|---|---|
| Fine-tune vs RAG vs prompt | When to change the model's weights vs its inputs; what LoRA/QLoRA are and cost; the argument you'll have in the room, plus an optional real LoRA fine-tune on a free GPU |
07 — Security
| Notebook | What you'll learn |
|---|---|
| Prompt injection & the trust boundary | Direct & indirect prompt injection, output handling, PII, excessive agency. The OWASP LLM Top 10 risks, failing live then defended |
08 — Operations
| Notebook | What you'll learn |
|---|---|
| Observability & LLMOps | Tracing every call, safe prompt logging, cost/latency/error metrics, drift detection, and the observe→eval feedback loop |
| Reliability & fallbacks | Retries with backoff, timeouts, fallback models, output validation, circuit breakers, graceful degradation |
| Experiment tracking & registry | MLflow end to end: log runs/params/metrics from the section-04 eval harness, register and version a model, and promote by stage: the tooling that turns "I ran an eval" into a tracked, reproducible workflow |
09 — Serving & inference performance
Where the free Groq API can't run the topic (these frameworks need a GPU), the notebook teaches it concept-first and fences an optional Colab-GPU appendix, the same pattern as the section-06 LoRA appendix.
| Notebook | What you'll learn |
|---|---|
| Serving frameworks | The serving stack an AI engineer actually picks between (vLLM, TGI, Triton, TensorRT-LLM): what each optimizes, how they map onto the raw API you've been calling, and when to reach for which |
| Inference performance | The levers behind throughput and latency: continuous batching, the KV cache, quantization, and the throughput-vs-latency trade, with the napkin math to size a deployment |
10 — ML system design & performance
| Notebook | What you'll learn |
|---|---|
| Designing an inference service | Concept: the ML system design interview, worked end to end: QPS/VRAM/latency/cost estimation, replica scaling, queueing, caching, and the SLA trade-offs, on a realistic LLM-serving prompt |
11 — Customer craft (the FDE differentiator)
| Notebook | What you'll learn |
|---|---|
| Scoping & discovery | Turn a vague customer ask into a scoped, evaluable system: discovery questions, a one-page scoping doc, the demo discipline: the customer-scenario interview round most engineers can't evidence |
12 — Case Studies & Capstone
Where the skills come together into projects. First a case study (one realistic scenario worked end to end, runnable), then the capstone: the deployed repo you build yourself. (Section overview.)
| Notebook | What you'll learn |
|---|---|
| Case study A — Customer-support assistant | One scenario scoped → built → served → debugged in production: a vague ask becomes a deployed, evaluated RAG+agent assistant, then a live quality regression (a stale index after a corpus migration) that you diagnose and fix. A build-to-debug arc threading sections 02–11 |
| Case study B — Contract extraction: pipeline vs agent | The judgment call interviewers love: build the same extraction task as both an agent and a pipeline, then prove with accuracy + token cost that the pipeline wins when the steps are known |
| Case study C — Red-team robustness benchmark | A different kind of system, a harness that evaluates a model instead of serving one: an attacker→target→judge (PAIR) loop that measures attack success rate, composing the agent loop, LLM-judge, security, and evals |
Capstone: the brief for the deployed project that goes on your resume, a real repo with a serving component and an eval report. Case studies are for learning; the capstone is for hiring.
Conventions
- Raw model APIs, no frameworks. Patterns are durable; wrappers churn.
- One shared corpus (
data/) across RAG and eval sections, so evals measure the retrieval you actually built. - Self-contained notebooks. First cell installs, second cell calls
from aien import setup; client, MODEL = setup()to load your key from Colab secrets (or a local env var). No hidden state between notebooks.aienis the tiny shared-setup package in this repo (one place to change credential loading), installed automatically by the first cell. - Every notebook ends with exercises. Do them before moving on.
Setup
- Get a free API key at console.groq.com, no credit card required.
- In Colab: the key icon in the left sidebar → add
GROQ_API_KEYas a secret, and toggle notebook access on. - Open any notebook via its badge and run top to bottom.
Running locally instead: pip install -r requirements.txt && pip install -e .
(the second installs the aien setup helper), export GROQ_API_KEY=...,
open with Jupyter.
Related reading
- Plan: Transitioning to FDE / AI Engineer is the roadmap these notebooks implement
- Guide: Building a Real LLM Project for Your Resume sets the capstone's requirements bar
- Walkthrough: Designing a RAG System covers the systems view of section 03
- Walkthrough: Designing an AI Agent Orchestration System covers the systems view of section 05
Contributors
86 commits
See what people are saying
Similar repos
Languages
Jupyter Notebook
99.6%
Hands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set — model APIs, structured output, tool calling, RAG, evals-as-the-spine, agents (loop from scratch, tool design, guardrails, MCP, Skills), fine-tuning vs LoRA, prompt-injection/security, LLMOps, and customer craft. Runs on the free Groq API.
620
stars
86
commits
Jupyter Notebook
primary language
Aug 28, 2026
updated
README
AI Engineer Notebooks
Learn the applied-LLM stack the way you'll actually be interviewed on it: framework-free, on a free API, from prompting all the way to serving, fine-tuning, and a red-team benchmark.
Runnable Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set. You build working systems on top of foundation models (model APIs, RAG, evals, agents, adaptation, serving) using raw APIs, not frameworks.
What makes this different
- Framework-free, on purpose. You write the agent loop, RAG, and evals from raw API calls first, so you understand what LangChain/LlamaIndex actually do before you reach for them (and can judge when not to). Patterns are durable; wrappers churn.
- Evals are the spine. "Measure before you tune" is installed early and returns in every section. It's the habit that separates an engineer who shipped a system from one who built a demo.
- Free to run, end to end. Everything runs on the free Groq API (no credit card). The two topics Groq can't host, LoRA fine-tuning (06) and self-hosted serving (09), are concept-first with optional, fenced Colab-GPU appendices that were verified on a real Colab T4.
- Real case studies, not toy demos. Three end-to-end case studies show the skills combined under real constraints: a support assistant debugged in production, a pipeline-vs-agent cost showdown, and a red-team robustness benchmark.
- OpenAI-compatible throughout, so every pattern transfers directly to OpenAI and (with small changes) Anthropic. Swap the base URL and the skills carry over.
Built as the hands-on companion to Plan: Transitioning to Forward Deployed Engineer / AI Engineer. The plan explains what to learn and why; these notebooks are where you run it.
Who this is for
Backend or full-stack engineers moving into AI Engineer, FDE, Applied AI, or Solutions Engineer (AI) roles. Different titles, largely the same job. You can ship production code; you want the applied-model layer on top.
Learning order
Work top to bottom. Each notebook is self-contained (installs its own dependencies, reads API keys from Colab secrets) and ends with exercises.
00 — Setup
| Notebook | What you'll learn |
|---|---|
| Environment & cost hygiene | API keys via Colab secrets, spend guards, model picking |
01 — Model APIs
| Notebook | What you'll learn |
|---|---|
| Prompting fundamentals | Clear instructions, few-shot, output-format specs, step-by-step reasoning: the cheapest lever, each shown moving a number |
| Structured output | Getting reliable JSON out of a model, and where it breaks |
| Tool calling | Function/tool calling end to end, error paths included |
| Streaming | Streaming responses and what UIs need from them |
| Context & caching | Context-window budgeting, prompt caching, batch vs real-time pricing |
02 — Evals I: measuring outputs
| Notebook | What you'll learn |
|---|---|
| Measuring outputs | Golden sets and metrics on the section-01 task; install the "measure before you tune" habit before building anything you'd need to tune. Evals is the spine; it returns in every section after this |
03 — RAG
| Notebook | What you'll learn |
|---|---|
| What is RAG? | The retrieve → augment → generate loop, why RAG beats a plain LLM, and why RAG isn't the same as embeddings, with a 15-line working demo |
| Embeddings & retrieval | Embedding choice, vector search, similarity pitfalls. Get retrieval working first |
| Hybrid & reranking | Keyword + vector hybrid retrieval, rerankers, when each earns its cost |
| Chunking | Chunking strategies on a real messy corpus, revisited last once you can judge them against retrieval |
| Why RAG fails | Diagnosing bad answers: retrieval quality, not generation, is usually the bottleneck |
04 — Evals II: the differentiator
| Notebook | What you'll learn |
|---|---|
| Golden sets | Building a golden set for the RAG system from section 03 |
| LLM as judge | Judge prompts, agreement with humans, and the judge's own failure modes |
| Regression evals | Evals as CI: catching quality regressions when you change a prompt or model |
05 — Agents
| Notebook | What you'll learn |
|---|---|
| Agent loop from scratch | A working agent loop in raw API calls, no framework |
| Tool design | Designing tools the model can actually use well |
| Guardrails & budgets | Stopping conditions, cost/latency budgets, when a pipeline beats an agent |
| MCP & the tool ecosystem | Concept: what the Model Context Protocol standardizes, how it maps to the raw tool loop, and when to reach for it |
| Skills & progressive disclosure | Concept: packaging reusable know-how an agent loads on demand. The SKILL.md pattern, the context-budget payoff, and Tools/MCP/Skills as one story |
| Harness engineering | Synthesis: the scaffold around the call: context assembly & compaction, tool-result shaping, and verification loops. Names the discipline the section has been teaching piece by piece |
06 — Adapting the model
| Notebook | What you'll learn |
|---|---|
| Fine-tune vs RAG vs prompt | When to change the model's weights vs its inputs; what LoRA/QLoRA are and cost; the argument you'll have in the room, plus an optional real LoRA fine-tune on a free GPU |
07 — Security
| Notebook | What you'll learn |
|---|---|
| Prompt injection & the trust boundary | Direct & indirect prompt injection, output handling, PII, excessive agency. The OWASP LLM Top 10 risks, failing live then defended |
08 — Operations
| Notebook | What you'll learn |
|---|---|
| Observability & LLMOps | Tracing every call, safe prompt logging, cost/latency/error metrics, drift detection, and the observe→eval feedback loop |
| Reliability & fallbacks | Retries with backoff, timeouts, fallback models, output validation, circuit breakers, graceful degradation |
| Experiment tracking & registry | MLflow end to end: log runs/params/metrics from the section-04 eval harness, register and version a model, and promote by stage: the tooling that turns "I ran an eval" into a tracked, reproducible workflow |
09 — Serving & inference performance
Where the free Groq API can't run the topic (these frameworks need a GPU), the notebook teaches it concept-first and fences an optional Colab-GPU appendix, the same pattern as the section-06 LoRA appendix.
| Notebook | What you'll learn |
|---|---|
| Serving frameworks | The serving stack an AI engineer actually picks between (vLLM, TGI, Triton, TensorRT-LLM): what each optimizes, how they map onto the raw API you've been calling, and when to reach for which |
| Inference performance | The levers behind throughput and latency: continuous batching, the KV cache, quantization, and the throughput-vs-latency trade, with the napkin math to size a deployment |
10 — ML system design & performance
| Notebook | What you'll learn |
|---|---|
| Designing an inference service | Concept: the ML system design interview, worked end to end: QPS/VRAM/latency/cost estimation, replica scaling, queueing, caching, and the SLA trade-offs, on a realistic LLM-serving prompt |
11 — Customer craft (the FDE differentiator)
| Notebook | What you'll learn |
|---|---|
| Scoping & discovery | Turn a vague customer ask into a scoped, evaluable system: discovery questions, a one-page scoping doc, the demo discipline: the customer-scenario interview round most engineers can't evidence |
12 — Case Studies & Capstone
Where the skills come together into projects. First a case study (one realistic scenario worked end to end, runnable), then the capstone: the deployed repo you build yourself. (Section overview.)
| Notebook | What you'll learn |
|---|---|
| Case study A — Customer-support assistant | One scenario scoped → built → served → debugged in production: a vague ask becomes a deployed, evaluated RAG+agent assistant, then a live quality regression (a stale index after a corpus migration) that you diagnose and fix. A build-to-debug arc threading sections 02–11 |
| Case study B — Contract extraction: pipeline vs agent | The judgment call interviewers love: build the same extraction task as both an agent and a pipeline, then prove with accuracy + token cost that the pipeline wins when the steps are known |
| Case study C — Red-team robustness benchmark | A different kind of system, a harness that evaluates a model instead of serving one: an attacker→target→judge (PAIR) loop that measures attack success rate, composing the agent loop, LLM-judge, security, and evals |
Capstone: the brief for the deployed project that goes on your resume, a real repo with a serving component and an eval report. Case studies are for learning; the capstone is for hiring.
Conventions
- Raw model APIs, no frameworks. Patterns are durable; wrappers churn.
- One shared corpus (
data/) across RAG and eval sections, so evals measure the retrieval you actually built. - Self-contained notebooks. First cell installs, second cell calls
from aien import setup; client, MODEL = setup()to load your key from Colab secrets (or a local env var). No hidden state between notebooks.aienis the tiny shared-setup package in this repo (one place to change credential loading), installed automatically by the first cell. - Every notebook ends with exercises. Do them before moving on.
Setup
- Get a free API key at console.groq.com, no credit card required.
- In Colab: the key icon in the left sidebar → add
GROQ_API_KEYas a secret, and toggle notebook access on. - Open any notebook via its badge and run top to bottom.
Running locally instead: pip install -r requirements.txt && pip install -e .
(the second installs the aien setup helper), export GROQ_API_KEY=...,
open with Jupyter.
Related reading
- Plan: Transitioning to FDE / AI Engineer is the roadmap these notebooks implement
- Guide: Building a Real LLM Project for Your Resume sets the capstone's requirements bar
- Walkthrough: Designing a RAG System covers the systems view of section 03
- Walkthrough: Designing an AI Agent Orchestration System covers the systems view of section 05
See what people are saying
Similar repos
Contributors
86 commits
Languages
Jupyter Notebook
99.6%