AI Engineer Portal
Your personal operating system for career transition.
Practice
Hands-on drills tuned for AI engineering.
Build implementation muscle memory. Each exercise includes starter code, a reference solution, and AI-powered feedback.
Agents (8)
Build a tool registry with schema validation
In any agent system, the model needs to know which tools are available, what arguments each tool expects, and how to...
Implement a ReAct reasoning loop
The ReAct (Reasoning + Acting) pattern is the most widely used agent loop in production. The model alternates between...
Add retry and fallback logic to tool calls
In production agent systems, tool calls fail. APIs time out, rate limits hit, services go down. A resilient agent does...
Parse structured outputs with error recovery
LLMs are asked to produce JSON constantly — tool call arguments, structured answers, classification labels, extraction...
Conversation memory manager with token budgeting
Agents that run multi-turn conversations or long-running tasks quickly blow through context windows. A memory manager...
Multi-agent handoff protocol
As agent systems grow, a single monolithic agent becomes unwieldy. The multi-agent pattern splits work among specialist...
Agent evaluation harness
Building an agent is one thing; knowing whether it actually works is another. An **evaluation harness** runs an agent...
Cost-tracking middleware for agent tool calls
Agent systems can burn through API budgets fast. Every LLM call, every tool invocation, every retry costs tokens and...
Api Async (23)
Retry a provider call with timeout and backoff
Implement an async wrapper that retries retryable provider failures with a timeout and bounded backoff.
Enforce provider timeout defaults from env config
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Batch provider requests with concurrency limits
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Build an async health check for provider adapters
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Throttle background ingestion tasks with backpressure
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Retry only idempotent ingestion steps
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Wrap streaming provider responses into stable chunks
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Implement bounded polling for long-running jobs
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Add health and readiness probes to an LLM service
Kubernetes and cloud load balancers use two distinct probes to manage container lifecycle:
Enforce per-request token budgets
LLM API costs are linear with token usage. Without enforcement, a single user pasting a 50,000-token document into a...
Build a rate limiter for AI API endpoints
LLM API calls are expensive and slow. Without rate limiting at your application layer, a single misbehaving client can...
Implement graceful degradation with fallback tiers
Production AI services fail in partial ways: the primary LLM provider might be rate-limited, the secondary might be...
Generate Kubernetes manifests for an LLM service
Deploying an LLM service to Kubernetes requires careful manifest design. The key differences from a typical web service:
Build a health check endpoint for an LLM service
A production LLM service needs two health endpoints with different semantics. A naive `/ping -> 200` is useless — it...
Implement token-based rate limiting for an LLM API
Request-count rate limiting is insufficient for LLM services. A user sending one 100,000-token request consumes the...
Build a cost monitoring middleware
Without per-request cost tracking, you discover cost problems on the monthly invoice. A cost monitoring middleware...
Implement distributed tracing spans for LLM calls
Standard application tracing shows HTTP latency. For LLM pipelines you need per-span visibility: retrieval, prompt...
Implement graceful degradation with multi-level fallback tiers
When the primary LLM provider goes down, the options are: fail hard, use cached responses, try a backup model, or...
Build a persistent semantic cache
An in-memory semantic cache loses all entries on restart. A production semantic cache needs persistence through a...
Async provider client with semaphore rate limiting
Running LLM calls serially on a batch is 10–50x slower than running them concurrently. But unlimited concurrency...
Track token usage and cost per model across async calls
Cost visibility is the first step to cost control. This exercise builds a thread-safe cost tracker that can be used as...
Build an async model fallback chain with latency budget
Provider reliability is imperfect. A fallback chain means one provider failing does not make your feature fail. A...
Parse and validate LLM JSON output with extraction fallback
LLMs return almost-JSON. Your parser needs to handle: raw JSON, markdown-fenced JSON, JSON embedded in prose, and...
Data Transformation (16)
Aggregate model latency metrics from JSONL traces
Given a JSONL file of request traces, compute per-model latency averages, P95, and failure counts.
Summarize token usage by endpoint and model
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Validate article metadata before publish
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Group API errors into actionable retry buckets
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Clean scraped text before chunking
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Compute rolling latency stats for model routes
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Prepare dashboard series from daily usage rows
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Reconcile mismatched event timestamps
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Transform JSONL trace logs into aggregated metrics
AI services emit structured logs for every LLM call: model used, input/output token counts, latency, feature name, and...
Build a data pipeline with generator composition
Document ingestion pipelines need to process large corpora efficiently. Loading an entire dataset into memory before...
Implement batch processing with rate limiting
Provider APIs enforce rate limits in two dimensions: requests per minute (RPM) and tokens per minute (TPM). Exceeding...
Build an error classification and retry policy engine
AI service calls fail in distinct categories, and each category calls for a different response. A 429 rate limit error...
Build a streaming response accumulator for LLM output
Most production LLM APIs support streaming: instead of waiting for the full response, you receive tokens as they are...
Compute per-model cost and usage summaries from request logs
Every AI service needs basic cost visibility: which models are being used, how many tokens are flowing through each...
Build a document chunking pipeline with overlap and metadata
Before a document can be embedded and indexed for RAG, it must be split into chunks that fit within the embedding...
Implement a JSONL ingestion pipeline with error isolation
JSONL (JSON Lines) is the standard format for AI datasets, evaluation sets, and training data. An ingestion pipeline...
Deployment (6)
Write a multi-stage Dockerfile for an LLM application
Write a multi-stage Dockerfile for a Python LLM application that:
Build a request queue with priority routing
Implement an in-memory `PriorityInferenceQueue` that:
Implement a semantic cache for LLM responses
Implement a `SemanticCache` class that caches LLM responses keyed by semantic similarity of the query embedding:
Create an eval gate for CI/CD pipelines
Implement a `EvalGate` class that can be used as a quality gate in a CI/CD pipeline:
Build a multi-provider failover chain
Implement a `ProviderFailoverChain` that:
Implement a cost-aware model router
Implement a `CostAwareRouter` that selects the cheapest model capable of handling a request:
Evaluation (36)
Compute faithfulness and citation coverage from judge results
Take evaluation outputs and summarize pass rate, faithfulness score, and citation coverage per prompt version.
Compute confusion matrix from classifier review logs
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Compare benchmark runs and highlight regressions
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Score prompt variants by pass rate and cost
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Audit benchmark failures by category
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Build a review queue from low-confidence outputs
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Calculate judge agreement on evaluation labels
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Build an LLM-as-judge evaluator
Automated evaluation with a language model as the judge is one of the most important patterns in applied AI...
Create a golden dataset test suite
A golden dataset is a curated set of test cases you trust to detect regressions. Build the infrastructure to load, run,...
Implement structured request tracing
Build a span-based tracing system for LLM requests -- the core data collection layer that feeds monitoring dashboards...
Build a quality monitoring dashboard data pipeline
Transform raw LLM request traces into hourly quality snapshots suitable for a time-series monitoring dashboard.
Implement A/B evaluation for prompt variants
Changing a prompt without measuring the impact is one of the most common mistakes in applied AI engineering. Build a...
Build a human feedback collection pipeline
User feedback is the ground truth that calibrates all other evaluation signals. Build a feedback collection and...
Implement A/B testing for model versions
When you want to test a new model version or prompt in production, rolling it out to all users at once is risky. A/B...
Build an LLM call tracer with span export
Observability for LLM applications goes beyond logging. When a user sees a bad response, you need to reconstruct the...
Track and alert on LLM cost per feature
Without cost visibility, you will discover that your most popular feature is generating $30,000/month in LLM costs via...
Implement A/B testing for model versions in production
Deploying a new model version without traffic splitting is risky. A/B testing routes a percentage of users to a new...
Build an observability metrics aggregator
Production LLM services emit request events. An aggregator processes them into the operational metrics that matter:...
Build a reference-based BLEU/ROUGE scorer
Reference-based metrics compare generated text to a known-good reference. They are the cheapest automated evaluation...
Compute semantic similarity for evaluation
Embedding-based similarity captures whether two texts mean the same thing even with different wording — essential for...
Build a human-in-the-loop annotation queue
Automated evaluation is fast but imperfect. An annotation queue routes only the cases where human judgment adds the...
Implement A/B evaluation framework for prompt variants
Before shipping a prompt change you need evidence it improves the metric that matters, not just anecdotal inspection of...
Build a regression test suite for LLM outputs
A regression suite catches when a change makes previously-passing cases fail. It focuses on non-negotiable behaviors...
Implement structured logging for AI call traces
Debugging bad AI outputs requires the full call context as a queryable structured event: prompt version, model,...
Trace an agent decision chain
A bad final answer might trace back to a wrong tool call in step 2, caused by a misinterpreted instruction in step 1....
Build a cost and latency dashboard aggregator
Without visibility into cost per feature and per model, you cannot make rational optimization decisions for production...
Build a multi-metric eval pipeline
Production evaluation rarely relies on a single metric. A useful pipeline runs several scorers in parallel, aggregates...
Implement a prompt version registry
When you run A/B evals or regression tests across prompt versions, you need a lightweight registry that ties eval...
Implement a quality threshold gate for CI
An eval harness only protects your system if it blocks bad deployments. This exercise builds the gate that translates...
Build an eval result storage and comparison layer
Eval results are only useful if you can compare them across runs. This exercise builds the persistence and comparison...
Implement token-level faithfulness scoring
Faithfulness measures whether a generated answer is supported by the retrieved context. A simple approach checks...
Build a production quality alert system
Quality alerts should trigger on patterns, not individual events. A single bad response is noise. A 10% decline in pass...
Implement an evaluation configuration loader
Eval pipelines need configuration: which metrics to run, what thresholds to enforce, which model to use as the judge,...
Implement a sampling strategy for production evaluation
You cannot run an expensive LLM judge on every production request. A good sampling strategy applies the expensive...
Implement a judge calibration pipeline
You have an LLM-as-judge that scores RAG responses for faithfulness, but you don't know if it agrees with humans. Build...
Build an evaluation-driven development workflow
Evaluation-driven development means: write the eval before you write the feature, then use eval results to guide every...
Llm Foundations (6)
Build a prompt assembly pipeline
Every LLM feature you ship assembles a messages array before making an API call. Doing this carelessly — concatenating...
Parse structured LLM output with fallback
LLMs are not databases. Even when you ask for JSON, you sometimes get JSON wrapped in markdown fences, JSON with...
Implement model routing by task complexity
Routing every request to your most powerful model is expensive. Routing every request to your cheapest model produces...
Build a token cost tracker
Without explicit cost tracking, AI features develop cost problems silently. Token counts accumulate, model choices...
Add prompt injection detection
Prompt injection is the LLM equivalent of SQL injection. Malicious users craft input that attempts to override your...
Stream LLM responses with progress callbacks
Streaming is the single highest-impact UX improvement you can make to an LLM feature. Without streaming, a user waits...
Prompt Formatting (5)
Render a safe prompt template from user input
Build a template renderer that separates trusted system instructions from untrusted user input and outputs a final...
Build a structured prompt template with system/user role separation
Prompt injection attacks happen when user-controlled content reaches the system role. The first defense is strict...
Count tokens and trim a messages array to a budget
Context window management is the most common source of silent failures in production LLM features. When history grows...
Implement a prompt version registry with promotion logic
Prompt changes that degrade quality need to be detectable and rollback-able. This exercise builds the minimal data...
Multi-turn conversation builder with JSON serialization
A conversation is a list of messages that must be assembled correctly and serialized for persistence. This exercise...
Python Ai (6)
Build a Pydantic model hierarchy for multi-provider LLM responses
Different LLM providers return fundamentally different response shapes. OpenAI wraps content in...
Implement async batch processing for LLM calls
Calling an LLM API once takes 500ms–3s. Processing 50 prompts serially takes 25–150 seconds. Async batch processing...
Create a streaming document processor with generators
Loading 10GB of documents into memory before processing them is the fastest path to an OOM kill. Generator pipelines...
Build a type-safe provider abstraction layer
Application code that calls OpenAI directly becomes brittle when you add Anthropic as a fallback, switch to a local...
Implement an LRU cache with TTL for LLM responses
LLM calls are expensive and slow. For many AI features — documentation lookups, FAQ answering, static content...
Profile and optimize a token-heavy pipeline
Before you can optimize an AI pipeline you need to know where the time actually goes. This exercise gives you a slow,...
Python Refresh (16)
Normalize provider payloads into a typed response model
Write a function that accepts a loose provider response and returns a normalized typed payload with explicit error...
Parse JSONL benchmark rows into normalized records
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Build a slug generator for lesson and project records
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Parse CLI args for an evaluation runner
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Write a pathlib-based artifact export helper
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Map provider enums into internal response states
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Serialize experiment configs to reproducible snapshots
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Protect temp files during dataset generation
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Validate LLM structured output with Pydantic
LLMs frequently return structured data that does not perfectly match what you asked for. A model asked to extract...
Build a streaming token accumulator
LLM providers stream responses as a sequence of small delta chunks. Each chunk contains a fragment of the final...
Implement token counting and cost tracking
Token usage is the billing unit for every major LLM API. Without explicit tracking, AI features run with no cost...
Build a type-safe configuration loader for AI services
AI services have more configuration surface than typical web backends: model names, temperature, token limits, provider...
Build a type-safe config loader from environment variables
AI services are configured through environment variables: API keys, model names, timeout values, rate limits, feature...
Parse and extract structured data from LLM JSON responses
LLMs asked to return JSON do so imperfectly. They wrap output in markdown code fences, add explanatory prose before the...
Count tokens accurately for prompt budget management
Token counting is a foundational skill in AI engineering. You need it to prevent context window overflow, to estimate...
Implement batch processing with configurable rate limiting
Production AI systems routinely send thousands of requests to provider APIs. Without rate limiting, you hit 429 errors,...
Rag Systems (6)
Implement semantic chunking with overlap
Fixed-size chunking is fast but blind to sentence boundaries — it cuts paragraphs in half, splits mid-sentence, and...
Build a hybrid retrieval pipeline
Pure vector search misses documents with specific entity names, codes, or technical identifiers. Pure keyword search...
Add reranking to retrieval results
First-stage retrieval optimizes for recall: get the right documents into the candidate set. Reranking optimizes for...
Evaluate RAG faithfulness with citation tracking
A RAG answer is only as trustworthy as its grounding. Faithfulness evaluation measures whether each claim in the answer...
Build an incremental document indexer
Re-indexing your entire corpus every time a document changes is wasteful and creates operational risk. Incremental...
Implement retrieval with metadata filtering
Pure semantic search returns results regardless of when documents were created, who authored them, or what type they...
Retrieval (15)
Score retrieval candidates with metadata boosts
Combine vector similarity and metadata weighting into a final retrieval score for candidate chunks.
Detect malformed retrieval chunks before indexing
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Merge keyword and vector retrieval results
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Filter retrieval contexts by recency and source type
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Assemble citation spans from retrieved snippets
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Rank support documents with freshness boosts
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Deduplicate retrieval candidates across indexes
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Expand metadata filters into SQL-ready conditions
Implement this task with explicit validation, predictable output shape, and enough error handling that it could survive...
Build a context window assembler for RAG prompts
One of the most underestimated problems in production RAG is fitting retrieved chunks into the LLM context window...
Implement RAG answer generation with inline citations
Most RAG implementations generate an answer and separately display source documents, with no connection between a...
Build a RAG evaluation harness with Recall@K and MRR
You cannot improve what you cannot measure. Most RAG pipelines are deployed and then tuned based on user complaints —...
Implement query expansion for improved RAG recall
Users do not always phrase their questions the way your documents phrase their answers. A user asks "how do I cancel my...
Build a production RAG pipeline with caching and fallbacks
A RAG pipeline that works in a demo rarely survives its first month in production. The embedding API goes down. The...
Implement BM25 keyword search from scratch
Every production RAG system uses hybrid search — vector (semantic) retrieval combined with keyword (BM25) retrieval....
Build a multi-stage RAG pipeline with query routing
Not all queries are the same. "What is our refund policy?" is a simple factual lookup. "Compare our Q3 2023 and Q4 2023...