Agents and Protocols
Why Agent Loops Run Away and How Step Budgets Stop Them
Why tool-calling agents loop until something kills them, the quadratic token cost of every extra step, and how step budgets and loop detectors stop it.
0 ★ earned in this browser
50 articles · 50 labs · up to 600 stars
Every guide in this series pairs a deep, production-focused article with an interactive lab you can play right on the page. Each lab is a small deterministic simulation with missions: start from a broken setup, change the controls, and earn up to three stars by meeting a target the way an engineer would in production.
The labs are synthetic teaching models. Their numbers are calculated estimates that make a mechanism visible; they are not benchmarks of any real model, GPU or service.
Agents and Protocols
Agent loops, tools, memory, MCP, A2A and the controls that keep autonomous systems bounded.
Agents and Protocols
Retrying Agent Tool Calls Without Executing the Same Action Twice
Why agent retries and re-plans repeat refunds and emails, how idempotency keys deduplicate them, and how to choose the key, the scope and the retention.
Agents and Protocols
Agent Context Compaction: What to Keep When the Window Fills Up
What to drop, summarise or pin when a long-running agent fills its context window, with the cost arithmetic and the failure modes of each strategy.
Agents and Protocols
MCP Streamable HTTP in Production: Sessions, Resumption and Load Balancers
Why MCP sessions break behind load balancers, what revision 2026-07-28 changed in Streamable HTTP, and how keep-alives and idempotency stop lost calls.
Agents and Protocols
MCP Tool Poisoning: How One Malicious Description Hijacks an Agent
How hidden instructions in MCP tool descriptions hijack agents, and how scanning, hash pinning, taint policies, routing and confirmations stop them.
Agents and Protocols
MCP Authorization Explained: OAuth 2.1, Resource Indicators and Token Audience
How MCP authorization uses OAuth 2.1, resource indicators and audience checks so a token for one server cannot open another, with lifetime and scope maths.
Agents and Protocols
A2A Task Lifecycle Under Failure: Streaming, Push Notifications and Reconnects
How A2A tasks fail in production when streams drop, webhooks miss or agents restart, and how SubscribeToTask, retries and reconciliation recover them.
Agents and Protocols
Supervisor or Swarm: When Multi-Agent Systems Actually Beat One Agent
When splitting an agent into a supervisor and specialists, or a swarm of peers, beats one agent: the arithmetic of tool count, handoffs, tokens and latency.
Agents and Protocols
Tool Selection at Scale: Why Hundreds of Tools Break Agents and How to Route
Why agents pick the wrong tool once the catalog reaches hundreds, what every definition costs per call, and how retrieval and grouping route each request.
Agents and Protocols
Agent Memory That Does Not Rot: Write Policies, Decay and Conflicts
Why long-term agent memory goes stale, wrong or poisoned, and how write gates, reconciliation, decay and provenance keep it accurate over months.
Agents and Protocols
Human Approval Gates That Keep Agents Safe Without Becoming the Bottleneck
Why gating every agent tool call fails twice: reviewers fatigue and queues grow. How to size, tier and time out approval gates with the actual arithmetic.
Agents and Protocols
Evaluating Agent Trajectories: Why pass@k Hides Unreliable Agents
Why pass@k flatters agents that fail users, how pass^k and trajectory grading measure reliability, and how many tasks and trials an honest eval needs.
LLM Core
Tokenizers, attention, sampling, preference training, quantization and calibration, from the inside.
LLM Core
Tokenizers Decide Your Bill: How BPE Merges Treat Languages and Code
Why the same message can cost several times more tokens in Hindi or in nested code, how byte-level BPE merges decide it, and what to measure before you budget.
LLM Core
Grouped-Query Attention: How Sharing KV Heads Shrinks Inference Memory
Why the KV cache, not the weights, caps concurrency, how grouped-query attention cuts it 4x or more, and what conversion and FP8 cost in quality.
LLM Core
RoPE and Long Context: Why Models Fail Past Their Trained Length
Why RoPE models break just past their trained context, how interpolation, NTK scaling and YaRN remap angles, and what each costs in quality and compute.
LLM Core
Sampling Settings That Fix Real Failures: Temperature, Top-p and Min-p
How temperature, top-k, top-p, min-p and repetition penalties change the next-token distribution, which failure each one fixes, and what each one breaks.
LLM Core
Mixture-of-Experts Routing: Load Balancing, Capacity Factors and Dropped Tokens
How MoE routers pick experts, why capacity factors drop tokens, how load balancing works, and what imbalance costs in latency and quality when serving.
LLM Core
Constrained Decoding: How JSON Schema Mode Masks Invalid Tokens
How JSON schema mode turns a grammar into per-token masks, what the masks cost in latency, and why valid JSON can still contain invented values.
LLM Core
Reasoning Models and Thinking Budgets: When Extra Tokens Buy Accuracy
How thinking-token budgets trade accuracy for cost and latency, where extra reasoning stops paying, and when routing or majority voting beats a bigger budget.
LLM Core
From RLHF to DPO and GRPO: How Preference Training Reshapes a Model
How RLHF with PPO, DPO and GRPO turn preferences into gradients, what beta and group size really control, and how reward hacking shows up.
LLM Core
LoRA Rank, Alpha and Target Modules: Choosing Settings That Actually Work
What LoRA rank, alpha and target modules really control, the parameter and memory arithmetic, and how to pick settings that learn without overfitting.
LLM Core
Why INT4 Breaks Some Layers: Outliers, Group Size and Activation-Aware Scaling
Why INT4 weights work for most layers but break a few: the rounding arithmetic, outliers, group size, zero points and activation-aware scaling.
LLM Core
Compute-Optimal Model Sizing: Applying Chinchilla Scaling Laws to a Real Budget
How to split a training budget between parameters and tokens with Chinchilla-style scaling laws, and when serving cost or a data wall should change it.
LLM Core
Logprobs and Calibration: Measuring When a Language Model Is Guessing
How to turn token logprobs into confidence you can act on: reliability diagrams, ECE, temperature and Platt scaling, abstention thresholds and drift.
Inference and Serving
Batching, KV cache, scheduling, autoscaling, routing and caching for production LLM serving.
Inference and Serving
Continuous Batching Explained: Why Static Batches Leave GPUs Idle
Why request-level batching strands GPU slots behind the longest sequence, how iteration-level scheduling fixes it, and the per-token latency it costs.
Inference and Serving
PagedAttention Block Tables: How vLLM Ends KV Cache Fragmentation
Why reserving KV cache for the maximum length wastes most GPU memory, how block tables page it like an OS, and how to pick block size and memory.
Inference and Serving
Prefix Caching With Radix Trees: Reusing Prompt Work Across Requests
How a radix tree lets an LLM server reuse KV for shared prompt prefixes, why one timestamp can zero the hit rate, and how routing keeps hits.
Inference and Serving
KV Cache Preemption in vLLM: Swap, Recompute and Behavior Under Memory Pressure
Why vLLM evicts running requests when KV blocks run out, what swap and recompute cost, and how to size memory and admission to avoid preemption storms.
Inference and Serving
Disaggregated Prefill and Decode: When Splitting the GPU Pool Pays Off
When separate prefill and decode GPU pools beat colocated serving: interference, KV transfer bandwidth, the pool ratio and goodput per GPU.
Inference and Serving
Chunked Prefill: Stopping Long Prompts From Freezing Every Other Stream
Why one long prompt can pause every stream on a GPU, how chunked prefill bounds each step, and how to pick the token budget from the roofline.
Inference and Serving
KV Cache Capacity Planning: How Many Concurrent Users Fit on One GPU
How to turn GPU memory, model shape, KV dtype and tensor parallelism into a KV token budget, concurrent sequences and users per GPU, with worked numbers.
Inference and Serving
TTFT, TPOT and Goodput: Setting LLM Latency SLOs That Mean Something
Why tokens per second and mean latency hide unhappy users, how to define TTFT and TPOT SLOs, and how to measure goodput: requests that meet both.
Inference and Serving
Autoscaling LLM Servers: Why GPU Utilization Lies and Queue Depth Wins
GPU utilization reads 100% at a tenth of an LLM server's capacity. Learn which signals to autoscale on, how much headroom cold starts need, and why.
Inference and Serving
Model Cascades and Routing: Cutting LLM Cost Without Losing Answer Quality
Most LLM queries do not need your largest model. Learn the cost, accuracy and latency arithmetic of cascades and routers, and how each fails in production.
Inference and Serving
Token-Based Rate Limiting for LLM APIs: Buckets, Fairness and Priority Lanes
Request limits let one long-context tenant starve everyone. Learn token buckets, max_tokens reservation, bucket sizing, fair queueing and priority lanes.
Inference and Serving
Tensor or Pipeline Parallelism for Serving: Choosing How to Split a Model
When a model outgrows one GPU you can split layers or split matrices. Learn the memory, latency and communication arithmetic that decides between TP and PP.
Inference and Serving
Semantic Caching for LLMs: Similarity Thresholds and the Wrong-Answer Risk
How semantic caches decide a hit, why one cosine threshold trades LLM cost against wrong answers, and how to calibrate, scope and invalidate safely.
RAG in Production
Chunking, vector indexes, hybrid retrieval, reranking, freshness, permissions and RAG evaluation.
RAG in Production
Chunking Strategies Compared: Fixed, Recursive, Semantic and Late Chunking
Fixed, recursive, semantic and late chunking compared on the numbers that matter: answers split, recall@k, context tokens per query and index size.
RAG in Production
HNSW Tuning for RAG: M, efSearch and the Recall Versus Latency Trade
How HNSW layers, M, efConstruction and efSearch set recall, latency and memory, why filters and deletes break it, and how to tune it.
RAG in Production
Reciprocal Rank Fusion Tuning: When BM25 Rescues Dense Retrieval Misses
Why dense retrieval misses codes and rare terms, how reciprocal rank fusion with BM25 recovers them, and how to tune k, weights and list depth.
RAG in Production
Cross-Encoder Reranking Depth: How Many Candidates Are Worth the Latency
How deep a cross-encoder should rerank: the recall ceiling, diminishing nDCG gains, per-pair GPU cost and the queueing that makes p95 explode.
RAG in Production
Embedding Quantization: Binary and Int8 Vectors Without Losing Recall
How int8, int4 and binary embeddings cut vector index memory by 4x to 32x, why recall drops, and how calibration and rescoring win it back.
RAG in Production
Query Rewriting and HyDE: Fixing the Question Before Retrieval Runs
Why raw user queries miss in RAG, how condensing, multi-query and HyDE fix them, and how rewrites break exact IDs, add latency and drift.
RAG in Production
RAG Failure Triage: Separating Retrieval Misses From Generation Errors
A stage-by-stage method for tracing wrong RAG answers to coverage gaps, retrieval misses, prompt truncation or generation errors, with the arithmetic.
RAG in Production
Permission-Aware RAG: Filtering by Access Control Without Destroying Recall
How post-filtering, filtered HNSW, pre-filter scans and query planners enforce document ACLs in RAG, why recall collapses, and how revocations leak.
RAG in Production
Keeping RAG Indexes Fresh: Incremental Updates, Deletes and Tombstones
How RAG indexes go stale, why deletes become tombstones that slow search, and how change capture, chunk IDs and compaction keep answers current.
RAG in Production
Lost in the Middle: Ordering and Packing Retrieved Context for Long Prompts
Why answers buried mid-prompt get ignored, how to order, dedupe and pack retrieved chunks under a token budget, and how to test position effects in production.
RAG in Production
Multi-Hop Questions in RAG: Iterative Retrieval and Query Decomposition
Why one vector search cannot answer bridge questions, how iterative retrieval and decomposition chain hops, and the latency, cost and error compounding each adds.
RAG in Production
Grounded Citations in RAG: Verifying Every Claim Against Its Source Passage
Why a citation marker proves nothing until checked, how claim-level entailment verification works, and how to trade unsupported claims against recall and cost.
RAG in Production
RAG Latency Budget: Where the Milliseconds Go Between Query and Answer
Where time goes in a RAG request, why p95 is not the sum of stage medians, and how rewrite, rerank depth, prompt size and hedging move time to first token.
Views are my own and do not represent any employer, past or present.