All 102 articles, sorted alphabetically
Keep Table Headers With Rows to Improve Retrieval Answers
Text chunkers cut tables between the header and the rows, so the retrieved chunk says 7 days without saying what 7 days is. A measured BM25 comparison of three chunkings on a synthetic store corpus, a row serializer that carries headers and titles, and the checks that catch the problem.
Read article →Two Sources of Eval Noise: Separating Generation Variance From Grader Variance With Repeated Trials and Re-Grading
An agent eval score is noisy for three separate reasons: which tasks are in the set, what the model happened to generate, and what the grader happened to say. How to estimate each from repeated generations and re-grades, and where extra samples buy the most precision.
Read article →Tokenizers Decide Your Bill: How BPE Merges Treat Languages and Code
Why the same message can cost several times more tokens in Hindi or in nested code, how byte-level BPE merges decide it, and what to measure before you budget.
Read article →Grouped-Query Attention: How Sharing KV Heads Shrinks Inference Memory
Why the KV cache, not the weights, caps concurrency, how grouped-query attention cuts it 4x or more, and what conversion and FP8 cost in quality.
Read article →RoPE and Long Context: Why Models Fail Past Their Trained Length
Why RoPE models break just past their trained context, how interpolation, NTK scaling and YaRN remap angles, and what each costs in quality and compute.
Read article →Sampling Settings That Fix Real Failures: Temperature, Top-p and Min-p
How temperature, top-k, top-p, min-p and repetition penalties change the next-token distribution, which failure each one fixes, and what each one breaks.
Read article →Mixture-of-Experts Routing: Load Balancing, Capacity Factors and Dropped Tokens
How MoE routers pick experts, why capacity factors drop tokens, how load balancing works, and what imbalance costs in latency and quality when serving.
Read article →Constrained Decoding: How JSON Schema Mode Masks Invalid Tokens
How JSON schema mode turns a grammar into per-token masks, what the masks cost in latency, and why valid JSON can still contain invented values.
Read article →Reasoning Models and Thinking Budgets: When Extra Tokens Buy Accuracy
How thinking-token budgets trade accuracy for cost and latency, where extra reasoning stops paying, and when routing or majority voting beats a bigger budget.
Read article →From RLHF to DPO and GRPO: How Preference Training Reshapes a Model
How RLHF with PPO, DPO and GRPO turn preferences into gradients, what beta and group size really control, and how reward hacking shows up.
Read article →LoRA Rank, Alpha and Target Modules: Choosing Settings That Actually Work
What LoRA rank, alpha and target modules really control, the parameter and memory arithmetic, and how to pick settings that learn without overfitting.
Read article →Why INT4 Breaks Some Layers: Outliers, Group Size and Activation-Aware Scaling
Why INT4 weights work for most layers but break a few: the rounding arithmetic, outliers, group size, zero points and activation-aware scaling.
Read article →Compute-Optimal Model Sizing: Applying Chinchilla Scaling Laws to a Real Budget
How to split a training budget between parameters and tokens with Chinchilla-style scaling laws, and when serving cost or a data wall should change it.
Read article →Logprobs and Calibration: Measuring When a Language Model Is Guessing
How to turn token logprobs into confidence you can act on: reliability diagrams, ECE, temperature and Platt scaling, abstention thresholds and drift.
Read article →Continuous Batching Explained: Why Static Batches Leave GPUs Idle
Why request-level batching strands GPU slots behind the longest sequence, how iteration-level scheduling fixes it, and the per-token latency it costs.
Read article →PagedAttention Block Tables: How vLLM Ends KV Cache Fragmentation
Why reserving KV cache for the maximum length wastes most GPU memory, how block tables page it like an OS, and how to pick block size and memory.
Read article →Prefix Caching With Radix Trees: Reusing Prompt Work Across Requests
How a radix tree lets an LLM server reuse KV for shared prompt prefixes, why one timestamp can zero the hit rate, and how routing keeps hits.
Read article →KV Cache Preemption in vLLM: Swap, Recompute and Behavior Under Memory Pressure
Why vLLM evicts running requests when KV blocks run out, what swap and recompute cost, and how to size memory and admission to avoid preemption storms.
Read article →Disaggregated Prefill and Decode: When Splitting the GPU Pool Pays Off
When separate prefill and decode GPU pools beat colocated serving: interference, KV transfer bandwidth, the pool ratio and goodput per GPU.
Read article →Chunked Prefill: Stopping Long Prompts From Freezing Every Other Stream
Why one long prompt can pause every stream on a GPU, how chunked prefill bounds each step, and how to pick the token budget from the roofline.
Read article →KV Cache Capacity Planning: How Many Concurrent Users Fit on One GPU
How to turn GPU memory, model shape, KV dtype and tensor parallelism into a KV token budget, concurrent sequences and users per GPU, with worked numbers.
Read article →TTFT, TPOT and Goodput: Setting LLM Latency SLOs That Mean Something
Why tokens per second and mean latency hide unhappy users, how to define TTFT and TPOT SLOs, and how to measure goodput: requests that meet both.
Read article →Autoscaling LLM Servers: Why GPU Utilization Lies and Queue Depth Wins
GPU utilization reads 100% at a tenth of an LLM server's capacity. Learn which signals to autoscale on, how much headroom cold starts need, and why.
Read article →Model Cascades and Routing: Cutting LLM Cost Without Losing Answer Quality
Most LLM queries do not need your largest model. Learn the cost, accuracy and latency arithmetic of cascades and routers, and how each fails in production.
Read article →Token-Based Rate Limiting for LLM APIs: Buckets, Fairness and Priority Lanes
Request limits let one long-context tenant starve everyone. Learn token buckets, max_tokens reservation, bucket sizing, fair queueing and priority lanes.
Read article →Tensor or Pipeline Parallelism for Serving: Choosing How to Split a Model
When a model outgrows one GPU you can split layers or split matrices. Learn the memory, latency and communication arithmetic that decides between TP and PP.
Read article →Semantic Caching for LLMs: Similarity Thresholds and the Wrong-Answer Risk
How semantic caches decide a hit, why one cosine threshold trades LLM cost against wrong answers, and how to calibrate, scope and invalidate safely.
Read article →How QLoRA Trains Adapters Without Updating the Base Model
Follow one transformer projection through frozen four-bit weights, trainable adapters and backpropagation, then check the gradient path with a tiny CPU example.
Read article →OpenAI Service Tiers After the Project Controls of 21 October: Flex, Fast and Default, the Resolved Tier and the New 400
On 21 October 2026 OpenAI let projects and organisations restrict which service tiers requests may use. How default, flex, fast and auto resolve, why the documented flex fallback can now fail with a 400, what the response tier tells you, and a routing policy that survives the new controls.
Read article →Claude Models on Google Cloud, Anthropic Managed: IAM Roles, Workspace IDs, How It Differs From Google Managed and a Migration Checklist
Launched on 22 October 2026: the Claude API operated by Anthropic, reached through a Google Cloud project with IAM access and Marketplace billing. How it differs from the Google-managed offering, how requests and credentials change, and what to check before moving.
Read article →From Probabilities to Actions With the Decisions API: Predicate Thresholds, Choice Versus Score, Batched Questions and Calibration
The OpenAI Decisions API reached general availability on 22 October 2026. A practical guide to what its answers contain, how to turn a probability into approve, review or deny with a labelled sample and a cost model, why a score must not be rounded, and how to batch and cache questions safely.
Read article →Moving to Anthropic API Version 2026-10-21: Sampling Parameters Removed, Bearer Keys, Role-Typed Messages and What to Test
Anthropic published API version 2026-10-21 and the Python SDK 2.0.0 that defaults to it on 21 October 2026. This reads the migration guide, shows which changes break code versus proxies, and records what the 1.17.0 and 2.0.0 Python SDKs actually send, checked locally against a mock transport.
Read article →Pacing an Agent Fleet From Provider Rate-Limit Headers: Request and Token Limits, Reset Times, 429s and Shared Budgets
Provider responses carry the headers you need to pace a fleet of agents instead of hammering into 429s: request and token limits, remaining counts and reset times. This reads the Anthropic, OpenAI and Gemini fields, shows a pacer that respects them, and separates a retryable 429 from a spend cap.
Read article →Clearing Old Tool Results in Long Agent Runs: Context Editing, Compaction, What Gets Lost and What It Costs the Cache
When a long agent run fills the context window, you can clear old tool results by rule or compact the whole history into a summary. This compares the two, reads the Anthropic context-editing fields exactly, and shows why clearing breaks the prompt cache and how to size it so it is worth it.
Read article →Keeping the Prompt Cache Warm in Agent Loops: Prefix Breakers, Breakpoints, TTL Gaps and Reading the Usage Fields
Why agent loops lose prompt-cache hits: a timestamp in the system prompt, tools that change order, compaction, pauses longer than the cache lifetime and breakpoints in the wrong place, how to find the first changed block, and how to read cache usage fields and price the result.
Read article →Moving Deferrable Agent Work to Provider Batch APIs: What Can Wait, a Submission Ledger, Partial Results, Expiry and Resubmission
Which agent workloads can use the OpenAI, Claude and Gemini batch APIs at half the synchronous price, how their end-of-window behaviour differs, and a submission ledger that matches results by custom_id, resubmits only what is missing and never applies a result twice.
Read article →Model Deprecation With Automatic Re-Routing: Pinning, Re-Baselining Evals and Migrating Before the Provider Moves You
Providers now retire models by routing an old model string to a newer model instead of returning an error. What that does to evaluations, cost and output formats, how to detect and pin, and a migration runbook using the Gemini deprecations of 8 October 2026 as the worked example.
Read article →Choosing Reasoning Effort per Request: A Routing Policy and an Evaluation Harness
Treat reasoning effort as a per-request decision: what the current API controls do, how to pick a level per task class from an evaluation harness with confidence bounds, escalation on verifier failure, and the failure modes of reasoning budgets.
Read article →ACL-Aware Retrieval for Enterprise RAG
How to make enterprise retrieval-augmented generation respect source-system permissions: identity propagation, ACLs copied onto chunks, pre-filtered v…
Read article →Sentence Similarity with BERT and Flask
Build a sentence-similarity API with BERT-family models and Flask: tokenisation, mask-aware pooling, why raw BERT embeddings fail, Gunicorn serving, t…
Read article →Evaluating LLM Outputs, in depth
Evaluate LLM outputs you can trust: error analysis to criteria, the evaluator ladder, validating judges with kappa, pass@k vs pass^k, Wilson intervals…
Read article →Freshness and Incremental Embedding Pipelines
How to keep a retrieval index fresh without re-embedding everything: change capture, content-hash diffing of stable chunks, ordered and idempotent upd…
Read article →Building Golden Datasets for Production AI Systems
How to build, label, size and maintain the curated evaluation datasets that production AI decisions rest on: sourcing and stratification, case schemas…
Read article →LLM Agent Architecture in Depth
LLM agent architecture in depth: the agent loop, tool registry and dispatch, planning patterns, memory and compaction, guardrails, a worked refund exa…
Read article →LLM-as-a-Judge: Calibration, Bias, and Pairwise Evaluation
Using a language model as an evaluator without fooling yourself: pointwise versus pairwise judging, position and length bias, swap-consistent verdicts…
Read article →Beam Search, in depth
Beam search for LLMs as an engineering object: a tested reference loop, the 2B candidate rule and finished-hypothesis bookkeeping, KV-cache forking wi…
Read article →LLM Caching Layers, in depth
The LLM caching stack from GPU to gateway: block-hashed prefix caches, KV offload tiers, cache-aware routing, provider prompt caching, how hit rates c…
Read article →LLM Caching Strategies, in depth: layering caches, designing keys, invalidation and the cost model
How to compose caches across an LLM pipeline: the six cacheable layers, cache-key design with model, parameters, tools, context and scope, when reuse …
Read article →Chunked Prefill Explained: Prefill vs Decode, TTFT and ITL
Chunked prefill in LLM serving: why prefill is compute-bound and decode memory-bound, splitting long prompts under a per-step token budget, TTFT and I…
Read article →LLM Context Extension Architecture, in depth: position configs, long-data mixtures, staged training, evaluation gates and serving cost
An engineering guide to extending a pretrained LLM's context window: choosing and configuring the RoPE scaling method, building a long-document d…
Read article →LLM Context Length, in depth: what fits, what it costs and how much the model really uses
What an LLM context window holds and costs: tokens that count against it, prefill FLOPs and KV-cache memory worked for a 70B model, advertised versus …
Read article →LLM Context Management, in depth: building a context assembler that filters, packs, places and records
Managing LLM context as an engineered component: typed candidate items, tenant and freshness filters, value-per-token packing with an output reserve, …
Read article →LLM continuous batching architecture
Deep-dive on continuous (iteration-level) batching for LLM serving: instead of static batches that wait for their slowest member, a scheduler recompos…
Read article →A Practical LLM Evaluation Harness for Regression Testing
How to build an LLM evaluation harness that catches regressions: versioned cases, pinned runs, layered scorers, paired statistics, slice-level gates, …
Read article →Greedy Decoding, in depth: argmax generation, degeneration, temperature-0 nondeterminism and speculative verification
Greedy decoding from the logits up: the argmax loop with a KV cache, why stepwise argmax misses the likeliest sequence, repetition loops and their fix…
Read article →LLM Guardrails in Production, in depth
LLM guardrails in production, in depth: mapping risks to controls, deterministic, model and authorization checks, a decision engine with one verdict a…
Read article →LLM guided decoding architecture
Deep-dive on guided (constrained/structured) decoding: why validity can't be reliably obtained by prompting, how a schema or grammar compiles int…
Read article →LLM KV-cache offloading architecture
Deep-dive on KV-cache offloading for LLM serving: paged block management, LRU eviction with reuse hints, prefix caching, pinned host staging, async co…
Read article →KV Cache Quantization: INT8, FP8 and INT4 for LLM Serving
What KV cache quantization is: compressing keys and values from fp16 to int8, fp8 or int4, per-token vs per-channel scales, outliers, fused kernels, p…
Read article →MoE serving architecture, in depth: sizing a real model, choosing TP or expert parallelism, and running it in production
An operator's guide to serving mixture-of-experts models: parameter and KV arithmetic for Qwen3-30B-A3B, why active parameters do not predict dec…
Read article →LLM Priority Scheduling: GPU Preemption, SLAs and Fairness
Priority scheduling for LLM inference on shared GPUs: priority classes, admission control, KV-cache preemption by swap or recompute, fairness, load sh…
Read article →LLM prompt caching -- reusing computed context for cost and latency
Deep-dive on LLM prompt caching: KV cache reuse for repeated prompt prefixes, prefix matching, cost reduction (cached tokens billed less) and latency …
Read article →Repetition Penalty, in depth: Why Models Loop and How Penalties Really Work
Why language models fall into repetition loops and how repetition, frequency, presence, n-gram and DRY penalties change the logits: exact formulas, wo…
Read article →LLM Reranker Architecture, in depth: pointwise, pairwise and listwise LLM ranking, serving inside a latency budget, and distilling into small rerankers
How LLM rerankers work and how to run them: where reranking sits after hybrid retrieval, pointwise yes/no logit scoring with code, pairwise and listwi…
Read article →LLM Routing Explained: How a Model Router Picks Which LLM
How LLM routing works: difficulty prediction and learned routers, cascades vs predictive dispatch, quality gates, judge models, escalation, cost and d…
Read article →LLM Temperature, Top-k, Top-p: How Sampling Works
How LLM temperature, top-k, top-p and min-p shape output: from the logit vector through penalties, scaling and truncation to softmax and the final tok…
Read article →LLM Semantic Caching: Reusing Answers by Meaning, Not Bytes
How a semantic cache embeds queries, searches a vector index of prior Q&A, and serves on a similarity threshold — cutting cost and latency while m…
Read article →Speculative Decoding Explained: Draft Model, Medusa, Tree
How speculative decoding speeds up LLM inference: a draft model proposes tokens, the target verifies them in one pass, Medusa heads, tree and self-spe…
Read article →LLM Streaming Output Architecture, in depth: SSE framing, incremental detokenization, relays, cancellation and failure modes
How LLM token streaming works from the decode step to the screen: TTFT versus inter-token latency, server-sent event framing and provider wire shapes,…
Read article →Structured output architecture
Deep-dive on structured LLM output: JSON Schema to FSM compilation, token-mask decoding, tokenizer lifting, vLLM/TGI/llama.cpp guided decoding, JSON m…
Read article →LLM Temperature, in depth: one scalar, five jobs, from sampling, pass@k and RL rollouts, calibration, distillation and contrastive training
What temperature does to a softmax, worked by hand, and how the same scalar is chosen, fitted or learned in five places: decode-time sampling, best-of…
Read article →Tokenization, in depth: how token boundaries shape model behaviour and break prompts, fine-tuning and spans
What the integer view of text does to an LLM in practice, with real tiktoken splits: leading spaces and context-dependent ids, the trailing-space prom…
Read article →LLM tokenization architecture
Deep-dive on LLM tokenization: BPE training and vocabulary-size trades, the runtime encode pipeline with byte fallback, special-token security at trus…
Read article →LLM Tool Use Explained: Tool Schemas, Call Loop, Parallel Calls
How LLM tool use (function calling) works: tool schemas, the call-execute-observe loop, parallel tool calls, argument validation, error handling and M…
Read article →Top-p (Nucleus) Sampling, in depth
Top-p (nucleus) sampling as implemented in serving engines: the adaptive nucleus with a worked example, boundary and tie rules that make libraries dis…
Read article →Transformer Architecture Introduction, in depth: attention by hand, the residual stream, training vs decoding and a working model
A first-principles introduction to the transformer: why attention replaced recurrence, a hand-computed attention example, the MLP and residual stream,…
Read article →Mixture of Experts in 2026, in depth: how DeepSeek, Qwen3, Llama 4, gpt-oss and Kimi models are built, and how to choose, fine-tune and serve them
The state of mixture-of-experts language models in 2026 from their own configs: expert counts, top-k, shared experts, bias-based balancing, group-limi…
Read article →Positional Embeddings: RoPE, ALiBi, and the Quest for Perfect Long-Range Memory
Transformers revolutionized AI by processing all tokens in a sequence simultaneously, a key enabler for parallel training and superior long-range dependency…
Read article →Prefix Caching and Prompt Reuse in Large-Scale Inference
How prefix caching reuses KV cache across requests: block hashing versus radix trees, prompt layout, cache-aware routing, eviction and offload tiers, …
Read article →Prompt Engineering Patterns, in depth: a catalogue keyed by the failure each pattern fixes, composing them in code, and choosing by ablation
Prompt engineering patterns as reusable solutions: input, output and control-flow families, the failure each pattern fixes, its cost and when it backf…
Read article →Quality-Cost-Latency Frontiers for Model Selection
How to choose an LLM configuration by measuring quality, cost per task and tail latency on your own workload, computing the Pareto frontier with confi…
Read article →Retrieval-Augmented Generation (RAG): Bridging the Gap Between a Model’s Training and Today’s News
Large Language Models (LLMs) have revolutionized human-computer interaction, offering unparalleled fluency in understanding and generating text.
Read article →RAG Evaluation: Recall, Faithfulness, Relevance, and Answer Completeness
How to evaluate a retrieval-augmented generation system component by component: evidence recall, context precision, claim-level faithfulness, answer r…
Read article →Ray Serve for LLM Systems: Routing, Autoscaling, and Failure Recovery
How to use Ray Serve as a programmable control plane for model replicas, request routing, autoscaling, and recovery.
Read article →Reinforcement Learning from Human Feedback (RLHF): The Secret Sauce That Made ChatGPT 'Helpful'
When Large Language Models (LLMs) first emerged, pre-trained on vast swaths of internet data, they demonstrated an astounding ability to generate fluent, coherent…
Read article →Rotary Position Embeddings (RoPE), in depth: the rotation, relative scores and what RoPE means for KV caches, cache shifting and quantization
How RoPE works from first principles and what it means for LLM serving: the frequency ladder and pairing layouts, runnable checks of relative-position…
Read article →Speculative Decoding: How Using a Tiny Model to 'Guess' Makes the Big Model 3x Faster
Large Language Models (LLMs) are incredibly powerful, but their generation speed often lags behind the demands of real-time interactive applications.
Read article →Speculative Decoding: Faster Generation Without Changing the Model
A production guide to speculative decoding: the acceptance rule, expected speedup, draft strategies, batch-size limits, KV rollback, and the metrics t…
Read article →System One vs System Two Models: Using a Fast Typed Decision Model Alongside LLM Generation in One Pipeline
Designing a two-speed AI pipeline with TypeSafe's Jev for typed decisions and an LLM for generation: gate, filter, verify and escalate patterns w…
Read article →The Hallucination Problem: Why LLMs Lie and How 'Fact-Checking' Layers Are Being Built
Large Language Models (LLMs) possess an almost uncanny ability to generate fluent, coherent, and seemingly authoritative text.
Read article →The LLM as the Ultimate Compiler: From Natural Language to Executable Code
A traditional compiler is a marvel of engineering: a program that meticulously translates human-readable source code (like C++, Python, or Java) into the precise…
Read article →Triton Inference Server: Model Backends, Scheduling, and Dynamic Batching
A systems guide to Triton model repositories, backend boundaries, dynamic batching, ensembles, and production controls.
Read article →Jev and System One Models: TypeSafe AI's Typed Decision Model Explained
How TypeSafe's Jev works: state plus typed Choice, Score and Noul questions, calibrated probabilities, the exact confidence formulas, RLCD, limit…
Read article →Understanding Tokenization: Why 'Apple' Is One Token But 'antidisestablishmentarianism' Is Many
Before a Large Language Model (LLM) can perform its magic—generating text, answering questions, or translating languages—raw human text must first be converted into…
Read article →Vision-Language Models (VLM): How Transformers 'See' and Describe Images in Real-Time
First-generation Large Language Models (LLMs), while revolutionary in their command of language, were fundamentally limited by their single-modal nature.
Read article →vLLM Architecture: Continuous Batching, KV Cache, and Production Serving
How continuous batching, paged attention, and KV-cache management turn a language model into a predictable production service.
Read article →Zero-Shot vs. Few-Shot Learning: Why the Best Models Don't Need Training Anymore
For decades, the bedrock of machine learning was a simple truth: for every new task, you needed a large, meticulously labeled dataset, followed by extensive…
Read article →What an 8 GB GPU Actually Ran: Local Model Results
Explore real GPU measurements: 25/48 answers were correct and 48/48 were valid JSON. Compare latency, correctness and recorded adapter-training outcomes.
Read article →4-Bit or 8-Bit? Measure What Compression Costs
Explore rounding error on synthetic numbers and learn why model quality and inference speed require separate experiments.
Read article →LoRA and QLoRA: What Actually Gets Trained?
Inspect frozen weights and trainable adapters, then calculate how rank changes the parameter budget without pretending to train a model.
Read article →Why a Model Fits in VRAM but Still Runs Out of Memory
Separate weight storage from context, cache and runtime overhead, then explore a transparent inference-memory budget.
Read article →How Quantization Helps an 8 GB GPU Run Larger Models
See where lower precision saves GPU memory, why smaller weights do not guarantee faster inference, and how to evaluate quality before choosing a format.
Read article →