All 80 articles, sorted alphabetically
Coalesced Memory Access
Consecutive threads read consecutive addresses = one transaction.
Read article →CoreWeave
Fastest H100 availability. IPO 2025.
Read article →Ada Lovelace / Blackwell architecture
Deep-dive on NVIDIA Hopper→Blackwell arc: SM redesign, FP8/FP4, memory bandwidth, NVLink generations, GB200 Grace-Blackwell.
Read article →Admission Control for LLM Serving
Reject requests to protect SLOs.
Read article →AI Gateway Overview
AI gateway: central proxy for LLM traffic.
Read article →AMD Instinct GPUs
How AMD Instinct GPUs (MI300X) compete with NVIDIA for AI workloads.
Read article →GPU Architecture Overview
How GPUs achieve massive parallelism through streaming multiprocessors, warps, and thousands of cores. The architecture that powers modern AI.
Read article →GPU asynchronous copy and software pipelining
Deep-dive on the asynchronous global-to-shared copy pipeline that fast GPU kernels are built on: cp.async streams tiles directly into a multi-stage sh…
Read article →GPU Batching Strategies Overview
Overview of batching approaches for LLM serving.
Read article →Chunked Prefill
Split long prefills into chunks.
Read article →Collective Communication Overlap
Overlapping computation with collectives.
Read article →GPU-Enabled Containers
How to run GPU workloads in Docker containers via nvidia-container-toolkit.
Read article →Continuous batching
Deep-dive on continuous (in-flight) batching for LLM inference: the scheduling technique that forms a fresh batch every decode iteration, letting fini…
Read article →GPU Datacenter Cooling Overview
Cooling GPU datacenters: air vs liquid.
Read article →GPU Cost Optimization
How to reduce GPU costs: rightsizing, spot, reserved, MIG, off-peak scheduling.
Read article →GPU CUDA Programming
The CUDA programming model: how kernels launch threads organized into blocks and grids, and the __global__/__device__ function distinction.
Read article →CUDA streams and graphs -- overlap and launch-overhead elimination
Deep-dive on CUDA streams and graphs: streams as ordered work queues enabling concurrency and copy-compute overlap, events for cross-stream synchroniz…
Read article →GPU Datacenter Deployment
The physical realities of GPU datacenters: 700W+ per GPU, liquid cooling, high-density networking, and the constraints that shape modern AI infrastruc…
Read article →GPU Datacenter Power Requirements
Powering GPU datacenters: MW scale.
Read article →Expert Parallelism
How expert parallelism distributes MoE experts across GPUs for efficient MoE training.
Read article →GPU Fine-Tuning Costs
How to estimate GPU fine-tuning costs based on model, data, method.
Read article →FlashAttention architecture
Deep-dive on FlashAttention: how tiling queries, keys, and values into on-chip SRAM blocks and using an online (streaming) softmax computes exact atte…
Read article →GPU Sharing Strategies Overview
Overview of GPU sharing approaches.
Read article →GPUDirect
How NVIDIA GPUDirect enables direct GPU access to storage and network without CPU involvement.
Read article →GPUDirect Storage architecture
Deep-dive on GPUDirect Storage: the cuFile API and kernel driver, peer-to-peer PCIe DMA from NVMe and NVMe-oF into GPU HBM, the CPU bounce buffer it e…
Read article →NVIDIA H100
What makes NVIDIA H100 the workhorse of modern AI training and inference.
Read article →H100 SM architecture
Deep-dive on H100 SM: warp scheduler, CUDA cores, 4th gen tensor cores, shared memory, TMA async copy, cluster launch, FP8/INT8.
Read article →GPU HBM architecture
Deep-dive on GPU HBM tier: bandwidth, banks, coalescing, L2/L1 caches, roofline model, async copy (TMA), memory-bound tuning.
Read article →LLM Inference Optimization Overview
Techniques for LLM inference optimization.
Read article →InfiniBand + NVLink architecture
Deep-dive on GPU cluster fabric: NVLink intra-node, InfiniBand inter-node, GPUDirect RDMA, NCCL + SHARP collectives, topology-aware placement.
Read article →GPU kernel fusion -- fewer kernels, less memory traffic
Deep-dive on GPU kernel fusion: the memory-bound HBM-traffic problem, fusing operations to keep intermediates on-chip (registers/SRAM), elementwise/ep…
Read article →LLM Canary Deployment
Canary: send small % to new LLM.
Read article →LLM Deployment Pattern Overview
Deployment patterns for LLM serving.
Read article →LLM FinOps
FinOps discipline for LLM apps.
Read article →LLM Guardrails
Guardrails for LLM production.
Read article →LLM Load Testing
Load-testing LLM serving stacks.
Read article →GPU Serving Architecture for LLMs: The Full Stack
A 2500-word walkthrough of a modern GPU serving stack for LLMs: ingress, mesh, router, engine, autoscaler, pod, fabric, object store, and observabilit…
Read article →LLM Serving Stacks
The main LLM serving stacks and their trade-offs.
Read article →Lookahead Decoding
How Lookahead Decoding speeds LLM generation without separate draft model.
Read article →GPU Memory Hierarchy
How GPU memory is organized: registers, shared memory, L1/L2 cache, HBM. Bandwidth matters more than latency.
Read article →NVIDIA MIG architecture
Deep-dive on Multi-Instance GPU: compute and memory slices, GPU Instance and Compute Instance hierarchy, profile geometry on A100/H100, Kubernetes GPU…
Read article →MoE All-to-All Communication
All-to-all: token shuffle for MoE.
Read article →MoE Serving Overview
Serving Mixture-of-Experts models.
Read article →NCCL collectives
Deep-dive on NCCL collective communication: all-reduce and collective ops, bandwidth-optimal ring and latency-optimal tree algorithms, NVLink/NVSwitch…
Read article →nvidia-smi
How nvidia-smi provides quick GPU status, and the key views for debugging.
Read article →NVLink and NVSwitch architecture
Deep-dive on NVLink/NVSwitch: link anatomy and memory semantics, the non-blocking crossbar, SHARP in-switch reductions, NCCL topology mapping, link-fl…
Read article →GPU occupancy architecture
Deep-dive on GPU occupancy: the ratio of active warps per SM to the hardware maximum, determined by whichever per-SM resource — registers per thread, …
Read article →P2P KV Cache Transfer
Peer-to-peer KV cache transfer between GPUs.
Read article →Paged KV cache architecture
Deep-dive on PagedAttention-style KV cache management: physical block pools and block tables, copy-on-write prefix caching, continuous batching under …
Read article →PCIe and GPU
How GPUs connect to CPU via PCIe, and why PCIe bandwidth matters for GPU workloads.
Read article →Prefill/Decode Disaggregation Architecture in Depth
A 2500-word walkthrough of prefill/decode disaggregation: router, prefill pool, KV transfer, decode pool, continuous batching, independent SLOs, autos…
Read article →Pipeline Parallelism
How pipeline parallelism splits model layers across GPUs, with micro-batches for utilization.
Read article →GPU Preemption Handling
Handling GPU preemption in training + serving.
Read article →GPU Profiling
How to profile GPU workloads: Nsight Systems for timeline, Nsight Compute for kernel-level detail.
Read article →Ring Attention
How ring attention enables million-token context by distributing sequence across GPUs.
Read article →RLHF Pipeline on GPU
Full RLHF pipeline: SFT → RM → PPO.
Read article →GPU Scheduling
How multiple workloads share a GPU: time-slicing, MPS (Multi-Process Service), and MIG (Multi-Instance GPU).
Read article →GPU Scheduling With Slurm
How Slurm schedules GPU jobs on HPC clusters via GRES (Generic RESources).
Read article →Multi-Region LLM Serving
Deploy LLM across regions.
Read article →GPU shared memory architecture
Deep-dive on GPU shared memory: why the memory hierarchy makes on-SM shared memory the linchpin of performance, the 32-bank structure and how warp-wid…
Read article →SLO-Aware Scheduling
Schedule LLM requests to meet SLOs.
Read article →Speculative Decoding Architecture in Depth
A 2500-word walkthrough of speculative decoding: draft/target models, generate + verify, accept + reject sampling, correctness proof, acceptance rate,…
Read article →Static Batching
Static batching: fixed batch size per iteration.
Read article →StreamingLLM
How StreamingLLM enables efficient inference on very long conversations.
Read article →Switch Transformer Architecture
Google Switch Transformer: foundational MoE.
Read article →Tensor core architecture
Deep-dive on GPU tensor cores: WMMA/MMA, tile size, accumulator dtype, input dtype, sparsity, multi-buffering, kernel selection.
Read article →GPU Tensor Cores
How tensor cores accelerate matrix multiplication for deep learning, precision modes (FP16, BF16, FP8, INT8), and when they kick in.
Read article →GPU Time Slicing
Software GPU time slicing.
Read article →GPU Tensor Memory Accelerator (TMA): Architecture Deep-Dive
How Hopper's Tensor Memory Accelerator moves tensor tiles asynchronously via descriptors and mbarriers, freeing compute warps…
Read article →Google TPU
How Google TPU compares to GPU for training and inference on Google Cloud.
Read article →NVIDIA vGPU
vGPU: virtualization + graphics + compute.
Read article →GPU warp specialization architecture
Deep-dive on GPU warp specialization: splitting a thread block's warps into producer warps that drive cp.async/TMA copies and…
Read article →ZeRO sharding architecture
Deep-dive on ZeRO: the 16-bytes-per-parameter memory math, stages 1/2/3, reduce-scatter and all-gather volume, prefetch overlap, hybrid sharding, ZeRO…
Read article →Latitude.sh
Bare metal cloud. Reserved + on-demand GPUs.
Read article →Optimizer State Offloading to CPU/NVMe
Move optimizer state off GPU. Trade PCIe bandwidth.
Read article →GPU Register Pressure
Registers per thread. Limits threads per SM.
Read article →SambaNova RDU
Reconfigurable Dataflow Unit. Alternative arch.
Read article →Second-Hand GPU Market
Enterprise churn feeds resale.
Read article →SHARP
Aggregate on switch. Faster all-reduce.
Read article →Warp Scheduling on GPU
32 threads = 1 warp. SIMT execution model.
Read article →