A training step is only as fast as the batch it is given. You can buy the fastest accelerator on the market, write a perfect attention kernel, and still watch utilization sit at forty percent — because the dataloader cannot turn raw text into ready tensors quickly enough to keep the device fed. The pipeline that sits between a directory of text files and a GPU-resident batch is unglamorous but decisive: it tokenizes, it packs documents end-to-end into fixed-length sequences, it shuffles, it shifts inputs against targets for next-token prediction, and it does all of this on the CPU, one step ahead of the device, forever. This piece walks that pipeline from first principles: what tokenization produces, how token ids are stored in memory-mapped binaries, the arithmetic of packing efficiency and why separators matter, the input/target shift, how to shard and stream a corpus too big for RAM, and the throughput math that decides whether your expensive accelerator ever runs at full speed. Two worked examples — one for packing, one for throughput — make the numbers concrete.
Why the dataloader is a first-class citizen
Training a language model is a producer–consumer system. The consumer is the accelerator running forward and backward passes; the producer is the CPU-side dataloader assembling the next batch. If the producer is slower than the consumer, the accelerator stalls waiting for data, and every stalled millisecond is money spent on idle silicon. The single most important invariant of the whole pipeline is therefore this: the dataloader must stay ahead of the device.
That sounds obvious until you count the work. Between ‘raw text on disk’ and ‘a tensor of token ids in device memory’ lie tokenization, packing into fixed-length windows, shuffling, collating into a batch, casting dtypes, pinning host memory, and copying across the PCIe bus. Do all of that naively and synchronously, inside the training loop, and you serialize the CPU work with the GPU work — the device computes, then waits, then computes, then waits. The art of the dataloader is to overlap that CPU work with the previous step’s device work so the batch is already sitting in memory the instant the optimizer finishes. Everything below is in service of that overlap.
The pipeline, end to end
Strip the pipeline to its stages and it is a short assembly line: raw text → tokenize → pack → shuffle → batch → shift → to device. Each stage has a shape, and tracking the shapes is the fastest way to understand the whole thing.
Tokenize turns a string into a 1-D stream of integer ids: a document of D characters becomes some T tokens, T ≈ D / 4 for English with a byte-pair vocabulary. Pack concatenates many such streams, separated by a special token, into one long flat array and slices it into fixed-length windows of L tokens (the context length). Shuffle randomizes the order in which those windows are visited. Batch stacks B windows into a [B, L] integer matrix. Shift derives targets by offsetting inputs by one position, giving an x: [B, L] / y: [B, L] pair. The final copy moves that pair onto the accelerator. The rest of this article is each of these stages examined closely, plus the storage format and the throughput budget that ties them together.
Tokenization: from characters to integer ids
Models do not consume text; they consume integers that index into an embedding table. Tokenization is the map from a string to a sequence of those integers. Modern LMs use subword tokenizers — byte-pair encoding (BPE), WordPiece, or Unigram — which sit between two bad extremes. Character-level tokenization gives a tiny vocabulary but enormous sequences (one token per byte); word-level tokenization gives short sequences but a huge, brittle vocabulary that cannot spell an unseen word. Subword tokenization keeps frequent words whole, splits rare words into reusable pieces, and never fails on novel input because it can always fall back to bytes.
The output is a vector of ids, each in [0, V) where V is the vocabulary size — commonly V ≈ 32000 to 128000. A useful rule of thumb for English is roughly 4 characters or 0.75 words per token, so a 1000-word document is about 1300 tokens. That ratio, the fertility of the tokenizer, directly sets how many tokens your corpus contains and therefore how long training takes: a tokenizer that emits 5% more tokens for the same text makes every epoch 5% more expensive. The tokenizer is trained once, frozen, and shared by every stage that follows.
Storing tokens: memory-mapped binaries and dtype choice
Once tokenized, a corpus is just a very long list of integers, and the cheapest possible representation is a flat binary file of fixed-width integers — no JSON, no delimiters, no per-record overhead. The dtype is chosen to be the smallest that holds every id. If V ≤ 65536 the ids fit in uint16 (2 bytes); above that you need uint32 (4 bytes). The difference is not academic: a 300-billion-token corpus is 600 GB as uint16 but 1.2 TB as uint32, which is why keeping the vocabulary under 65,536 is a real storage lever.
These .bin files are read via memory mapping (np.memmap or an equivalent). A memmap does not load the file into RAM; it maps the file into the process’s address space and lets the operating system’s page cache pull in 4 KB pages on demand. So a training job can ‘open’ a 600 GB token file on a machine with 32 GB of RAM and read arbitrary windows of it as if it were an in-memory array, paying only for the pages it actually touches. Random access to a window at offset i is a single slice — tokens[i : i+L] — and the OS handles the caching. This is the storage substrate that makes packing and shuffling over a huge corpus cheap.
Document packing: concatenation with separators
Documents come in wildly different lengths — a tweet is 20 tokens, an article is 3000. The context window L is fixed. The naive approach, one document per sequence with padding, wastes catastrophic amounts of compute: pad every document out to L and a corpus of mostly-short documents becomes mostly padding tokens, on which the model does full attention and FFN work for zero learning signal. Packing fixes this by treating the corpus as one continuous token stream: concatenate every document’s ids end-to-end, insert a separator token between documents, and then slice the resulting flat array into contiguous windows of exactly L tokens.
The separator is usually the end-of-sequence token (<eos>, or <|endoftext|>). It does two jobs. First, it is a learnable boundary marker: the model sees it between documents and learns that content does not flow across it, which is also how the model learns to stop generating. Second, it lets you cheaply reconstruct where documents begin. A packed stream looks like [doc1 tokens] <eos> [doc2 tokens] <eos> [doc3 tokens] <eos> ..., and a window of length L is just any L-token slice of that — it may start mid-document, span a boundary, and end mid-document. Packing turns the ragged-length problem into a single-array slicing problem, and it is why production LM training is almost never padded.
The math of packing efficiency
Define packing efficiency as the fraction of processed tokens that carry real signal: η = useful_tokens / processed_tokens. Under per-document padding, if documents have mean length μ and you pad each to L, then η_pad = μ / L. For μ = 300 and L = 2048 that is η_pad ≈ 0.146 — you are paying full transformer cost and roughly 85% of it is spent on padding. Worse, since attention is O(L^2), the wasted work is disproportionate at long context.
Per-document padding: η_pad = μ / L
Packing: η_pack = L / (L + s) where s = separators per window
With 1 separator token per document and packing:
overhead per doc = 1 token, useful per doc = μ tokens
η_pack = μ / (μ + 1) → for μ = 300, η_pack ≈ 0.997Packing’s only waste is the one separator token per document, so efficiency is μ / (μ + 1) — essentially 100% for any realistic document length. The lift from 0.146 to 0.997 is a 6.8× reduction in wasted compute for this configuration. That single factor is often the difference between a training run finishing this week or next, which is why the very first thing to check on a slow run is whether the data is actually packed.
The input/target shift for next-token prediction
A language model is trained to predict the next token, so every position’s label is simply the token that follows it. Given a packed window w = [t_0, t_1, ..., t_L] of L+1 tokens, the training pair is formed by a one-position shift:
x = w[:-1] = [t_0, t_1, ..., t_(L-1)] # inputs, shape [L]
y = w[1:] = [t_1, t_2, ..., t_L] # targets, shape [L]
so target[i] = input[i+1] — at each position, predict the next token.The model produces logits of shape [L, V]; position i’s logits are scored against y[i] with cross-entropy, and a causal mask ensures position i can only attend to positions ≤ i, so it never cheats by looking at its own answer. Two practical notes. First, you fetch L+1 tokens per window and produce L supervised positions from them — a detail that quietly changes your stride. Second, if you want the model not to be penalized for predicting the token after an <eos> (i.e., predicting across a document boundary), you set those target positions to an ignore index so the loss skips them. Many pipelines skip this refinement and let the model learn the boundary from the <eos> token itself; both are defensible, but you should know which one you are doing.
Worked example: packing a small shard
Make it concrete. Suppose a shard has 10,000 documents with mean length μ = 300 tokens, a context length L = 2048, and one <eos> separator per document.
Total tokens after packing:
content = 10,000 × 300 = 3,000,000
separators= 10,000 × 1 = 10,000
packed = 3,010,000 tokens
Windows of length L+1 = 2049 (need L+1 to form the shift):
n_windows = floor(3,010,000 / 2049) ≈ 1469 windows
(remainder 1,038 tokens dropped, or carried to the next shard)
Compare per-document padding at L = 2048:
processed = 10,000 × 2048 = 20,480,000 tokens
useful = 3,000,000 tokens
η_pad = 3,000,000 / 20,480,000 ≈ 0.146
Packing processes 3,010,000 tokens for the same 3,000,000 useful:
η_pack = 3,000,000 / 3,010,000 ≈ 0.997
speedup = 20,480,000 / 3,010,000 ≈ 6.8× less computeThe packed shard yields 1469 training windows from three million tokens, versus needing to grind through twenty million padded tokens for the same signal. Note the small honesty of the arithmetic: forming the shift needs L+1 tokens per window, and the trailing 1038 tokens that do not fill a window are either dropped or stitched onto the next shard’s stream so nothing is silently lost.
Shuffling: why order is a hyperparameter
Stochastic gradient descent assumes each batch is a roughly unbiased sample of the data distribution. A packed token file violates that badly: contiguous windows come from adjacent documents, so a batch read in file order might be all one author, one topic, or one language. Training on such correlated batches raises gradient variance, can induce oscillation, and in the worst case lets the model overfit to whatever it saw most recently. Shuffling restores approximate independence, and how you shuffle depends on whether the data fits in memory.
Three levels, cheapest to strongest. Index shuffle: with a memmap you never move the data — you shuffle a list of window start offsets and read windows in that permuted order, so a full random permutation costs only an array of indices. Shard shuffle: with many shard files, shuffle the order of shards each epoch, and shuffle windows within the shard currently open. Buffer (reservoir) shuffle: for pure streaming where you cannot seek, keep a fixed-size buffer of K windows, and each step emit a random one and refill it from the stream — an approximate shuffle whose quality rises with K. The right default for a local memmap corpus is the index shuffle: it is a perfect permutation for free. Reserve buffer shuffling for the genuinely unseekable streaming case.
Batching, collation, and tensor shapes
A batch is B windows stacked into one tensor. Because packing already made every window exactly L tokens, collation is trivial — there is no ragged padding to reconcile, you simply stack them. The training pair is x: [B, L] and y: [B, L], both integer tensors, and the number of supervised tokens per step is B × L. That product is the unit almost every training budget is denominated in: a run described as ‘300B tokens’ means it took 300e9 / (B × L) optimizer steps.
The global batch size is B × L × grad_accum × world_size tokens when you use gradient accumulation across micro-batches and data-parallel replicas across devices. The dataloader must therefore hand each replica a disjoint slice of the shuffled index stream so no two devices train on the same window in the same step — typically done by having replica r of world_size take every world_size-th index starting at r, or by pre-partitioning shards across replicas. Getting this sharding wrong silently duplicates data and quietly hurts the model; it is one of the most common distributed-training bugs, and it produces no error, only a worse loss curve.
On-the-fly vs pre-tokenized
There are two philosophies for where tokenization happens. Pre-tokenized (offline): run the tokenizer once over the whole corpus, write .bin shards of ids, and let the training job only read integers. On-the-fly (online): keep raw text, and tokenize inside the dataloader as batches are requested. The trade is compute-versus-flexibility.
Pre-tokenizing is the default for serious pretraining because tokenization is surprisingly expensive — it is CPU-bound string processing — and you do not want to pay it on every epoch. Do it once, and every subsequent read is a cheap integer slice; packing and shuffling operate on compact fixed-width arrays; and throughput becomes predictable. The costs are storage (the .bin files) and rigidity: changing the tokenizer, the separator scheme, or the context length means re-tokenizing. On-the-fly tokenization avoids the storage and stays flexible — useful for rapidly changing data, augmentation, or fine-tuning on a small set — but it puts a heavy CPU task directly on the hot path, which is exactly where the dataloader can fall behind the accelerator. The common compromise: pre-tokenize the large static pretraining corpus, tokenize small dynamic fine-tuning sets on the fly.
Sharding and streaming a corpus that will not fit
Real pretraining corpora are terabytes; they do not fit in RAM and often not on one disk. The answer is sharding: split the token stream into many moderate files (say 0.1–1 GB each, shard_00000.bin ... shard_NNNNN.bin). Shards make everything tractable. They can be produced in parallel by many tokenizer workers; they can be distributed across machines and across data-parallel replicas; they can be memory-mapped one at a time; and an interrupted run can resume by recording (shard_index, window_offset) rather than replaying from the start.
For data that is too large even to enumerate locally — served from object storage or a remote dataset — you stream: pull shards over the network, decompress, tokenize (if online), and feed a shuffle buffer, prefetching the next shard while the current one is consumed. The engineering worry with streaming is the same as always — the network and CPU stages must, in aggregate, sustain the device’s token appetite — plus a reproducibility worry: to make a run resumable and deterministic you must seed the shard order and the shuffle buffer and checkpoint your position in the stream. A streaming pipeline that cannot resume to the exact same data order after a crash is a debugging nightmare, so bake in the bookkeeping from day one.
Throughput: keeping the CPU ahead of the accelerator
Return to the governing invariant. Let the accelerator consume a step in t_gpu seconds and let the dataloader produce a batch in t_cpu seconds of wall-clock work. If the two run serially, each step costs t_gpu + t_cpu. If they overlap — the CPU building batch n+1 while the GPU computes on batch n — each step costs max(t_gpu, t_cpu). The entire goal of dataloader engineering is to reach that max, and then to ensure the max is t_gpu, not t_cpu.
The mechanisms that buy the overlap: multiple worker processes (num_workers > 1) so tokenization/collation runs in parallel and sidesteps the Python GIL; a prefetch queue so several batches are always ready ahead of the device; pinned (page-locked) host memory so the host-to-device copy can use DMA and run asynchronously on a separate CUDA stream, overlapping the transfer with compute; and persistent workers so you do not pay process startup every epoch. The rule of thumb for worker count is to provision enough that aggregate producer throughput exceeds device demand: num_workers ≥ ceil(t_cpu_single / t_gpu), then a little headroom. Past that point more workers only add memory and contention. If you have tuned all of this and the device still starves, the bottleneck has moved to disk or network I/O, and the fix is faster storage or better prefetching, not more workers.
Worked example: is the dataloader fast enough?
Suppose one training step processes a batch of B = 32 sequences at L = 2048, and the accelerator runs that step in t_gpu = 200 ms. The device therefore demands
tokens per step = B × L = 32 × 2048 = 65,536 tokens
device demand = 65,536 / 0.200 s ≈ 327,680 tokens/s
Suppose one dataloader worker sustains 60,000 tokens/s
(reading memmap windows + collation; pre-tokenized, so no tokenizer cost):
workers needed = ceil(327,680 / 60,000) = ceil(5.46) = 6 workers
with 6 workers = 360,000 tokens/s > 327,680 → device stays fed
with 4 workers = 240,000 tokens/s < 327,680 → GPU starves 27% of the timeThe reading is direct: at four workers the producer sustains only 240k tokens/s against a 328k demand, so the device sits idle roughly 1 - 240/328 ≈ 27% of each step — you are renting an accelerator and using three-quarters of it. Six workers push producer throughput past demand and the step time collapses back to the device’s 200 ms. Now flip in on-the-fly tokenization: if tokenizing drops single-worker throughput to 15,000 tokens/s, you would need ceil(327,680 / 15,000) = 22 workers to keep up — which is precisely why heavy pretraining pre-tokenizes and reads integers.
Common pitfalls and CPU-SLM implications
The failure modes cluster. Silent padding: forgetting to pack and training on 85% padding, the single most expensive mistake and one that shows up only as a mysteriously slow run. Off-by-one in the shift: fetching L instead of L+1 tokens, so the last position has no target or targets are misaligned by one — the loss will train but converge to something worse. Duplicated data across replicas: bad index sharding so every GPU sees the same windows, cutting your effective dataset by world_size. Wrong dtype: writing uint16 when V > 65536 silently wraps ids and corrupts the corpus. Non-resumable streaming: a crash 20 hours in that cannot resume to the same data order. Each of these is invisible to a smoke test and only visible in the loss curve or the wall-clock, which is why they are so pernicious.
For CPU-only SLM training, the balance shifts in an instructive way. When the ‘accelerator’ is itself the CPU, the compute cores and the dataloader cores compete for the same silicon, so you cannot naively spawn many workers — you budget cores between model compute and data preparation. That makes pre-tokenizing to a memmap uint16 binary almost mandatory: it removes tokenization from the hot path entirely, so the data workers do nothing but slice an mmap and stack a tensor, leaving the cores for the forward and backward passes. Packing matters even more, because a CPU has less compute to waste on padding. The whole transformer-math lesson applies in miniature: know your token budget (B × L per step), pack to keep efficiency near 1.0, store compactly, and size the producer so the consumer — whatever it runs on — never waits.