DataLoader Pipeline: Bounded Prefetch and Compute Stalls

DataLoader Pipeline: Bounded Prefetch and Compute Stalls: experiment guide

This lab schedules a finite stream of numbered batches through loading and serial computation. Number of batches selects one through twenty-four jobs. Loader workers selects zero through eight independent loader slots. A positive worker count permits loading to overlap computation; zero selects a synchronous baseline that loads and computes each batch before beginning the next. Assumed load duration per batch and Assumed compute duration per batch supply explicit, positive durations in milliseconds. They describe a hypothetical workload, not measurements from a CPU, GPU, storage system or framework. Every batch has the same selected load duration and the same selected compute duration.

Worked details 1

Total prefetch slots bounds all dispatched batches that are loading or ready but have not started computation. A batch releases its slot when computation starts, rather than when computation finishes. The currently computing batch is excluded from this capacity. This is a global teaching-model capacity, not a per-worker prefetch factor from a particular library. Increasing the capacity can permit more loader concurrency, but it cannot create additional workers or accelerate the serial compute stage. With zero workers, only one synchronous load is permitted and the prefetch setting is intentionally inactive. The result explicitly states this conditional behavior.

Worked details 2

At each event time, completed computation is recorded, completed loads enter the ready queue, the lowest-numbered ready batch begins computation if that stage is free, and idle loader slots take undispatched batches while capacity allows. Equal load durations and ordered dispatch ensure the batch order is preserved. Simultaneous loader completions are resolved by worker index; ready batches are ordered by batch number. Dispatch chooses the lowest available worker index. The event inspector shows worker assignments, the ready queue, computing batch, completed count and outstanding count. These states partition the dispatched jobs, so no batch can disappear or be counted in two stages at once.

Worked details 3

Use the opening settings: eight batches, two workers, two prefetch slots, six milliseconds loading and four milliseconds computation. Batches zero and one load together from zero to six milliseconds. Batch zero computes from six to ten, while batch two loads from six to twelve. Batch one computes from ten to fourteen. The final compute interval ends at thirty-eight milliseconds. Computation is busy for thirty-two milliseconds and idle for six, giving approximately 84.210526 percent utilization over the entire finite run. The initial fill is included. There is no warm-up subtraction or hidden efficiency multiplier.

Worked details 4

Change workers to zero: the schedule becomes eight repetitions of six milliseconds loading plus four computing, totaling eighty milliseconds with forty-eight compute-idle milliseconds. Reset and choose one worker instead: the total becomes fifty-two milliseconds, because loading can overlap computation but does not supply batches fast enough to keep it continuously busy. Reset and reduce prefetch slots to one while keeping two workers: the same fifty-two millisecond span demonstrates a capacity limit that prevents both loaders from working together. Restore two slots and increase load duration to twelve: the span is fifty-six milliseconds with twenty-four compute-idle milliseconds. Reset and increase compute duration to eight: the span becomes seventy milliseconds. These are exact consequences of the declared scheduling rules, not hardware speedup claims.

Worked details 5

Set Number of batches to one to isolate startup: the total is always load duration plus compute duration, regardless of extra workers. The visual uses blue load intervals and brown compute intervals, with batch labels and a moving inspection line. The table gives exact interval endpoints and each batch state as a text alternative. Play maps the complete hypothetical schedule onto sixty teaching seconds; teaching time and simulated milliseconds are separate units. Pause freezes the inspection line. Next event advances to the next scheduler event, Seek pauses at any selected teaching time, and Restart returns to the start while preserving inputs. Reset parameters restores the opening workload. Speed selects quarter, normal or four times the teaching rate without changing the schedule.

Worked details 6

Changing any model input rebuilds the schedule and pauses playback. Blank values, fractional worker or batch counts, zero durations and values outside the displayed bounds are rejected; Play and Next event remain disabled until the input is valid. Real pipelines also include tokenization variability, disk contention, multiprocessing overhead, device transfers, pinned memory, variable batch sizes and failures. None is modeled here. More workers need not help a real workload, and this lab does not invent an overhead curve to suggest an optimum worker count. Compare the explicit queue and interval changes before drawing a conclusion about the bottleneck under these limited assumptions.