Shuffling
Uniform random. Standard default. Prevents temporal correlation. Especially important with multi-source data.
Advertisement
Bucketed shuffling
Group by length for efficient batching. Shuffle within buckets. Balance efficiency + randomness.
Advertisement
Curriculum learning
Easy → hard order. Motivated by human learning. Mixed evidence for LLMs. Some benefit on reasoning tasks.