Why it matters

Data dedup reduces memorization + boosts efficiency. Standard step in pretraining.

Advertisement

The architecture

MinHash signatures.

LSH bucketing.

Remove near-duplicates.

Training data dedupMinHashsignature per docLSH bucketingcandidate pairsRemoveduplicatesBroder MinHash 1997; SlimPajama, FineWeb-Edu use LSH pipelines
Training data dedup.
Advertisement

How it works end to end

MinHash + LSH.

SlimPajama.

FineWeb-Edu.