Deduplication
MinHash + LSH near-dedup at scale. Removes memorization risk + wasted compute. Standard in modern pretraining.
Advertisement
Quality classifier
Fine-tuned model scores text quality. Retain top-K %. FastText or small BERT typical. Reduces low-quality content.
Advertisement
PII filter
Regex + NER for emails, phone, SSN. Redact or drop. Privacy + legal + memorization prevention.