Data Collection and Cleaning for Large Language Model Pretraining at Web Scale

Training a large language model isn’t about throwing more GPUs at the problem anymore. It’s about data. The difference between a model that understands context and one that just repeats garbage comes down to what you feed it - and how clean it is. Companies like OpenAI, Meta, and Apple now spend more time and money cleaning data than building architectures. In 2025, the most advanced models are trained on datasets of 10-13 trillion tokens, pulled from billions of web pages. But 90% of that raw data? It’s useless. Or worse - harmful.

Where Does the Data Even Come From?

Most large language models start with Common Crawl, a non-profit archive that’s been scraping the web since 2012. It holds over 25 billion web pages - text from blogs, forums, news sites, product pages, and spammy SEO dumps. It’s the closest thing we have to a mirror of the public internet. But it’s messy. A single crawl might include duplicate product listings from Amazon, Reddit threads with 500 identical replies, and entire websites copied verbatim across dozens of domains.

Other sources include RefinedWeb (a cleaned version of Common Crawl), GitHub repositories, Wikipedia, books from Project Gutenberg, and curated academic datasets. Some companies also license proprietary data - legal documents, medical records (with consent), or customer support logs. But even then, you’re not just copying and pasting. You’re building a pipeline.

The Four-Stage Cleaning Pipeline

There’s no single tool that fixes everything. Cleaning web-scale data is a multi-stage process, like filtering water through layers of sand, charcoal, and membranes.

1. URL and Domain Filtering - First, you remove entire domains known for low quality: spam sites, link farms, adult content, and sites that auto-generate text. This alone cuts 40-60% of the data. Tools like the Dolma pipeline use heuristics: if a page has fewer than 100 words or more than 50% non-alphabetic characters, it’s tossed.
2. Document Quality Scoring - Not all pages from good domains are good. You run each document through a lightweight classifier - often a small transformer model trained to recognize “high-quality” text. It looks for things like sentence structure, lexical diversity, and whether the text reads like a human wrote it. Pages with repetitive phrases (“buy now,” “click here,” “best price guaranteed”) get flagged. This removes another 25-35%.
3. Deduplication - This is where things get technical. Duplicate content doesn’t just waste compute - it causes “double descent,” where the model memorizes exact phrases instead of learning patterns. Simhash fingerprints (64-bit hashes of text chunks) are used to find near-identical paragraphs. One team reduced deduplication time on a 50TB corpus from 14 days to 9 hours using this method. Some teams do paragraph-level deduplication instead of document-level - it’s slower but improves downstream performance by 7.3%.
4. Safety and Toxicity Filtering - This is the hardest part. You don’t want your model spitting out hate speech, misinformation, or illegal content. But over-filtering kills nuance. A 2024 survey of 127 ML engineers found 68% struggled with false positives - especially in medical or legal text. One model flagged “abortion” as toxic because it appeared in anti-choice forums. Others flagged scientific terms like “chemical weapons” because they were mentioned in news reports. The solution? Use multiple filters: rule-based lists for obvious cases, then a reward model trained to score safety on a sliding scale. Meta’s research suggests you can retain 30-40% of the data after this stage without hurting performance.

Why More Data Isn’t Always Better

You might think: “If 13 trillion tokens are good, 20 trillion must be better.” Not true. Apple’s BETR method (Benchmark-Targeted Ranking), published in November 2024, showed something surprising: models trained on smaller, smarter datasets outperformed those trained on massive, unfiltered ones. BETR didn’t just filter out bad data - it picked documents that looked like the kinds of questions and answers used in evaluation benchmarks. The result? A 2.1x improvement in performance with the same compute. Larger models (70B+ parameters) actually benefit less from aggressive filtering - they can handle noise better. But smaller models? They need precision.

This flips the old assumption: scaling up isn’t the answer anymore. Targeting is.

Synthetic Data: The New Wild Card

When real data is scarce - say, for high-level math reasoning or legal analysis - you generate it. DeepSeek-R1’s “cold-start” method uses reinforcement learning to create synthetic chain-of-thought examples. A model generates a math problem, then a second model checks the reasoning. Only examples with correct logic and clear steps get kept. Rejection sampling filters out 90% of the output. The result? High-quality, verified data that didn’t exist on the web.

But there’s a catch. Synthetic data can introduce hallucinations that look real. If you train on too many generated examples, the model starts believing its own fiction. That’s why teams use hybrid approaches: 70% real, 30% synthetic. Gartner predicts 65% of enterprise LLMs will use synthetic data by 2026 - up from 25% in 2024.

The Hidden Costs

Most people don’t realize how expensive this process is. Building a pipeline takes 3-6 months. You need distributed systems experts (Spark, Flink), NLP engineers, and cloud architects. One Reddit user spent eight weeks just removing forum spam - it made up 22% of their raw corpus. Copyright filtering eats up 35-40% of resources, yet adds little to model performance. And GDPR compliance? Processing user deletion requests can consume 15% of your pipeline’s bandwidth.

The Dolma dataset, which processed 3.8 trillion tokens, required 50 specialized nodes running for months. The learning curve? Four to six months just to get new engineers up to speed.

Legal and Ethical Landmines

The EU AI Act, effective February 2025, requires full documentation of every piece of training data - where it came from, who owns it, whether consent was obtained. That’s a nightmare for web-scraped data. Law firms like DLA Piper estimate this adds 20-30% more preprocessing work. Fenwick & West warns that up to 25% of existing datasets could be legally questionable due to copyright violations.

And then there’s privacy. Princeton researchers developed Min-K% Prob, a method that can detect if a specific sentence was in a model’s training data - just by analyzing its probability scores. That means even if you delete a website, the model might still remember it. This isn’t theoretical. In late 2024, a model was shown to regurgitate entire patient records from anonymized medical forums.

What Works Today - And What Doesn’t

Here’s what the best teams are doing in 2025:

• Using simhash for deduplication - fast, scalable, accurate.
• Training small models to score quality before using big ones - saves millions in compute.
• Aligning data to benchmarks, not just volume - Apple’s BETR is now a standard.
• Keeping synthetic data under 30% - enough to fill gaps, not enough to distort.
• Documenting everything - not just for compliance, but for debugging.

And here’s what fails:

• Trying to clean everything with rules - language is too messy.
• Filtering out all “controversial” topics - you end up with a bland, useless model.
• Ignoring multilingual data - 100+ languages need separate identification pipelines.
• Thinking data cleaning is a one-time task - models evolve, so must your data.

The Future Is Data-Centric

McKinsey found that in 2024, 57% of companies spent more on data preparation than model development. That’s up from 38% in 2022. The era of “bigger models” is over. The next leap won’t come from a new attention mechanism - it’ll come from better data.

By 2027, Gartner predicts 80% of enterprise LLMs will use task-specific corpora: a model for legal advice won’t train on Reddit threads. It’ll train on court rulings, contracts, and legal FAQs. A medical assistant won’t read Wikipedia - it’ll read peer-reviewed journals and clinical notes.

The bottleneck isn’t compute. It’s curation. The teams that win aren’t the ones with the most GPUs. They’re the ones with the cleanest data.