AI training data: what are the biggest unsolved problems?
Whether you're building or investing in the AI training data market, our report covers everything you need to know: the players, the money, and what's coming next.
Summary
The biggest unsolved problem in AI training data is securing a continuous supply of high-quality, trustworthy information that developers can legally access and independently verify. The industry is getting better at processing data, but finding genuinely new, reliable knowledge is becoming harder.
High-quality public text may become a serious constraint on frontier AI development. Epoch AI estimated the usable stock at approximately 300 trillion tokens and projected that models could exhaust it between 2026 and 2032 under historical scaling trends.
More training data does not automatically produce smarter models. DataComp-LM showed that careful filtering and dataset composition can deliver competitive results with substantially less training compute, making the value of individual examples increasingly important.
Synthetic data offers a partial escape from scarcity. Large experiments found that mixing roughly 30% synthetic content with real text can improve training efficiency, but repeatedly replacing authentic information with generated material risks losing rare knowledge.
Data ownership is becoming a major competitive advantage. Anthropic's $1.5 billion copyright settlement and large publisher-licensing agreements show how expensive unclear acquisition rights can become, particularly for developers without the budgets of major AI laboratories.
Provenance remains surprisingly weak. Research examining nearly 4,000 datasets found substantial differences between dataset-level licenses and restrictions on the original material, making it difficult to establish whether training records can actually be reused commercially.
Privacy and security introduce problems that better filtering alone cannot solve. Researchers still cannot reliably prove that memorized private information has been removed, while experiments showed that as few as 250 malicious documents could introduce a specific training-data backdoor.
Even the data used to measure AI progress is unreliable in places. OpenAI's audits of SWE-bench Verified and SWE-bench Pro exposed contamination and defective evaluation tasks, raising questions about how much apparent model improvement represents genuine new capability.
The most valuable missing examples are often the hardest to collect: rare medical conditions, unusual software failures, regional languages, expert decisions and physical robot interactions. Human annotation and simulation help, but neither guarantees that training examples reflect reality.
Our conclusion is that the strongest long-term advantage will belong to companies capable of collecting rare, valuable information, documenting its origins and proving that models genuinely learn from it. The largest scraped dataset will not necessarily be the best one.
Written by the New Market Pitch teamWe track new markets so founders and investors can move faster. About us
Is AI running out of training data?
AI is getting dangerously close to exhausting the supply of high-quality public text that can improve frontier models, although the internet itself keeps producing enormous amounts of information.
Epoch AI estimated in June 2024 that the effective stock of human-generated public text was approximately 300 trillion tokens after adjusting for quality and repetition. Its researchers projected that frontier models could fully use that stock between 2026 and 2032 if historical scaling trends continued. That forecast remains relevant today, although synthetic data and repeated training on existing material have complicated the original timeline.
The speed of consumption explains the concern. Meta disclosed that Llama 3 was trained on more than 15 trillion tokens, roughly 7.5 times the 2 trillion used for Llama 2. Meanwhile, DataComp-LM researchers assembled a candidate pool of 240 trillion web tokens but still needed extensive filtering to produce effective training datasets.
Those figures reveal a problem with simply measuring how much information exists online. A model can read billions of repetitive articles without encountering much genuinely new knowledge. Scientific discoveries, expert reasoning and unusual real-world situations are far less abundant.
We are approaching a shortage of useful new information, particularly in domains where public internet content provides only a shallow picture of reality.
Why isn't more training data making AI models much smarter?
Adding more AI training data still improves models, but the gains increasingly depend on which examples developers select rather than how many tokens they collect.
DataComp-LM demonstrated the difference. Its 7-billion-parameter model achieved approximately 64% accuracy on MMLU using 2.6 trillion curated training tokens, outperforming MAP-Neo by 6.6 percentage points despite using approximately 40% less training compute. It also approached Llama 3 8B's MMLU performance, even though Meta reported training that model on 15 trillion tokens.
The models had different architectures and training procedures, so these comparisons cannot isolate data quality perfectly. However, DataComp-LM's controlled experiments independently showed that filtering and dataset composition could substantially improve performance.
A trillion tokens of repetitive online discussions may contribute less than a much smaller collection of carefully selected mathematical proofs, software failures or expert decisions. The difficult part is identifying those valuable examples before paying to train on them.
How much additional intelligence does each new training example actually buy? Larger datasets alone cannot answer that question.
If you want more recent data on this point, please see our latest AI training data market report.
A good product starts with a real problem. In our report covering the AI training data market, we show you the ones customers still live with.
Can synthetic data solve the AI training data shortage?
Synthetic data is already helping AI companies train better models, but replacing too much human-generated information with AI-generated examples remains risky.
A study presented at EMNLP 2025 examined more than 1,000 language models using over 100,000 GPU-hours. Researchers found that mixing approximately one-third rephrased synthetic text with two-thirds natural web text could accelerate progress toward the same validation loss by five to ten times under certain larger training budgets.
The researchers also found that favorable synthetic-data proportions often converged near 30%, although the ideal mixture varied with model size and training budget. Purely generated textbook-style material performed considerably worse on several downstream domains.
The distinction comes down to verification. For mathematics and programming, synthetic examples can often be checked through equations, executable code or automated tests. For clinical judgment, economic forecasting or human behavior, a convincing AI-generated answer may still be wrong.
This helps explain why synthetic data can be so useful for reasoning models. Developers can generate problems, test solutions and retain examples with reliable feedback. Generating new factual knowledge without an independent source of truth is considerably harder.
Synthetic data should keep expanding the amount of useful training material available. We have much stronger evidence for its value as a supplement to real data than as a complete replacement.
Will AI models eventually collapse from training on AI-generated content?
AI model collapse is a real risk, but recent research suggests that keeping authentic data in the training mixture can prevent much of the degradation observed in early experiments.
Research published in Nature in 2024 demonstrated that repeatedly training models on previous models' outputs could progressively erase rare information. Unusual examples and minority patterns disappeared before the most common material.
Later studies found that the outcome depends heavily on how synthetic data is introduced. A paper presented at ICML 2025, titled "Collapse or Thrive," reported that accumulating synthetic content alongside retained real data could avoid the runaway degradation seen when generated content replaced earlier datasets.
The large EMNLP 2025 experiment reached a compatible conclusion: certain mixtures improved training efficiency without producing the collapse predicted under other conditions.
AI-generated content already circulates through websites, social platforms and search results, so completely excluding it from future training datasets will be increasingly difficult. Developers need better ways to recognize it, track how often the same information is recycled and preserve independent sources.
The question still open is whether those safeguards will work reliably over many generations of frontier models.
If you build or invest in the AI training data market, you need to know which business models pay off now and which ones are fading. That's why our report covering it has a whole section on it.
Can AI companies prove where their training data came from?
AI companies still struggle to prove the complete origin and usage rights of their training data, especially when material passes through several datasets before reaching a model.
An ICLR 2025 study examined nearly 4,000 public datasets spanning text, speech and video. Researchers found that fewer than 33% of datasets carried restrictive licenses, while more than 80% of the underlying source content in widely used datasets had non-commercial restrictions.
Those percentages concern different layers of the data supply chain. Their contrast shows how a collection can appear broadly reusable even when material inside it has narrower permissions.
The OECD's 2025 research on AI training data reached a related conclusion: openly licensed datasets can still lack complete documentation of their original sources.
Consider a dataset assembled from Common Crawl, research repositories, online forums and other existing collections. Each source may have its own permissions, privacy obligations and restrictions. Once records have been filtered, deduplicated, transformed and combined, reconstructing those conditions becomes difficult.
The problem also affects datasets advertised as suitable for commercial use. Public availability does not automatically establish commercial rights, and a permissive dataset-level license cannot necessarily override restrictions on the original content.
We see provenance as one of the industry's most serious unresolved problems. Companies need a reliable way to trace individual training records through collection, processing and reuse. Today, that is still difficult to achieve at frontier-model scale.
If you want more recent data on this point, please see our latest AI training data market report.
Is copyrighted AI training data becoming too expensive?
Copyrighted training data is already creating major financial liabilities for AI developers, but licensing costs have not yet been shown to make frontier-model training economically unworkable.
Anthropic's $1.5 billion settlement with authors over books downloaded from pirated libraries provides the clearest example. The agreement received final judicial approval in July 2026 and covered more than 482,000 books, with compensation of approximately $3,000 per covered work.
Crucially, the underlying litigation distinguished between training on copyrighted works, which the court found could qualify as fair use in the circumstances considered, and acquiring pirated copies, which created separate liability. The settlement did not establish a universal obligation to license every copyrighted training document.
Meanwhile, commercial licensing has become an established market. OpenAI's agreement with News Corp was reported to exceed $250 million over five years, while Google's Reddit deal was reported at approximately $60 million annually. Shutterstock has also licensed image collections to AI developers.
These arrangements involve different rights and products, so their prices are not directly comparable. They nevertheless show that access to valuable collections increasingly carries an explicit cost.
The harder question is whether smaller AI companies can obtain comparable rights. Large developers can negotiate portfolio-wide deals, while startups may lack the money or bargaining power to access the same material.
| Training data source | Example | Main unresolved issue |
|---|---|---|
| Publisher archives | OpenAI-News Corp | Licensing cost and permitted uses |
| Social platforms | Google-Reddit | Rights to user-generated content |
| Stock-image collections | Shutterstock | Creator compensation |
| Public-domain books | Project Gutenberg | Limited contemporary coverage |
| Web-crawled collections | Common Crawl | Copyright and inherited restrictions |
| Research datasets | Hugging Face collections | Inconsistent licensing and provenance |
| Expert-created datasets | Specialist annotation providers | Cost and ownership |
| Private company records | Enterprise databases | Confidentiality and consent |
| Synthetic datasets | AI-generated examples | Reliability and originality |
Learning from others' mistakes is the cheapest lesson there is. In our market report about AI training data, we look at what broke startups before you, so you don't repeat it.
Can AI models really forget private information from training data?
Removing private information from an AI model remains extremely difficult to verify, even after the original records have been deleted from its training dataset.
Research presented at ACL 2025 examined how personally identifiable information behaves during model training. The researchers found that adding or removing particular information could unexpectedly change the memorization of other information. In their experiments, assisted memorization accounted for as much as one-third of relevant memorization effects.
That finding complicates attempts to delete one person's information without affecting unrelated knowledge.
An ICLR 2025 study separately examined ordinary, non-adversarial text reproduction and found that up to 15% of generated text overlapped with internet snippets in certain testing conditions. That does not imply that 15% of AI output exposes private information, but it demonstrates that reproduction can happen without someone deliberately trying to extract memorized content.
Researchers are developing differential privacy, machine unlearning, model editing and checkpoint-based retraining. These approaches can reduce specific risks, although they introduce costs and do not yet provide a general solution for proving complete removal from a large model.
For companies training on medical records, customer conversations or confidential legal documents, the unresolved issue is straightforward: can they demonstrate that sensitive information is gone without rebuilding the entire model?
How easy is it to poison an AI model's training data?
AI training data can be poisoned with surprisingly few malicious examples, and larger datasets do not automatically protect models against targeted attacks.
A 2025 study involving the UK AI Security Institute, Anthropic and the Alan Turing Institute examined models ranging from 600 million to 13 billion parameters. Researchers trained them on datasets containing between 6 billion and 260 billion tokens.
They found that approximately 250 malicious documents could introduce the tested backdoor across model sizes, even when larger models received more than twenty times as much clean training data.
In the largest experiment, the poisoned material represented approximately 0.00016% of training tokens. That is roughly one malicious token for every 625,000 training tokens.
The limitation is important: the experiment focused on a narrow attack causing models to produce meaningless text. It did not establish that 250 documents could reliably steal information, bypass safety protections or compromise any frontier model.
Still, the result challenges the idea that poisoning attacks need to control a meaningful percentage of the training corpus. Attackers may instead target particular behaviors with carefully constructed examples.
Detecting these attacks is difficult because malicious documents can be embedded among enormous quantities of legitimate material. Aggressive filtering also risks removing unusual but useful examples.
When you build or invest in the AI training data market, you need clarity, not confusion. In our document detailing everything you need to know about AI training data, we always try to put numbers on and quantify whatever is usually vague and confusing.
Are AI benchmarks becoming useless because models have already seen the answers?
Several popular AI benchmarks are currently too contaminated or poorly designed to measure frontier-model progress reliably, and the problem has spread beyond the most famous coding tests.
In February 2026, OpenAI announced that it would stop reporting SWE-bench Verified scores. Its analysis found evidence that every frontier model tested had encountered at least some benchmark problems or solutions during training.
OpenAI also audited 138 of the benchmark's 500 tasks, focusing on problems models struggled to solve consistently. It found material flaws in 59.4% of those audited tasks. Because the sample was not random, that percentage cannot be applied to the entire benchmark.
The concern was serious enough that Epoch AI classified SWE-bench Verified as flawed in September 2026.
More revealingly, switching benchmarks did not completely solve the problem. In July 2026, OpenAI published another audit estimating that approximately 30% of SWE-bench Pro tasks had broken evaluation criteria. Human reviewers identified problems in 249 of the 731 tasks examined.
That second finding broadens the issue. Even when training-data contamination is reduced, automated tests can reject correct solutions, reward incomplete ones or demand behavior the original question never specified.
Private evaluations, continuously refreshed questions and expert reviews offer better protection. Yet developing and maintaining them is expensive, and private benchmarks are harder for independent researchers to examine.
A benchmark score that rises from 75% to 80% looks impressive. Without knowing how much came from genuine reasoning, memorized examples or defective tests, we cannot reliably interpret the improvement.
If you want more recent data on this point, please see our latest AI training data market report.
Why do AI models still fail on rare but important situations?
AI models remain unreliable on many rare situations because their training datasets contain too few verified examples of events that hardly ever happen.
For a model trained to minimize average error, correctly handling millions of ordinary examples can outweigh failure on a small number of exceptional cases. That becomes dangerous when those exceptional cases involve medical emergencies, industrial accidents or security incidents.
Autonomous driving illustrates the difficulty. The Waymo Open Dataset and nuScenes provide large collections of real-world driving observations, but collecting enough examples of unusual weather, unexpected pedestrian behavior and rare equipment failures remains expensive.
Healthcare faces a comparable shortage for uncommon diseases and unusual patient presentations. Cybersecurity has another version of the problem: genuinely new attacks are absent from historical training records until someone discovers them.
Researchers can deliberately generate rare cases through simulation or synthetic data, but those examples must reproduce the relevant physical, biological or technical conditions. An unrealistic simulation may teach the wrong response.
Rare events are, by definition, difficult to collect in statistically meaningful quantities. Better average performance offers little reassurance when the mistakes that matter most sit outside the common training distribution.
In our report covering the AI training data market, we look at what the winners do differently, so you can learn from the best instead of guessing.
Why is AI training data still so heavily focused on English?
English remains heavily overrepresented in AI training data, and the growth of multilingual datasets has not eliminated the resulting performance differences.
The World Bank's Digital Progress and Trends Report 2025 found that English accounted for approximately 45% of global web URLs in the Common Crawl data it examined. English also represented 56.3% of the open-source Hugging Face datasets included in its language comparison.
Chinese represented around 5% in both measures, while Hindi, despite its enormous speaker population, did not appear among the individually listed leading web-content languages.
The imbalance has persisted for years. An ICLR 2025 audit covering nearly 4,000 datasets found that relative multilingual and geographic coverage had not improved significantly since 2013.
And the issue extends beyond major languages. Stanford's 2026 AI Index highlighted evaluations in which leading models lost close to half their accuracy when tested on a regional Slovenian dialect rather than standard Slovenian.
Translation alone cannot resolve these gaps. Models need local technical terminology, dialects, regulatory knowledge, cultural context and examples of how people actually solve problems in different regions.
There is a clear imbalance between where AI training information is concentrated and where people are expected to use the resulting systems. The shortage is especially severe for languages and communities with limited digitized material.
Why can't human data labeling fix unreliable AI answers?
Human labeling improves AI training data, but even highly trained annotators cannot consistently provide one correct answer to every complex question.
Some tasks are easy to verify. A mathematical result can be checked, and a program can be run against tests. Other tasks depend on subjective preferences, incomplete information or changing professional standards.
Consider an AI model learning from medical decisions. Two qualified clinicians may recommend different treatments because they weigh risks differently. A customer-service dataset can contain answers that were correct under an older company policy but are wrong today.
The same issue affects preference training. If annotators repeatedly favor confident, well-written answers, a model may learn to sound persuasive even when its reasoning is unreliable.
More annotations can improve consistency, but agreement should not be confused with correctness. In many high-stakes tasks, the training record needs to preserve uncertainty, competing interpretations and the evidence behind a decision.
The industry still lacks a universally reliable way to turn complex expert judgment into scalable training feedback. That becomes increasingly important as AI systems move from answering factual questions toward making consequential recommendations.
In our research about AI training data, we always try to gather several sources and make estimates that make sense, so you can work with solid numbers.
Can AI learn expert judgment without access to private company data?
Public training data gives AI models only a partial view of how professionals actually make decisions, making private company records potentially valuable but difficult to use.
Open-source software repositories contain enormous amounts of code, yet they usually reveal little about production incidents, abandoned engineering approaches or internal trade-offs. Medical research explains clinical findings without exposing the full range of decisions made during hospital care. Financial filings describe outcomes while omitting most internal deliberations.
The OECD's 2025 analysis of AI training data identified this wider access problem: no centralized market supplies every type of information needed for model development, and different sources come with different restrictions.
Private company records could fill important gaps. Internal support tickets contain recurring customer problems, engineering logs show actual system failures, and operational databases record decisions and outcomes unavailable on the public internet.
However, these records can include trade secrets, personal information and historical mistakes. Companies also need ways to separate genuinely useful expertise from outdated procedures or inconsistent human behavior.
The opportunity is substantial for specialized AI systems. A model trained on carefully verified internal workflows could learn capabilities that general internet datasets barely cover. Access rights, confidentiality and outcome validation currently limit how widely that approach can be used.
If you want more recent data on this point, please see our latest AI training data market report.
Is robotics facing a bigger training data problem than language AI?
Robotics faces an even more expensive data collection problem than language AI because useful robot demonstrations require physical actions, specialized equipment and reliable measurements.
Stanford's DROID dataset shows the effort involved. Researchers collected approximately 76,000 robot manipulation trajectories representing 350 hours of interaction across 564 scenes and 84 tasks. The project involved 50 contributors across North America, Asia and Europe over roughly twelve months.
Larger datasets have since emerged. Open X-Embodiment assembled roughly one million robot trajectories, while AgiBot World reported approximately one million episodes. These collections represent major progress, but their counts do not imply equivalent diversity or usefulness across different robots.
A video of someone opening a drawer, for example, does not provide the same training information as a recorded robot demonstration containing joint movements, forces and control signals.
Synthetic video and simulation can help. Researchers can generate thousands of variations in lighting, camera positions, obstacles or object placement without physically recreating every scene.
However, visual realism can hide inaccurate physics. A generated warehouse video may look convincing while misrepresenting friction, object weight or collisions. Training on those mistakes can produce behavior that fails when deployed.
Robotics therefore has two connected problems: collecting enough real physical experience and ensuring that simulated experience transfers correctly to real machines.
In our market report about AI training data, we turn what is vague and complex into simple, useful visuals. Market maps like this one show you at a glance where every startup stands.
Which AI training data problems are actually getting closer to being solved?
Some AI training data problems are improving quickly, especially filtering and synthetic-data generation, while privacy, provenance and verified real-world coverage remain much harder.
DataComp-LM demonstrated measurable gains from selecting better training material. Large synthetic-data experiments have identified mixtures that improve training efficiency. Robotics datasets now contain hundreds of thousands or millions of recorded episodes, far beyond the scale of many early collections.
These advances are real, although each solves only part of the underlying problem.
The legal picture is less settled. Large licensing agreements have created ways to acquire some copyrighted material, but there is still no universal standard for the rights needed across every jurisdiction and source.
Privacy is similarly difficult. Machine-unlearning techniques can reduce memorized information, but developers still struggle to prove that particular records have been completely removed.
Benchmarking has shown how stubborn the quality problem can be. OpenAI's successive audits of SWE-bench Verified and SWE-bench Pro demonstrated that replacing a flawed evaluation does not automatically produce a reliable one.
Engineering improvements are advancing faster than the systems needed to establish trust, ownership and independent verification.
| Unsolved problem | Evidence of progress | Remaining obstacle |
|---|---|---|
| High-quality text scarcity | Better filtering and reuse | Genuinely new information |
| Synthetic-data reliability | Successful mixed-data training | Independent factual grounding |
| Model collapse | Retaining real data reduces degradation | Long-term recursive training |
| Data provenance | Better audits and documentation | Complete source traceability |
| Copyright | Large commercial licensing deals | Consistent permissions and pricing |
| Privacy | Unlearning and privacy techniques | Verifiable removal |
| Benchmark contamination | Private and updated tests | Reliable independent evaluation |
| Data poisoning | Large-scale attack research | Detection and prevention |
| Language imbalance | More multilingual datasets | Deep regional coverage |
| Human labeling | Expert feedback and verification tools | Disagreement and uncertainty |
| Rare-event training | Simulation and targeted collection | Reliable real-world validation |
| Robotics | Million-episode datasets | Physical diversity and transferability |
What is the biggest unsolved problem in AI training data today?
The biggest unsolved problem in AI training data is securing a continuing supply of genuinely useful, trustworthy information that developers can legally access and independently verify.
The evidence points consistently in that direction.
Epoch AI's estimate of approximately 300 trillion usable public-text tokens shows why the supply of high-quality human-generated information could become constrained. DataComp-LM demonstrated that carefully chosen examples can outperform far larger training collections. Synthetic-data experiments showed that generated material can help enormously, although its reliability depends on how it is created and checked.
Meanwhile, the $1.5 billion Anthropic copyright settlement exposed the financial risks of unclear data acquisition. The SWE-bench audits showed that even the datasets used to measure AI progress can contain serious defects.
Taken together, these findings put data quality, originality, access rights and verification at the center of the problem.
We rank the shortage of independently verified, high-value information above synthetic model collapse or raw dataset size because it affects almost every major application. Medical AI needs reliable examples of uncommon conditions. Coding agents need real software failures and trustworthy tests. Robotics systems need physical experience that accurately represents the world.
Much of that information is expensive to collect, privately held, legally restricted or difficult to validate.
The strongest opportunity now lies in building systems that can repeatedly obtain such information, document its origin and prove that training on it actually improves performance.
That is likely to become a more durable advantage than simply owning the largest collection of scraped internet content.
If you want more recent data on this point, please see our latest AI training data market report.
Our document detailing everything you need to know about AI training data gets updated several times a month, so you always get the latest view of where things stand.
Our methodology
This analysis examines the biggest unresolved problems in AI training data as of October 2026. We divided the question into distinct areas covering data availability, quality, synthetic-data reliability, provenance, copyright, privacy, security, evaluation, linguistic coverage, human annotation and physical-world training. For each area, we reviewed recent technical experiments, academic research, institutional findings and documented industry developments.
We prioritized original evidence and measurable results over industry predictions or general commentary. We compared findings across independent sources, examined the limitations of individual experiments and distinguished demonstrated improvements from problems that remain difficult to solve at scale. A controlled synthetic-data experiment, for example, tells us something different from a commercial licensing agreement or a reported benchmark audit.
Our final assessment considers both the severity of each problem and how widely it affects AI development. We gave particular weight to obstacles that persist across different model types and applications, including the scarcity of independently verified information, incomplete data provenance, unclear usage rights and the difficulty of obtaining reliable examples of rare or specialized situations. The resulting conclusion reflects the combined evidence rather than a single research finding or industry forecast.
Key research sources include Epoch AI's study of public-text scarcity, Meta's Llama 3 training disclosure, the DataComp-LM research repository, Kang et al.'s large-scale synthetic-data experiments, Nature's original model-collapse research, and the ICML paper "Collapse or Thrive."
For data rights, provenance and privacy, we used the ICLR study "Bridging the Data Provenance Gap Across Text, Speech, and Video", the OECD's research on AI data access and sharing, the Authors Guild's announcement of the Anthropic settlement approval, OpenAI's official News Corp partnership announcement, and ACL research on assisted memorization.
Our assessment of security, evaluation and real-world data coverage draws on Anthropic's training-data poisoning experiments, OpenAI's SWE-bench Verified audit, OpenAI's SWE-bench Pro evaluation audit, the World Bank's Digital Progress and Trends Report 2025, Stanford HAI's 2026 Responsible AI findings, the Waymo Open Dataset, and the original DROID robotics dataset research.
In our report covering the AI training data market, we don't want to repeat what boring consulting reports say. Instead, we show you the real and latest market signals that matter.
Written by the New Market Pitch teamWe track new markets so founders and investors can move faster. About us
