By Dr. Tamara Patzer
Key Takeaways
- A July 2024 study by the MIT-led Data Provenance Initiative found that nearly 45 percent of the news-site text in the most heavily used web domains was fully off limits to AI crawlers by April 2024. A year earlier the figure was 3 percent.
- News and periodicals made up half of the 2,000 most-used domains in the study, but only about 5 percent of a random sample of the web.
- Fewer than 1 percent of sampled ChatGPT questions in the study were about news or current affairs, even though news sites supplied nearly 40 percent of the text in one major training set.
- In June 2026, New York lawmakers passed a bill that would require makers of generative AI to post a summary of what their models were trained on. As of November 8, 2026, the state Senate’s bill page showed no signing.
The Sign on the Back Door
Every website has a small text file called robots.txt. It sits quietly at the back of the site and tells automated visitors, called crawlers, where they are welcome and where they are not. For years it mostly guided search engines. Then AI developers began sending crawlers of their own, and publishers started rewriting the sign.
Researchers at the Data Provenance Initiative, a volunteer group led from the MIT Media Lab, set out to count how many had done it. Their study, “Consent in Crisis,” was published in July 2024. The team, led by Shayne Longpre, looked at 14,000 web domains behind three widely used training sets: C4, RefinedWeb and Dolma. They compared monthly web archive snapshots from 2016 through April 2024, with a close look at the change between 2023 and 2024.
The sharpest change was in news. Among the most heavily used domains in the three sets, nearly 45 percent of news-site text was fully restricted by robots.txt in April 2024. In 2023 the figure was 3 percent. One year, 3 percent to nearly 45.
Why News Matters Most
News sites matter to AI developers for a simple reason: they are everywhere in the data. Table 4 of the study shows news and periodicals made up 50 percent of the 2,000 domains that supply the most text, against 5.3 percent of a random sample of the web. In one training set, C4, news sites supplied nearly 40 percent of the text.
That is a strange match for how people actually use chatbots. The researchers compared the training data against WildChat, a collection of real conversations with ChatGPT gathered between April 2023 and May 2024. Fewer than 1 percent of the sampled questions were about news or current affairs.
So the sources AI leans on most are the ones its users ask about least, and they are also the ones closing fastest. The authors point to a likely result: models drift toward older, less fresh text, because the newest reporting sits behind the locked door. They also note that some journalists may want to block crawlers to protect how fresh their work stays.
What the Closing Doors Look Like Up Close
Behind those percentages are ordinary decisions made in newsrooms and on small publisher sites. Someone reads a crawler’s name in a server log, then adds a line to a file. Someone updates the terms of service. The Data Provenance team found that those two signals often do not match, so a site may say no in its terms and say nothing in robots.txt. The researchers called the tools blunt, built for a web that did not have AI in mind.
For readers, the effect is quiet. Nothing on the page changes. What changes is what a machine can learn from it. The Mozilla Foundation, which backed the project, quoted the team as saying the freshest, most actively maintained sources, including news, are being restricted at the highest rates.
The study’s authors also warned that the closing affects more than AI companies. Academics and noncommercial researchers who depend on open web data are caught by the same files, because robots.txt cannot easily tell a company from a university lab.
Albany Takes Up the Question
New York has now turned to a related question: what, exactly, went into the models?
The Artificial Intelligence Training Data Transparency Act is sponsored in the Senate by Andrew Gounardes as S6955A. Assembly Member Alex Bores carries the Assembly version, A6578, according to Route Fifty’s June 12, 2026, report on the state’s legislative year. The Assembly passed its bill on May 5, 2026. On June 4, 2026, the Senate substituted the Assembly bill for the Senate version and passed it 54 to 6, clearing both chambers before the session ended.
As written, the bill would require developers of generative AI that is available to New Yorkers to post documentation about their training data on their websites. The summary would cover the sources or owners of the data, the number and types of data points, whether the data includes copyrighted material or personal information, whether it was purchased or licensed, and any cleaning or processing, including whether synthetic data was used. The first disclosures would be due by January 1, 2027.
The Senate’s bill page showed no further action after the June vote. It had not been delivered to the governor, signed or vetoed. Readers who want the current status can check the page directly.
What It Means for Readers and Publishers
None of this settles how AI should treat the news. The research shows a trend and a gap. News is the raw material AI draws on most, news sites are locking it away at the fastest rate, and the people who use the tools ask about news least.
For local publishers, the practical points are modest. Check what your own robots.txt file actually says. Make sure it matches your terms of service. And keep your byline and source attribution clear on everything you publish, because a named reporter is the one thing a crawler cannot manufacture.
For readers, the takeaway is to know where your information comes from. An answer from an AI tool may reflect older reporting than the story published this morning, and the training summaries New York is weighing would be one more way to see what sits behind an answer.
Frequently Asked Questions
What is robots.txt? It is a small text file on a website that tells automated crawlers which pages they may visit. It is a request, not a lock, and not every crawler follows it.
How much of the news was blocked? In the Data Provenance Initiative’s study, nearly 45 percent of news-site text in the most heavily used domains was fully restricted by robots.txt in April 2024, up from 3 percent in 2023.
Is New York’s training data bill law? Not as of November 8, 2026. The Senate’s bill page showed it passed in June 2026 with no signing or veto recorded.
What would the bill require? Generative AI developers would post a high-level summary of their training data, including its sources, whether it includes copyrighted material or personal information, and whether synthetic data was used.
About the Author
Dr. Tamara “Tami” Patzer is a publisher and former newspaper editor who was on the staff of The Charlotte Sun when it was named a 2005 Pulitzer Prize finalist in Breaking News Reporting. She writes about AI and media literacy. Connect with her on LinkedIn. Contact: info@dailysuccessinstitute.com.
