A new study has showed that the leading AI models are terrible at summarizing or relaying news stories, casting doubt on their role in journalistic applications.
The study was conducted by the BBC and looked OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini, and Perplexity AI. The study involved giving the AI models content from the outlet’s website and then asking questions about the news stories.
Deborah Turness, CEO of BBC News and Current Affairs, detailed the study in a blog post.
Our researchers tested market-leading consumer AI tools – ChatGPT, Perplexity, Microsoft Copilot and Google Gemini – by giving them access to the BBC News website, and asked them to answer one hundred basic questions about the news, prompting them to use BBC News articles as sources.
Unfortunately, the results were less than encouraging.
The results? The team found ‘significant issues’ with just over half of the answers generated by the assistants.
The AI assistants introduced clear factual errors into around a fifth of answers they said had come from BBC material.
And where AI assistants included ‘quotations’ from BBC articles, more than one in ten had either been altered, or didn’t exist in the article.
Part of the problem appears to be that AI assistants do not discern between facts and opinion in news coverage; do not make a distinction between current and archive material; and tend to inject opinions into their answers.
The results they deliver can be a confused cocktail of all of these – a world away from the verified facts and clarity that we know consumers crave and deserve.
Why the Study Is Concerning
The BBC’s study is particularly disturbing in the context of the tech and news industries’ wholesale adoption of AI. Countless outlets have turned to AI “reporters” and “writers” in an effort to speed up production or cut costs.
Unfortunately, much of modern news writing involves writing about stories other outlets have already covered, while trying to put a different and unique spin on the news. As the study demonstrates, however, this is exactly the kind of task that AI is still ill-suited for, failing to distinguish between fact and opinion, hallucinating details, and making up quotes.
Companies looking to replace human workers with AI agents, especially in fields where accuracy matters, would do well to heed the BBC’s findings.
For more information, the study can be found here in its entirety.
from WebProNews https://ift.tt/iNmlhX2
No comments:
Post a Comment