As people increasingly surrender their thinking to artificial intelligence (AI), building safeguards against AI hallucinations is becoming critical to prevent fabricated information from being accepted as fact.
This is according to Willy Seyama, founder of AI start-up Orkestra, delivering a presentation − titled “Content integrity in the age of hallucinations” − yesterday at the Social Media Summit organised by public relations firm Decode.
Seyama warned that society has entered what he describes as "the era of hallucinations", where AI-generated falsehoods are no longer an occasional flaw but an unavoidable feature of generative AI.
Rather than asking whether AI hallucinations can be eliminated, organisations should be asking how they can reduce their impact, he said.
"Fabricated citations and false claims are not a bug that a future update is able to remove − they are a consequence of how large language models work as probabilistic next-word predictors.
“A confident, well-formatted, false citation is the technology working as designed. That's why a verification layer between 'the AI wrote it' and 'we published it' remains necessary regardless of how models improve.”
Seyama argued that cognitive surrender − the growing phenomenon of adopting AI-generated answers or decisions in place of the human’s own reasoning − is creating a dangerous cycle in which users become less inclined to verify information independently, increasing the risk that inaccurate content finds its way into public reports, research, legal documents and policy.
He described AI hallucinations as fabricated information presented confidently as fact.
The consequences, he said, extend well beyond individual mistakes, particularly as governments, corporations and professionals increasingly rely on AI to draft high-value documents where factual accuracy is paramount.
Dire consequences
Seyama warned that the erosion of human judgement poses one of the greatest risks associated with generative AI. As users place greater trust in AI-generated responses, fewer people take responsibility for independently verifying the information before it is published or used to make decisions.
"When we are government and when we are private companies and we sit with information that we cannot trust, then we have a problem."
He said hallucinations can influence legislation, legal proceedings, corporate submissions and public communications, creating reputational, financial and legal risks for organisations.
“The problem is compounded by generative AI's ability to produce convincing, authoritative content that appears credible even when key facts, citations or references have been fabricated.
“AI should enhance human judgement rather than replace it. Users who become overly dependent on the technology risk surrendering the very critical thinking needed to identify errors.”
Emerging worldwide
Seyama said his research has identified 59 high-profile AI hallucination cases globally, although he believes the true number is significantly higher because many organisations choose not to disclose such incidents publicly.
"What we have found is that the bulk of these cases are actually coming from the legal fraternity. Fabricated case law, fabricated precedents, and so it goes."
He said that of the 29 major hallucination cases identified during 2025, 14 occurred within the legal sector.
Closer to home, Seyama pointed to SA’s draft National AI Policy Framework, which was withdrawn after the discovery of AI-generated references that could not be verified.
Seyama also highlighted incidents involving major international consulting firms whose AI-assisted submissions contained fabricated information, demonstrating that even well-resourced organisations are vulnerable when AI-generated content is not independently verified.
"This is not happening to the little guys. It's been done by those who are supposed to have the resources and shouldn't be allowing it to happen."
Slow down
Accepting that AI hallucinations cannot be eliminated does not mean organisations should simply tolerate them, Seyama said.
Instead, he urged businesses and public sector organisations to introduce structured verification processes that prioritise factual accuracy over speed.
"I'm not about speed. I'm about precision because speed is what also compounds the prospect of hallucinations."
He recommended that users establish customised "guardrails" that require AI-generated content to undergo multiple checks for factual accuracy, precision and contextual believability before publication.
Seyama also advised users to avoid treating AI as an unquestionable authority. He encouraged users to validate important facts, citations and claims against multiple credible sources, particularly when producing documents for public consumption or making business and policy decisions.
“Users should also make AI part of the verification process by instructing the AI tool to review and challenge its own output instead of simply accepting the first response. Rather than asking AI to generate content as quickly as possible, users should prompt it to fact-check, test for precision, assess the believability of claims and identify anything that appears inconsistent or unsupported.”
He further recommended consistently using one large language model instead of constantly switching between platforms, because a single model develops a better understanding of the user's work and can apply more relevant contextual guidance over time.
"There is no quick fix until I am actually happy with the verified results. When I'm happy, that's when I say good stuff, I have exhausted all verification options and now I'm satisfied that this document meets my own standards before I even send it to the client."
To address these challenges, Seyama has developed Ukweli, an AI content verification platform to identify fabricated facts, citations and inconsistencies before content is published or used.
The platform is currently in beta testing with a small, select group of users as additional verification capabilities are refined.
Share
