The core argument is that hallucinations are no longer best understood as isolated technical failures. They are an economic tradeoff.
The leading AI companies know their models sometimes fabricate facts, citations, legal cases, scientific papers, and quotations. Yet they continue releasing increasingly capable systems because reducing hallucinations to near zero would likely make those systems slower, more expensive, more constrained, and in some cases less useful. The business incentives reward fluency, speed, and user engagement more than perfect accuracy.
Here’s how that business model works.
1. Speed beats certainty
When you ask an AI a question, users expect an immediate answer. The model is optimized to respond, not to say, “I don’t know.”
Admitting uncertainty is often the technically correct answer, but it feels like failure. A confident answer, even if partly invented, keeps users engaged.
The platform’s incentive is to maximize successful interactions, not necessarily factual precision.
2. Bigger models create more confidence, not necessarily more truth
Modern language models are trained to predict the next word. They are extraordinarily good at producing language that sounds authoritative.
Synthetic quotations are a natural consequence of that architecture. If a model has learned the style of someone like Kara Swisher or Jonathan Haidt, it can generate something that feels exactly like something they would say, even when they never said it.
From the user’s perspective, it doesn’t feel like a guess. It feels like evidence.
3. Every safeguard has a cost
Companies could dramatically reduce fabricated citations by requiring retrieval from verified sources before answering many questions.
But that creates tradeoffs:
- Responses become slower.
- Infrastructure costs rise.
- More questions end with “I can’t verify that.”
- The product feels less magical.
For consumer AI, those costs directly affect growth and revenue.
4. The cost of hallucinations is externalized
When an AI invents a quote, who pays?
Usually not the platform.
Instead, the costs fall on:
- authors
- journalists
- lawyers
- students
- researchers
- physicians
- publishers
The platform gains the benefit of a fast, compelling answer, while someone else absorbs the reputational, legal, or financial damage.
Economists call this an externality. The company captures the upside while others bear much of the downside.
5. Current incentives reward adoption over reliability
Most frontier AI companies compete on:
- daily active users
- API usage
- enterprise adoption
- subscription growth
- benchmark performance
None of those metrics directly penalize synthetic quotations unless they trigger major public backlash.
If reducing hallucinations slowed adoption by even a few percentage points, a rational business might accept a certain error rate.
That makes hallucinations not merely an engineering problem but an economic optimization.
Why this matters
Calling hallucinations “bugs” suggests they’re accidental defects that engineers simply haven’t fixed yet.
Calling them part of the business model makes a different claim:
The companies understand the risk, know how to reduce much of it, but choose a balance that maximizes product growth and market adoption rather than minimizing factual error.
That doesn’t mean engineers want hallucinations. It means the organization accepts a certain level of them because eliminating them completely would reduce competitiveness.
The policy question
This is why some legal scholars argue that hallucinations deserve scrutiny similar to other product design choices.
If a company knowingly releases a product that predictably fabricates authoritative-sounding information because that design improves engagement or market share, regulators may ask whether those fabrications are simply unavoidable technical limitations or foreseeable harms that should carry greater responsibility.
That distinction could become increasingly important as AI systems are relied on in publishing, medicine, law, education, and journalism, where a fabricated quotation isn’t just an incorrect answer. It can damage reputations, alter public understanding, and undermine trust in the factual record.
Citation/Disclosure: * Written after a series of prompts and interactions with ChatGPT
