You would never take a stranger’s word for something that matters, not without checking who they are, what they are selling, and whether they have a reason to shade the truth. Yet you will open an AI assistant, ask it something that actually matters, and take the answer at face value.
The machine is mostly quoting strangers, real professionals in the best cases and some throwaway handle like randomuser1234 in plenty of others, and the citations sitting beside the answer lend all of them the same aura of credibility. Between that confident tone and the faint sycophancy the model brings to everything, you lose the inclination to interrogate any of it, which means the answer layer we have started to trust is quietly weighted toward the sources with the least accountability wrapped around them.
Authenticity as a budget line
Two currents are meeting to make that true, and the interesting part is that they push in the same direction for entirely different reasons.
The first is cultural. We stopped trusting institutions and went looking for real people, rewarding what feels authentic. And the money follows the feeling: creators now take a median 26% of brand social budgets, up from 18% two years ago, according to a Gigapay report, and close to half of US creator spend now flows to micro and nano accounts, according to eMarketer. Brands will tell you the reason is that smaller voices read as more trustworthy, but that also means we are deliberately giving a megaphone to the people with the least accountability apparatus around them, because the missing apparatus is exactly what makes them feel real.
The second current is technical. When Meltwater analyzed 9.5 million AI citations, it found that on LinkedIn, 75% pointed to individual profiles rather than company pages, and 51% came from accounts with fewer than 10,000 followers. The machine and the market arrived at the same conclusion: small, authentic, individual voices are more credible. But they also happen to be the least accountable, so the move to feel we are getting the truth, going to real people instead of official channels, turns out to be the same move that routes the machine’s answers through the layer with the least friction against being wrong.
Accountability as friction
Because friction is the thing corporate content has and individual content usually does not. Behind a company claim sits legal review, disclosure law, and a regulator who can open an investigation or levy a fine. None of that makes a company honest, but it does make lying expensive enough to become a calculated decision rather than a reflex.
Smaller creators don’t have the same incentives, and upholding standards becomes more of an honor code than real self-governance: a Marketing Science study of more than 100 million Twitter posts found that 96% of sponsored content carried no disclosure at all.
Polymarket is the cleanest case I have seen this year. A Wall Street Journal investigation found the company paid creators to stage fake winning bets on near-perfect copies of its own site and told them to hide that they were paid, and across 118 of those videos creators celebrated roughly $900,000 in winnings on positions that would have lost more than $166,000 in reality, all of it pulling past 140 million views. What matters is where the accountability landed, because the regulators are circling Polymarket, the company with the balance sheet, while the creators who filmed the fabrications made their few thousand a month and mostly kept it. The legal weight falls on the entity you can sue, and the individual who served as the actual delivery mechanism mostly walks. The answer layer inherits the same asymmetry.
The scarce intersections problem
The obvious pushback is that the model cross-checks itself, so one false claim gets washed out by every other source, and sometimes that is exactly what happens, just not for the queries that matter most here. LLMs reward fresh, specific, niche content, and Meltwater’s own term for what gets cited is scarce intersections, the narrow topics with almost no existing coverage, and for precisely those questions there is no corroborating body of work to check against, so a single confident voice becomes the answer by default. I could declare myself the best scuba diving shark lawyer alive, and if I posted about it consistently in a niche thin enough, I would get picked up for the simple reason that I was the only one talking.
Thin niches are the easy case, but the problem is real for more substantial topics, because with enough money and enough coordinated people you can build volume around any claim until it starts to look like a genuine conversation. We still have a meaningful number of people who believe the earth is flat and that the moon landing was staged, on questions where the evidence could not be more settled, which is all the proof you need of what sustained volume does to belief even when the truth was never actually in doubt. Point that same machinery at something genuinely unsettled, a new product category, an investment strategy, a health claim, and you are poisoning the well that the model drinks from.
The gatekeepers we tried to escape
So the real question is what we do about it, and the honest answer is uncomfortable, because until the machine can validate the credibility of a source in a hard, checkable way rather than inferring it from patterns of structure and frequency, the job of deciding who to believe stays with us. Hard validation is possible in small doses, whether by restricting a model to a vetted pool of sources or by leaning on provenance standards that cryptographically sign where content came from, but none of that scales to a tool whose entire promise is answering any question from everything ever written. The moment you curate the pool tightly enough to trust it, you have shrunk it, slowed it, and quietly reappointed the gatekeepers that the whole flight to authenticity was trying to escape, which is the trap here: trust and total scale pull against each other, and the model was built for scale. What that leaves us with is a small and unglamorous habit of verifying what an LLM or a content creator shared, then checking the ledger of where it came from.
We democratized the ability to publish, which was mostly a good thing, and the same act made content abundant and credibility scarce, so that when anyone can say anything and the machine will repeat the most confident version of it back to us, trust stops being the default and becomes the expensive part.
Information is free now, knowing who to believe is the last thing worth paying for.