← The Vault
Explainer

Why our brains see “intelligence” where there is only math

We often mistake AI programs for thinking, human-like entities. Linguist Emily Bender explains why tools like ChatGPT are actually just advanced pattern-matching machines that create text without understanding a word of it. Understanding this distinction is key to navigating the risks of automation, from biased results to the hidden labor used to build these systems.

Edition № 142Room: Explainer30 June 20262 min readSources: 4
Article

When you use a tool like ChatGPT, it is incredibly easy to feel like you are conversing with a digital person. It responds to your prompts, apologizes when it makes a mistake, and seems to grasp the nuance of your questions. But according to experts like computational linguist Emily Bender, this feeling is a trick of our own psychology, not a feature of the software itself.

WHAT'S HAPPENING

The term "stochastic parrot," coined in a 2021 research paper, is a description of how large language models—the technology powering chatbots—actually function. The word "stochastic" simply means involving random chance or statistical probability, and "parrot" refers to the system’s ability to repeat back patterns it has seen before. These systems do not "know" anything. Instead, they have been fed massive amounts of text and trained to calculate the likelihood of which word should follow another in a given sequence. When one of these models produces a persuasive sentence, it isn't because the software understood you; it's because you are highly skilled at finding meaning in the patterns the machine assembled.

The Mirror Effect

HOW IT WORKS

Think of a large language model as a massive, high-speed game of "autocomplete" that has read almost everything on the internet. During its training, the model digests billions of sentences and turns them into a complex mathematical map of how words tend to appear next to one another. When you give it a prompt, it doesn't consult a database of facts or a library of "thoughts." It simply consults its map to predict the most statistically probable string of words to put next.

The reason these systems often seem "sycophantic"—meaning they immediately agree with you or apologize if you correct them—is due to an extra layer of training applied after the initial learning phase. Designers explicitly teach the model to favor polite, agreeable responses to ensure it feels helpful to human users. This added layer is essentially a "politeness filter" pasted on top of a statistical engine, which further obscures the fact that the machine has no opinion, no world view, and no grasp of the truth.

WHY IT MATTERS

Treating these tools as "intelligent" rather than "statistical" has real-world consequences. If we believe a system is smart, we are more likely to trust its output without verification, even though it can be easily manipulated or hallucinate facts. Furthermore, the label "AI" acts like an umbrella that groups together fundamentally different things—a protein-folding tool that solves medical problems is not the same as a chatbot that predicts the next word in a sentence. By lumping them together, we lose the ability to regulate them effectively or understand what they are actually doing. The next time you find yourself impressed by a bot’s "logic," remember: it isn't reasoning, it is just guessing what you want to hear based on the math of its training data.

Sources
← PreviousWhy Big Tech is Racing to Make AI Faster and Easier to UseNext →Anthropic's New Research Tool and Evolving AI Export Rules
Tomorrow's edition · free

Liked this one? The next lands at breakfast.

Every story in tomorrow's AI news, rebuilt in plain English — five minutes, sources linked, free forever.

By joining you agree to receive Article's daily newsletter — unsubscribe in one click. Privacy

← Back to the Vault