When you apply for a job today, your resume might be filtered by an algorithm before a person ever lays eyes on it. Most people assume the main danger is that the AI will inherit old, existing prejudices found in its training data. However, new research from Princeton and the University of Chicago reveals a more unsettling reality: AI can actually teach itself to be prejudiced through the simple act of doing its job.
Researchers created a simulation where AI models acted as consultants tasked with hiring people for various roles like doctors, lawyers, and janitors. The candidates were divided into four fictional groups, and in reality, every candidate had an equal chance of success. As the models made hiring decisions, they received feedback on who succeeded in their role. Within just 40 rounds, the AI began to segregate applicants by their group labels based on tiny, random streaks of success or failure. If a model saw an individual from one group fail at a medical job, it didn't just avoid that person; it stopped hiring that entire demographic for medical roles, pigeonholing them into lower-responsibility jobs instead. When compared to actual humans performing the same task, the AI models were significantly more prone to these rigid stereotypes.
Why smart systems make dumb assumptions
To understand why this happens, you have to look at the goal of these systems. A large language model—the technology powering tools like ChatGPT—is essentially a pattern-recognition engine. It is built to look at a limited amount of information and reach a conclusion as quickly as possible. In fields like math or coding, this is highly effective because those fields follow consistent rules. But when this efficiency is applied to human behavior, it backfires.
Psychologists have a term for the challenge of balancing what we already know with the need to try new things: the exploration-exploitation dilemma. Imagine you are deciding where to have lunch. Do you go to your favorite reliable spot, or do you try a new restaurant that might be better? AI models are optimized to exploit patterns they find early on. Because their core logic is designed to solve problems by finding the shortest path to an answer, they settle on a generalization after seeing just a few examples. They interpret a handful of coincidences as a universal law, and once that assumption is set, they stop looking for better alternatives.
This research suggests that as companies give AI more memory and autonomy, they risk creating systems that form biases no human ever taught them. These systems are not just repeating old prejudices; they are actively inventing new, unfair categories based on their own experiences. The study also offers a potential path forward: when researchers specifically incentivized the models to value diversity, the bias dropped significantly. It turns out that AI does exactly what we reward it for, which means for these systems to be fair, we must be much more precise about how we define the finish line. If we train our tools to prioritize only raw efficiency, we should not be surprised when they decide that discrimination is the shortest path to the goal.
Liked this one? The next lands at breakfast.
Every story in tomorrow's AI news, rebuilt in plain English — five minutes, sources linked, free forever.
By joining you agree to receive Article's daily newsletter — unsubscribe in one click. Privacy