AI models develop hiring biases faster than humans, Princeton study finds

Large language models are more likely than humans to form hiring biases, according to research from Princeton University and the University of Chicago published at ICML in July.

The researchers ran ChatGPT, Claude, and Gemini through a simulated hiring scenario where each model acted as a hiring consultant for 40 job openings across four fictional ethnic groups. All candidates were equally likely to succeed at every job, but the models didn't know this.

The models quickly began segregating candidates by ethnic group into specific job categories. On a segregation scale where 2 indicates complete separation, human participants in the original psychology study scored 0.84. The AI models scored roughly 65% higher. OpenAI's reasoning model o3 scored 1.83, approaching the maximum possible segregation.

"LLMs really are eager to create generalizations from limited data," says Ryan Liu, a PhD student at Princeton and study coauthor. "That's literally a lot of what they're optimized for."

The bias stems from how language models are trained. They excel at extracting patterns from small datasets—a skill that makes them useful for coding and logic puzzles but dangerous in hiring contexts. When an LLM observed one candidate from a group fail at a job, it quickly assumed all members of that group would fail at similar roles, even with no statistical basis.

Newer models with higher reasoning capabilities showed stronger biases. DeepSeek's R1 and OpenAI's o3 both demonstrated more aggressive stereotyping than older versions.

Telling models to "be fair" did little to reduce bias. However, two interventions worked. When models were promised a bonus for diverse hiring outcomes, they became significantly less biased. When given relevant personal information about candidates—age, education, skills—rather than irrelevant details like hair color, they also stereotyped less.

The finding carries immediate weight as companies deploy LLMs to screen resumes and conduct interviews. In real-world hiring, models don't receive instant feedback about whether hires succeed, but eventually performance data does trickle in. When it does, models risk forming biases based on incomplete or unrepresentative samples.

"These novel biases—they're sort ever present," Liu says. Unlike human stereotypes learned from society, AI-generated biases emerge from a model's specific experience with data.

Cornell computer scientist Angelina Wang notes that improved memory features in chatbots could amplify the problem. As models retain more conversation history, they may "over-index on the same kinds of behaviors" they've seen before. The challenge: users want personalization, but excessive memory can entrench biases.

OpenAI and Anthropic did not respond to requests for comment.