Bias and Errors: How AI Gets Things Wrong

An AI system learns from examples. If the examples are lopsided, the system is lopsided. If the goal it is trained toward is slightly wrong, it gets very good at the slightly wrong thing. And a language model, built to produce likely text, will produce likely-sounding false text whenever it does not know. These are separate problems with separate fixes, and lumping them all under "AI is biased" makes them harder to fix or even talk about.

Where bias gets in

• Data. If a face dataset is mostly light-skinned men, the system will be worse on everyone else. If historical hiring data reflects who got hired, a model trained on it learns to repeat that.
• Target. What the model is told to predict. This is the sneakiest stage, because the data can be accurate and the target still wrong.
• Model and tuning. A match threshold set to catch more shoplifters also flags more innocent shoppers. Human feedback used to tune chatbots carries the raters' views.
• Deployment. A tool tested on one population and used on another. A tool used on some neighborhoods more than others.

Documented cases

The hospital algorithm

In 2019, Ziad Obermeyer and colleagues published a study in Science of a commercial algorithm used by health systems to pick patients for extra care programs. It predicted future health costs as a stand-in for health need. Because less money had historically been spent on Black patients with the same conditions, the algorithm scored them as healthier than equally sick white patients. The researchers estimated that removing the bias would raise the share of Black patients flagged for extra help from 17.7 percent to 46.5 percent, and they worked with the developer to reduce it. Race was never an input. The proxy did the damage.

Faces

Joy Buolamwini and Timnit Gebru's 2018 Gender Shades study tested three commercial gender classifiers. The worst error rate for lighter-skinned men was 0.8 percent. For darker-skinned women, error rates reached 34.7 percent. A year later the National Institute of Standards and Technology tested 189 face recognition algorithms from 99 developers on 18 million images (report NISTIR 8280). Most showed demographic differentials. In one-to-one matching, false positives for some groups were often 10 to 100 times higher. NIST also found that the most accurate algorithms tended to show the smallest differences, which means "face recognition is biased" is too blunt. Some systems are much worse than others, and you have to know which one an agency bought.

In December 2023 the Federal Trade Commission charged Rite Aid over its use of face recognition in hundreds of stores, saying it lacked reasonable safeguards to prevent harm to customers wrongly flagged. The settlement banned Rite Aid from using face recognition for surveillance for five years.

Confidently wrong

In March 2024 The Markup tested New York City's MyCity business chatbot, built on Microsoft's Azure AI services. It told users that landlords could refuse housing vouchers, that businesses could go cash-free and that bosses could take workers' tips. City law says otherwise on all three. It gave different answers to the same question. The city called it a pilot and kept it running.

In May 2025 the Chicago Sun-Times and at least one edition of The Philadelphia Inquirer ran a syndicated summer reading list in which 10 of 15 books did not exist. The freelance writer said AI had partly generated it and called it a huge mistake. Nobody in the chain checked whether the books were real.

Why language models make things up

A language model predicts the next likely word. It has no separate fact database it consults, unless one is connected. When it knows a fact well, the likely words are the true ones. When it does not, the likely words are the ones that sound like a true answer: a plausible book title, a plausible court name, a plausible statistic.

A September 2025 paper by Adam Tauman Kalai and colleagues argues the habit persists because of how models are graded. Most benchmarks give a point for a right answer and nothing for "I don't know," so a model that always guesses scores higher than one that admits uncertainty. Their proposed fix is to change how the popular benchmarks score abstaining.

Reading a hallucination rate

Every hallucination rate is the answer to a particular test. Change the test and the rate changes.

Benchmark	Task	Grader	What it tells you	What it does not
Vectara leaderboard	Summarize a supplied document	Vectara's HHEM model checks consistency with the source	How often a model adds things that are not in the text it was given. Rates listed May 2026 ran from 1.8 to 24.2 percent.	How it does on open questions with no source
OpenAI SimpleQA (Oct 2024)	4,326 short fact questions	A model compares answers to reference answers	Correct, incorrect and not attempted rates. GPT-4o scored under 40 percent correct when released.	Long answers, reasoning, or answers with web search turned on
Your own spot check	Questions about your beat	You, against records	How it does on the things you will actually ask	General performance

Notice the gap. A model can be very faithful when summarizing a document you hand it and still invent facts when asked from memory. That is why the safest use for anyone checking facts is giving the model the source and asking it to work only from that, then checking anyway.