Artificial intelligence promises to help us make better decisions, automate tasks and solve complex problems. But what happens when an algorithm gets it wrong? What if it makes a decision that looks perfectly logical mathematically, but turns out to be disastrous in the real world?
We open up the black box of AI to revisit four cases that show its limitations: from a chatbot that seemed to understand human emotions to algorithms capable of amplifying financial chaos or reproducing social prejudice.
1 ELIZA, the chatbot that seemed to understand us (but didn't)
In 1966, MIT researcher Joseph Weizenbaum created ELIZA, one of the first programs capable of holding a conversation. Its trick was simple: it rephrased users' sentences and answered with questions, without actually understanding what they were telling it.
Despite its limitations, some people came to trust it and share their personal problems with it. That's how the so-called ELIZA effect was born: our tendency to attribute intelligence and empathy to a machine just because it responds convincingly.
The lesson still holds: an AI seeming to understand us doesn't mean it actually does.
2 A medical algorithm that mistook asthma for an advantage
An AI system designed to predict mortality risk in pneumonia patients found that those who also had asthma had better outcomes. The problem is that it read that correlation as if asthma were a protective factor, without accounting for the fact that these patients might receive more intensive medical care.
The algorithm spotted a pattern in the data, but it didn't grasp the context that explained it. This case shows that correlation doesn't imply causation and that, especially in medicine, an AI's recommendations need to be carefully evaluated before being applied.
3 Flash Crash: the algorithms that amplified financial panic in the US
On May 6, 2010, US financial markets suffered a sudden and extreme drop. The interaction between large sell orders and automated trading systems helped intensify the plunge, as prices swung at a speed that human traders struggled to handle.
The episode, known as the Flash Crash, showed that automation can amplify unexpected movements when several systems interact without sufficient controls. Speed is an advantage, but it can also multiply the consequences of a mistake.
4 COMPAS, the judicial algorithm that inherited our prejudices
In the United States, the COMPAS system has been used to assess the risk of reoffending among people accused of crimes and to support certain judicial decisions.
In 2016, a ProPublica investigation found significant racial differences in its prediction errors: Black defendants were more likely to be wrongly classified as high risk.
The case opened a debate about algorithmic bias and the supposed neutrality of mathematics. If an AI learns from historical data that reflects inequalities, it can reproduce them and give them an air of objectivity.
What do these mistakes teach us?
Although they come from very different fields, all four cases share the same warning: artificial intelligence is neither infallible nor necessarily objective.
Algorithms can mistake correlations for causes, learn from biased data or set off unforeseen consequences. That's why developing AI solutions isn't only about getting them to work, but also about understanding their limitations, evaluating their results and putting oversight mechanisms in place.
More thoughts about this.
Read more.
The Importance of Prompt Engineering.
Cassandra, the lady of NoSQL databases.
Langfuse vs. LangSmith II: Datasets and Evaluation.
Artificial Intelligence in Big Data Spain 2017.
Comments are moderated and will only be visible if they add to the discussion in a constructive way. If you disagree with a point, please, be polite.
Tell us what you think.