Could AI bias make health inequalities worse?

Artificial intelligence is becoming increasingly common in healthcare, but concerns remain about whether biased data could make existing health inequalities worse.

A Medscape UK feature, published in editorial collaboration with the Medical Protection Society, has highlighted the risk that AI systems may reproduce, or even increase, the inequalities already present in healthcare data and clinical decision-making.

The concern is not that AI is unsuitable for healthcare. It is that AI tools depend on the information they are trained on, how they are designed, and how their outputs are used in real clinical settings.

If the data reflects unequal access to care, delayed diagnosis, under-recorded symptoms, or different patterns of treatment between patient groups, those gaps may become part of the system itself.

Research examples

The article points to a 2019 US study published in Science, which found racial bias in a commercial algorithm used to guide health decisions. The system used healthcare costs as a proxy for health need. Because less money had historically been spent on Black patients with the same level of illness, the algorithm underestimated their need for extra care.

A more recent study looking at nine large language models also found differences in recommendations when cases were labelled with different sociodemographic identifiers. Examples included mental health assessments being recommended more often for cases labelled as LGBTQ+, and more advanced imaging being recommended for cases labelled as higher income.

These studies are US based, but the issue is relevant more widely. AI tools are developed, sold, adapted and adopted across borders. Healthcare systems may differ, but the risk of unequal data producing unequal outcomes is not limited to one country.

Why this matters

Healthcare data is not neutral. It reflects who accessed care, who was believed, who was diagnosed, what was recorded, and what was missed.

If AI systems are trained on that data without proper safeguards, existing inequalities may be carried forward into future decision-making. In some cases, they may be amplified.

The article also highlights research suggesting that repeated use of biased AI systems may affect clinicians themselves, with one study finding that clinicians who repeatedly interacted with biased AI became more biased over time.

This raises an important question for healthcare systems: how can AI be used safely without allowing technology to give old problems a new form?

The regulatory picture

The EU AI Act includes specific requirements for high-risk AI systems, including some used in healthcare.

The UK does not currently have one single piece of AI legislation. Instead, AI is covered through a mixture of existing data protection, equality, medical device and professional regulation.

The Information Commissioner’s Office has recently consulted on draft guidance on automated decision-making, including profiling. The Medicines and Healthcare products Regulatory Agency is also considering the regulation of AI in healthcare through its National Commission on AI.

The direction is clear: AI in healthcare needs oversight, transparency and safeguards.

For clinicians, that means understanding the limits of any AI tool being used, applying human judgement, and remaining able to challenge or override an AI output where needed.

For patients, the issue is trust. People need to know that technology is supporting care, not quietly shaping decisions in ways they cannot see or question.

AI may become an important part of healthcare, but fairness has to be built into how it is designed, tested, used and monitored.

Back to home