Harvard study finds AI matched or beat ER doctors on diagnoses

In short: A Harvard-led study found an AI system often matched or outperformed emergency room doctors when making diagnoses from patient records.

What happened

Researchers from Harvard Medical School and Beth Israel Deaconess Medical Center published a study in the journal Science looking at how “large language models” perform in medical work. Large language models are AI systems that generate text, like a very fast autocomplete that can also explain its choices.

In one experiment, the team looked at 76 real patients who came to the Beth Israel emergency room. They compared diagnoses from two attending physicians with diagnoses generated by two OpenAI models called o1 and 4o.

Two other attending physicians graded the diagnoses without knowing whether they came from a human or the AI. The study reported that the o1 model performed slightly better than or about the same as the doctors overall, and it did best at the first “touchpoint,” which is initial ER triage (the first sorting step, like a quick check-in to decide how urgent a case is).

Using the same text information available in the electronic medical record at the time, the o1 model gave an exact or very close diagnosis in 67% of triage cases. One physician scored 55% and the other scored 50%.

Why it matters

This does not mean AI should replace doctors in the emergency room. The researchers said the results point to a need for careful real-world clinical trials and clearer accountability if AI is used. For patients, the key question is whether AI can help doctors make faster, safer decisions, especially when time is tight and information is limited.

Source: TechCrunch AI