Microsoft says its AI tool beats doctors on diagnosis tests

In short: Microsoft says a new AI system diagnosed medical cases more accurately than human doctors in published tests.

What happened

Microsoft AI CEO Mustafa Suleyman told The Verge that the company has built an AI tool it says can diagnose illnesses “up to four times more accurately” than doctors. Microsoft shared results in the journal Science.

The tool is called the Diagnostic Orchestrator. Microsoft says it works like a panel of medical experts, not a single opinion. It uses multiple “AI agents” (separate AI helpers that each focus on part of the problem) to review symptoms and medical history, decide which tests to request, and then pick a likely diagnosis. This is meant to copy the step by step way doctors investigate a case.

To test it, Microsoft compared the system with human physicians using medical scenarios taken from published case reports and emergency department records. Microsoft also created its own testing set, called SDBench, and says it plans to open-source it, meaning the dataset would be shared publicly so others can check and repeat the tests.

On cases drawn from hundreds of reports in The New England Journal of Medicine, Microsoft says its system reached about 83 percent diagnostic accuracy. In the same setup, generalist doctors scored about 36 percent when they could use reference materials, and about 22 percent when they could not.

On emergency department cases from a large database of admissions at Beth Israel Deaconess Medical Center in Boston, Microsoft reports 80 percent accuracy for its system versus 47 percent on average for human physicians.

Why it matters

If results like this hold up in real hospitals, AI could help doctors avoid missed or delayed diagnoses, especially in busy settings like emergency rooms. But these were tests on prepared case scenarios, not a full clinical rollout, so independent validation and real world trials will matter.

Source: The Verge AI