Microsoft AI beats doctors in simulated diagnosis test

Microsoft's new artificial intelligence system could help doctors solve some of medicine's toughest diagnostic puzzles, with early numbers showing impressive results. In a study published on Oct. 22 in the journal Science, Microsoft's AI diagnostic system correctly identified 83% of complex medical cases, compared with 36% for generalist physicians who could consult reference materials.

The technology, called the MAI Diagnostic Orchestrator (MAI-DxO), uses multiple AI agents that work together like a panel of doctors discussing a difficult case. The AI begins with limited information, asks follow-up questions, considers which medical tests might help identify the problem, and has its agents challenge one another's conclusions as new information becomes available.

Study limitations

The research took place in a simulation using 304 complex medical cases from The New England Journal of Medicine. The physicians in the comparison were generalists working individually, not specialists who typically collaborate on difficult cases in hospitals. Microsoft did not directly compare its system against teams of medical specialists.

The simulation relied on an AI system to supply information from existing case reports, generating plausible responses when details weren't available. This approach cannot fully reproduce the uncertainty doctors encounter with actual patients, who sometimes struggle to describe symptoms and may have medical records with gaps.

Performance comparison

Researchers evaluated the technology using 304 complex medical cases and found MAI-DxO correctly diagnosed approximately 83% of those cases. Generalist physicians working alone achieved 36% accuracy when they could use reference materials, dropping to 22% without those resources. In emergency department cases, the AI achieved approximately 80% accuracy, compared with 47% for physicians.

AI as a tool to enhance, not replace, human doctors, stressing the irreplaceable value of personal interaction and physician oversight for diagnoses.

Cost implications

Microsoft believes its technology could help doctors make better decisions about which tests patients need. In one cost-conscious configuration, MAI-DxO achieved approximately 81% accuracy on complex cases while generating an estimated $2,241 in diagnostic costs per case. Physicians using reference materials averaged approximately $4,725 in estimated diagnostic costs.

However, some configurations designed to maximize AI accuracy generated higher testing costs. The researchers estimated diagnostic workup costs, including medical testing and standardized visit expenses, but these figures don't represent actual patient bills or the cost of operating the AI.

Future testing

Microsoft CEO of AI Mustafa Suleyman has described the company's larger ambition as "medical superintelligence," developing AI systems capable of eventually exceeding today's strongest medical teams. Microsoft is collaborating with Mayo Clinic on AI technology that could help clinicians anticipate how a patient's medical condition might develop.

The company is preparing to test its AI in real hospitals to determine whether the demonstrated benefits hold up in actual clinical settings. Whether Microsoft's AI can deliver similar advantages during everyday patient care remains an open question that will require further investigation.