🔍 Read the full analysis: What The Microsoft AI Study Compared—and What It Didn’t Measure on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A study published in Science on October 22 reported that Microsoft’s MAI-DxO system achieved 83.2% diagnostic accuracy on 304 difficult medical cases, compared with 36.1% for generalist physicians using reference materials. The test used historical, text-based cases rather than real patients, and compared a system of AI agents with individual generalists—not specialists or medical teams.
Microsoft’s MAI-DxO diagnostic AI system scored 83.2% on 304 difficult medical case records, compared with 36.1% for generalist physicians using textbooks and the internet, in a peer-reviewed study published October 22 in Science. The results indicate strong performance on the study’s test, but the cases were historical and text-based, and the comparison did not include specialists, physician teams or real-time patient care.
The paper, titled “Sequential diagnosis with language models,” reports results from two evaluations. On 304 complex case records from the New England Journal of Medicine, MAI-DxO reached 83.2% diagnostic accuracy. Generalist physicians averaged 36.1% with textbooks and internet access, and 22.3% without those resources. In a separate set of 325 emergency department cases, the reported scores were 80.3% for the AI and 46.8% for physicians.
The benchmark, called the Sequential Diagnosis Benchmark, begins with a short case summary. The system must request additional information—such as answers to questions or test results—from a simulated gatekeeper holding the case file. The benchmark scores the final diagnosis and assigns costs to requested tests, aiming to capture some of the decisions involved in working through a diagnosis rather than simply selecting an answer from a fully described case.
Microsoft’s reported results are its own evaluation of its system. The paper is peer-reviewed, but that does not mean independent researchers have reproduced the findings. The supplied account notes that the full Science paper is behind a paywall and that some detailed benchmark descriptions come from an earlier, non-peer-reviewed preprint; those details may differ from the final paper.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Scores Can Tell Hospitals
The findings offer evidence that language models can perform well on selected, difficult diagnostic cases when given an iterative process for asking questions and requesting tests. The benchmark’s cost scoring also addresses a practical issue: diagnosis involves choices about what information to seek, not just the final answer.
But the results do not show that MAI-DxO can replace doctors or improve outcomes in routine care. The comparison was between an AI system operating as a panel of agents and individual generalist physicians working alone. The physicians were not specialists, colleagues working as a team, or the full clinical staff involved in actual care. The cases were selected for difficulty, and the reported test did not involve patients receiving care.
For hospitals, insurers and patients, the distinction matters. A high score on a curated case set is not evidence by itself that a system is safe, affordable or effective in a hospital workflow. Those questions require testing with real patients, appropriate clinical comparisons and measures beyond whether the system reaches the published diagnosis.
Products Worth Considering

Logitech MX Creative Console with 9 Customizable Keys, Graphite

Jackery HomePower 3000 Portable Power Station, 3072Wh, 3600W

Bose QuietComfort Wireless Noise-Cancelling Headphones, Moonlight Grey
As an affiliate, we earn on qualifying purchases.
How the Benchmark Was Built
Microsoft’s paper appeared in Science on October 22, 2026. An earlier version was posted on arXiv in June 2025 and had not been peer-reviewed. According to the supplied source material, that preprint reported an 80% score for an earlier system running on OpenAI’s o3 model, compared with 20% for physicians. Results from that earlier version should not be treated as identical to the figures in the Science paper.
MAI-DxO organizes a language model as a group of virtual physicians that propose possible diagnoses, choose questions or tests and challenge one another over multiple rounds. In Microsoft’s chart, GPT-5 alone scored 78.9% on the difficult cases, close to MAI-DxO’s 83.2%. That comparison suggests much of the measured performance may come from the underlying model; the supplied material says the orchestration layer added about four percentage points and reduced estimated test costs.
The earlier preprint described further methodological choices: a model judged diagnoses against the published final diagnosis; the gatekeeper generated synthetic results when a requested test was absent from the record; and test costs were estimated using billing codes and a 2023 U.S. hospital price list, plus a fixed physician-visit fee. The authors described those cost figures as approximations. The preprint also said physicians assessed a subset of cases while the AI was tested across all 304, a difference relevant to interpreting that version’s comparison.
“1.7 times as accurate as physicians on emergency department cases, more than twice as accurate as physicians using textbooks and the internet on complex cases, and nearly four times as accurate as physicians without those resources.”
— Microsoft, in a blog post titled “Building Medical Superintelligence”
Limits the Study Leaves Open
The study does not establish how MAI-DxO would perform with live patients, physical examinations or imaging. It also does not compare the system with specialist physicians or teams, or show whether using it improves patient outcomes, reduces delays or changes the rate of missed diagnoses in routine practice.
The source material says the NEJM case set contained no healthy patients and no harmless explanations. That means the evaluation could not measure how often the system would raise a false alarm when a patient did not have a serious condition. The study’s diagnostic-accuracy score alone cannot answer that safety question.
Other details remain limited by access to the final paper. The supplied material draws some specifics from the 2025 preprint and cautions that the final Science version may differ. Independent replication has not been established in the information provided. It is also unclear how the estimated test costs would translate across health systems, where prices, available tests and clinical workflows vary.
Testing MAI-DxO With Patients
The next development described in the source material is a prospective study at Beth Israel Deaconess Medical Center in Boston. Consenting patients are expected to interact with the system alongside their normal care. This would move evaluation beyond historical case records, although the supplied account does not give a start date, sample size or planned outcome measures.
Results from that work could help clarify how the system handles real clinical information and fits alongside clinicians. Readers should look for details on patient safety, false alarms, diagnostic accuracy, clinician oversight and whether the study compares outcomes against usual care. Until such evidence is reported, the Science results are best understood as performance on a specific benchmark—not proof of effectiveness in hospitals.
Key Questions
What did Microsoft’s MAI-DxO study report?
The Science paper reported 83.2% diagnostic accuracy on 304 difficult NEJM case records for MAI-DxO. Generalist physicians scored 36.1% with textbooks and internet access, and 22.3% without them, on the reported comparison.
Did the study test the AI on real patients?
No. The evaluation used historical, text-based cases in a simulator. The supplied source material describes a prospective study with consenting patients at Beth Israel Deaconess Medical Center as a planned next step.
Were specialists included in the comparison?
The comparison described in the source material involved primary care doctors and hospital generalists working alone. It did not include specialists or physician teams.
Does peer review mean the results have been independently confirmed?
No. Peer review means the paper was reviewed for publication; it does not establish that independent researchers have reproduced the results. The supplied material identifies the results as Microsoft’s evaluation of its system.
What remains unknown about the system?
The study does not establish performance in routine clinical care, patient outcomes, safety with false alarms, or how estimated test costs would translate to hospitals. Those questions require further evaluation, including the planned prospective patient study.
Source: ThorstenMeyerAI.com
