AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: What The Microsoft AI Study Compared—and What It Didn’t Measure on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

A study published in Science on October 22 reported that Microsoft’s MAI-DxO system achieved 83.2% diagnostic accuracy on 304 difficult medical cases, compared with 36.1% for generalist physicians using reference materials. The test used historical, text-based cases rather than real patients, and compared a system of AI agents with individual generalists—not specialists or medical teams.

Microsoft’s MAI-DxO diagnostic AI system scored 83.2% on 304 difficult medical case records, compared with 36.1% for generalist physicians using textbooks and the internet, in a peer-reviewed study published October 22 in Science. The results indicate strong performance on the study’s test, but the cases were historical and text-based, and the comparison did not include specialists, physician teams or real-time patient care.

The paper, titled “Sequential diagnosis with language models,” reports results from two evaluations. On 304 complex case records from the New England Journal of Medicine, MAI-DxO reached 83.2% diagnostic accuracy. Generalist physicians averaged 36.1% with textbooks and internet access, and 22.3% without those resources. In a separate set of 325 emergency department cases, the reported scores were 80.3% for the AI and 46.8% for physicians.

The benchmark, called the Sequential Diagnosis Benchmark, begins with a short case summary. The system must request additional information—such as answers to questions or test results—from a simulated gatekeeper holding the case file. The benchmark scores the final diagnosis and assigns costs to requested tests, aiming to capture some of the decisions involved in working through a diagnosis rather than simply selecting an answer from a fully described case.

Microsoft’s reported results are its own evaluation of its system. The paper is peer-reviewed, but that does not mean independent researchers have reproduced the findings. The supplied account notes that the full Science paper is behind a paywall and that some detailed benchmark descriptions come from an earlier, non-peer-reviewed preprint; those details may differ from the final paper.

At a glance
reportWhen: Published October 22, 2026; a prospecti…
The developmentMicrosoft’s MAI-DxO diagnostic system was evaluated in a peer-reviewed Science paper using simulated case records, with results that came with important limits on what the comparison shows.
Thorsten Meyer AI
Science · 22 Oct 2026

Infographic 01 · The scoreboard

83% vs. 36%: the number, and who it was measured against.

Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.

83.2%MAI-DxO on 304 hard NEJM cases
36.1%generalist physicians with textbooks and internet
22.3%generalist physicians without references
80.3 / 46.8emergency cases: MAI-DxO vs. physicians, in %

Hard cases (NEJM)

Diagnostic accuracy, 304 case records

  • MAI-DxOno test budget83.2%
  • GPT-5 alonesame benchmark, no orchestration78.9%
  • Physicians, open booktextbooks and internet36.1%
  • Physicians, closed bookno references22.3%

Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.

Who was compared

The fine print behind “beats doctors”

  • Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
  • Working alone: no colleagues; in the preprint also no search engines or language models.
  • Text-only, historical cases: no real patients, no physical exam, no imaging.
  • Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
  • Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.

Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI

What the Scores Can Tell Hospitals

The findings offer evidence that language models can perform well on selected, difficult diagnostic cases when given an iterative process for asking questions and requesting tests. The benchmark’s cost scoring also addresses a practical issue: diagnosis involves choices about what information to seek, not just the final answer.

But the results do not show that MAI-DxO can replace doctors or improve outcomes in routine care. The comparison was between an AI system operating as a panel of agents and individual generalist physicians working alone. The physicians were not specialists, colleagues working as a team, or the full clinical staff involved in actual care. The cases were selected for difficulty, and the reported test did not involve patients receiving care.

For hospitals, insurers and patients, the distinction matters. A high score on a curated case set is not evidence by itself that a system is safe, affordable or effective in a hospital workflow. Those questions require testing with real patients, appropriate clinical comparisons and measures beyond whether the system reaches the published diagnosis.

Products Worth Considering

Logitech MX Creative Console with 9 Customizable Keys, Graphite

Logitech MX Creative Console with 9 Customizable Keys, Graphite

Bose QuietComfort Wireless Noise-Cancelling Headphones, Moonlight Grey

Bose QuietComfort Wireless Noise-Cancelling Headphones, Moonlight Grey

As an affiliate, we earn on qualifying purchases.

How the Benchmark Was Built

Microsoft’s paper appeared in Science on October 22, 2026. An earlier version was posted on arXiv in June 2025 and had not been peer-reviewed. According to the supplied source material, that preprint reported an 80% score for an earlier system running on OpenAI’s o3 model, compared with 20% for physicians. Results from that earlier version should not be treated as identical to the figures in the Science paper.

MAI-DxO organizes a language model as a group of virtual physicians that propose possible diagnoses, choose questions or tests and challenge one another over multiple rounds. In Microsoft’s chart, GPT-5 alone scored 78.9% on the difficult cases, close to MAI-DxO’s 83.2%. That comparison suggests much of the measured performance may come from the underlying model; the supplied material says the orchestration layer added about four percentage points and reduced estimated test costs.

The earlier preprint described further methodological choices: a model judged diagnoses against the published final diagnosis; the gatekeeper generated synthetic results when a requested test was absent from the record; and test costs were estimated using billing codes and a 2023 U.S. hospital price list, plus a fixed physician-visit fee. The authors described those cost figures as approximations. The preprint also said physicians assessed a subset of cases while the AI was tested across all 304, a difference relevant to interpreting that version’s comparison.

“1.7 times as accurate as physicians on emergency department cases, more than twice as accurate as physicians using textbooks and the internet on complex cases, and nearly four times as accurate as physicians without those resources.”

— Microsoft, in a blog post titled “Building Medical Superintelligence”

Limits the Study Leaves Open

The study does not establish how MAI-DxO would perform with live patients, physical examinations or imaging. It also does not compare the system with specialist physicians or teams, or show whether using it improves patient outcomes, reduces delays or changes the rate of missed diagnoses in routine practice.

The source material says the NEJM case set contained no healthy patients and no harmless explanations. That means the evaluation could not measure how often the system would raise a false alarm when a patient did not have a serious condition. The study’s diagnostic-accuracy score alone cannot answer that safety question.

Other details remain limited by access to the final paper. The supplied material draws some specifics from the 2025 preprint and cautions that the final Science version may differ. Independent replication has not been established in the information provided. It is also unclear how the estimated test costs would translate across health systems, where prices, available tests and clinical workflows vary.

Testing MAI-DxO With Patients

The next development described in the source material is a prospective study at Beth Israel Deaconess Medical Center in Boston. Consenting patients are expected to interact with the system alongside their normal care. This would move evaluation beyond historical case records, although the supplied account does not give a start date, sample size or planned outcome measures.

Results from that work could help clarify how the system handles real clinical information and fits alongside clinicians. Readers should look for details on patient safety, false alarms, diagnostic accuracy, clinician oversight and whether the study compares outcomes against usual care. Until such evidence is reported, the Science results are best understood as performance on a specific benchmark—not proof of effectiveness in hospitals.

Key Questions

What did Microsoft’s MAI-DxO study report?

The Science paper reported 83.2% diagnostic accuracy on 304 difficult NEJM case records for MAI-DxO. Generalist physicians scored 36.1% with textbooks and internet access, and 22.3% without them, on the reported comparison.

Did the study test the AI on real patients?

No. The evaluation used historical, text-based cases in a simulator. The supplied source material describes a prospective study with consenting patients at Beth Israel Deaconess Medical Center as a planned next step.

Were specialists included in the comparison?

The comparison described in the source material involved primary care doctors and hospital generalists working alone. It did not include specialists or physician teams.

Does peer review mean the results have been independently confirmed?

No. Peer review means the paper was reviewed for publication; it does not establish that independent researchers have reproduced the results. The supplied material identifies the results as Microsoft’s evaluation of its system.

What remains unknown about the system?

The study does not establish performance in routine clinical care, patient outcomes, safety with false alarms, or how estimated test costs would translate to hospitals. Those questions require further evaluation, including the planned prospective patient study.

Source: ThorstenMeyerAI.com

You May Also Like

Transforming Patient-Doctor Interactions With AI: AMIE’s Real-Time Clinical Video Capabilities

Google Research and DeepMind showcase AMIE conducting live clinical video consultations with actors, but it remains experimental and unapproved for patient care.

How Benefit Check Bots Improve Access To Medicaid, SNAP, And EITC

Automated benefit screening tools are helping clinics and nonprofits identify more eligible low-income families for vital programs, reducing manual effort and increasing efficiency.

Comb-Over No More: Why Men’s Hair Transplants Are Flourishing

Men’s hair transplants are increasingly popular, driven by changing perceptions and advancements in procedures, signaling a shift away from comb-overs.

5 Gut Symptoms That Deserve A Place On Your Wellness Radar

IdeaNavigator AI proposes testing a role-filtered monitor for fast-moving wellness developments, using a gut-symptom article as an example.