Microsoft Says Its AI Beats Doctors—Here’s The Study’s Scope

🔍 Read the full analysis: Microsoft Says Its AI Beats Doctors—Here’s The Study’s Scope on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get smart everyday buys delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI scored higher than generalist physicians on selected medical cases in a peer-reviewed Science paper published Oct. 22, the company reported. The study tested the system on historical case records in a text-based simulation—not on patients receiving care—so its results do not establish how the system would perform in hospitals or clinics.

On 304 complex cases drawn from the New England Journal of Medicine, MAI-DxO achieved 83.2% diagnostic accuracy. The Science paper reports 36.1% for generalist physicians with textbooks and internet access and 22.3% without those resources. On a separate set of 325 emergency department cases, the reported scores were 80.3% for the AI and 46.8% for physicians. These figures are results from the study, not evidence of outcomes among patients in clinical practice.

The benchmark, called the Sequential Diagnosis Benchmark, began with a short case summary. The AI or physician then had to request more information, such as a question or test result, from a simulated case file. The system was scored on its final diagnosis and the estimated cost of the steps it took. MAI-DxO used a language model with a panel-like process in which virtual agents proposed diagnoses, chose follow-up questions or tests, and challenged one another.

The comparison was between this multi-agent system and individual generalist doctors, not specialist teams. The source account says the physicians were primary care doctors and hospital generalists, and that the cases were difficult teaching cases. It also reports that the doctors were not allowed to use search engines or language models. Microsoft’s published Science results add an open-book comparison, with access to textbooks and the internet.

Why the comparison matters

The findings matter because they suggest that AI systems may help with diagnostic reasoning and test selection, particularly when a case is complicated. But the study’s design limits what can be concluded. It measured performance on curated records, where the correct diagnosis was already known, rather than whether using the system improves care, reduces delays, or avoids harm for people arriving with uncertain symptoms.

The benchmark also assigned costs to tests, making it possible to compare diagnostic accuracy with estimated spending. The source account says these costs were approximations based on billing codes and a 2023 U.S. hospital price list, plus a fixed physician-visit cost. They should not be read as a forecast of what the system would cost in a specific health system. Likewise, the test did not include healthy patients or harmless explanations, so it could not establish how often the system might raise false alarms in a broader population.

For hospitals, insurers and patients, the central takeaway is that higher benchmark accuracy is not the same as proven clinical benefit. Use in care would raise questions about oversight, patient safety, cost, and whether doctors can rely on the system without missing other explanations. The study offers evidence to investigate those questions; it does not answer them.

How the benchmark was built

Medical AI is often tested by giving a system a complete case and asking it to select a diagnosis. Microsoft’s benchmark instead attempted to model a sequence of decisions: what to ask, which investigation to request, and when to stop. A separate “gatekeeper” model held the full case file and supplied requested details. According to the source account, if a requested result was absent from the record, the gatekeeper generated a synthetic result rather than signaling that the information was unavailable.

The source account draws some methodological details from an earlier, non-peer-reviewed 2025 preprint, and cautions that the final Science paper may differ. It says a model judged diagnoses against the published answer using a five-point scale, with physician review of a sample showing good but imperfect agreement. The account also describes the test-price estimates as first-order approximations. Those caveats matter because both scoring and simulated information can affect benchmark results.

The published paper is titled “Sequential diagnosis with language models” and appeared in Science, volume 394, pages 423–428. The source account says all 15 authors were affiliated with Microsoft. It also reports that an earlier preprint used OpenAI’s o3 model and showed different accuracy figures. That earlier result is not the same evaluation as the final paper’s reported figures and should not be combined with them.

““1.7 times as accurate as physicians on emergency department cases””

— Microsoft, in its blog post, “Building Medical Superintelligence”

What the results cannot establish

The study does not show whether MAI-DxO can improve diagnosis or patient outcomes when used alongside clinicians. The source account says a prospective study at Beth Israel Deaconess Medical Center is planned, with consenting patients interacting with the system alongside their normal care. Details such as the study’s size, timeline, evaluation measures and findings are not provided here.

It is also unclear how the system would compare with specialists or teams of clinicians, who were not the physician group in the reported comparison. The source account notes that difficult cases may be referred to specialists or discussed by teams in practice. The benchmark’s selected cases also do not show how often the system would flag rare conditions when a common, less serious explanation is correct.

Finally, peer review does not mean the results have been independently reproduced. The source material identifies the Science paper as peer-reviewed but says the results are Microsoft’s own and have not been reproduced by outside researchers. The reported accuracy and cost estimates should be understood within those limits.

The planned patient study

The next reported step is a prospective study at Beth Israel Deaconess Medical Center. Consenting patients are expected to interact with MAI-DxO alongside their usual care. That setting could provide evidence about how the system operates when information unfolds during real medical encounters, although the source material does not specify when the study will begin or what outcomes it will measure.

Readers should look for details on the study protocol, including whether clinicians can see or override the system’s recommendations, how safety events will be tracked, and whether the evaluation measures patient outcomes as well as diagnostic accuracy. Until those results are available, the Science paper is best read as a benchmark comparison—not a demonstration that the AI can replace a doctor or safely direct routine care.

Key Questions

What did Microsoft’s study find?

The Science paper reports that MAI-DxO scored 83.2% on 304 complex NEJM cases. Generalist physicians scored 36.1% with textbooks and internet access and 22.3% without them. These are benchmark results, not scores from treating patients.

Did the study test the AI on real patients?

No. The reported test used historical, text-based case records in a simulator. The source account says a separate prospective study involving consenting patients is planned at Beth Israel Deaconess Medical Center.

Were specialists included in the comparison?

The source account describes the physician participants as primary care doctors and hospital generalists. It says the comparison did not include specialists, colleagues working together, or physician teams.

Does the paper show that MAI-DxO can replace doctors?

No. The paper reports performance on a specific benchmark. It does not establish that the system improves patient outcomes, works safely in routine care, or can replace clinicians.

Source: ThorstenMeyerAI.com