27 Physicians, Nearly 10,000 Ratings: What a Blinded Study Reveals About AI Answers

Most evaluations of AI in medicine report averages. For deciding whether a system is safe to use in practice, averages are the wrong measure. What matters is how bad the worst answer can get.

A paper published in Communications Medicine in April 2026 measured both. It is open access and unusually careful in its methodology.

How it was set up

A group at a teaching hospital in Shenzhen built an evaluation framework with three distinguishing features. First, no exam questions. The material came from real cases pulled from the hospital information system, supplemented by structured clinical examination scenarios, 685 cases in total across thirteen specialties. Second, only qualified specialists rated the answers, twenty-seven of them with an average of around 26 years in practice. Third, rating was blinded: which answer came from which model was not visible, and telltale markers such as emoji or distinctive formatting were stripped out.

Seven models were tested, all released within a few weeks of each other in early 2025 so that no system enjoyed a generational advantage. That produced 4,795 answers and 9,856 individual ratings.

The scale is the clever part

Rather than marking answers right or wrong, the group used a ten-point scale tied to physician career stages. Roughly: below two means medically untrained, around four to five means a beginning practitioner, around six corresponds to a physician with three to six years of experience, around eight to an experienced specialist, above nine to a recognised expert.

This makes the results legible without softening them. You immediately know what a six means.

The headline result

On raw scores, a Google model led. After statistical adjustment for confounders, notably answer length and differences between raters, a different system came out on top with a mean of 6.36. On this scale that corresponds to a physician with roughly three to six years of practice.

Four of the seven models scored significantly above six. The two small distilled variants fell well behind.

The finding that matters more than the average

Across all models, 7.2 percent of answers were rated incompetent, meaning below the level of a beginning physician. Depending on the model, that rate ranged from roughly three to nineteen percent.

This is the real message. A system that answers at resident level on average but falls below any defensible standard in one case in twenty cannot be deployed without review. A good average is not a floor.

Failure rates also varied by specialty. Models went wrong more often in intensive care, neonatology and paediatric surgery than elsewhere.

Forty fabricated facts

The panel documented forty instances of invented medical content. These included fabricated drugs and dosages, fabricated conditions, fabricated physical examination findings and, most frequently, incorrect interpretation of genetic variants.

Two paediatric examples make the risk concrete. In one case a model recommended an antibiotic at nearly twice the guideline dose. In another, a model endorsed a drug for patients under eighteen against national prescribing guidance. In further cases, models invented family and exposure histories that appeared nowhere in the presented case.

The authors note that errors of this kind would be treated as malpractice if a human made them.

The finding almost nobody expects

One secondary observation deserves particular attention: longer answers received systematically higher ratings. This held regardless of whether the extra length contributed anything clinically.

That does not just affect research. It affects anyone judging AI output day to day. A long, well-structured text reads as more competent than a terse one, whether or not it is. If you are evaluating a documentation tool, be aware of the bias and look deliberately at substance rather than volume.

What the study does not show

Three limitations belong here. The models tested date from early 2025 and are several generations old by now, so the numbers do not describe the current state of the art. The work was single-centre and monolingual. And expert rating of open-ended answers is inherently more subjective than marking multiple choice, even though agreement between raters at model level was very high.

What follows for practice

The authors are unambiguous: computational testing alone is not sufficient, and continuous oversight by qualified humans is required.

For your own practice that reduces to one question. Not how good the system is on average, but how quickly you can tell when it is wrong.

That shifts the decisive criterion from model quality to verifiability. A tool that traces every statement back to its source is worth more than one that scores a point higher on a benchmark but keeps its reasoning opaque.

This is what the citation feature in Psynex is for. Every statement in a generated report carries a reference that links back to the original passage in your notes or transcript. You can see immediately what a formulation rests on, instead of signing off a finished text on trust. Free to try.