AI in Psychiatry: What a New Benchmark Reveals About Strengths and Blind Spots

Discussion of AI in mental health tends to split into two stories. One says the models are essentially there. The other says they are dangerous nonsense. Neither side brings many numbers.

A paper published in npj Digital Medicine in April 2026 supplies them. It is open access, and the interesting part is not the leaderboard. It is where the models fall over.

What the benchmark measures

Earlier evaluations leaned heavily on interview transcripts, social media posts or synthetic dialogue. All three share a problem: they barely capture the structure of real diagnostic reasoning.

PsychiatryBench takes a different route. Every item is drawn from established psychiatric textbooks and casebooks, selected and annotated by licensed psychiatrists. The result is 5,188 tasks across eleven categories, spanning diagnosis, treatment planning, longitudinal follow-up, management planning and structured exam formats.

Fifteen models were tested, including current systems from Anthropic, OpenAI, Google and DeepSeek alongside medically specialised models such as MedGemma.

Where the models are strong

Two systems lead with averages around 84 percent, followed by a tightly packed field. Performance is highest where extended, coherent reasoning is required: on multi-step case progressions and on structuring a diagnostic approach, the best models exceed ninety percent.

This fits a pattern visible elsewhere. Language models are good at organising existing information, summarising it and rendering it in a form someone else can follow.

Where they fail

The most instructive number in the paper is a low one. On the task of classifying specific disorders, even the strongest model reached only around 45 percent exact agreement with expert labels.

The cause is not a technical shortcoming. The task is multi-label by nature, and overlapping symptoms and comorbidity blur the category boundaries. You know this from your own caseload. Where an adjustment disorder ends and a depressive episode begins is not always settled among colleagues either.

That is an important qualification for anyone considering automated coding. Models are useful at proposing candidates and catching formal errors. They are not reliable at deciding which of several applicable diagnoses is the leading one.

A second finding: on tasks requiring fine discrimination between clinically similar options, results varied substantially between models. Some also shifted measurably when the question format changed while the content stayed the same.

The specialist paradox

The expectation was that medically specialised models would beat general-purpose systems on psychiatric tasks. The opposite happened. Specialised models held an edge on knowledge-intensive work such as exam questions and fine-grained classification, but lost clearly on open-ended tasks demanding abstraction and revision of hypotheses as new information arrives.

The authors conclude that aligning capable general models to clinical norms is the more promising path, rather than building ever narrower specialty models.

The limitations the authors name themselves

The most honest section of the paper is the one on limitations, and for interpretation it matters more than the leaderboard.

The sources are public. The textbooks used are almost certainly present in the training data of the models tested. Part of the performance may reflect memorisation rather than reasoning. The authors say so explicitly.

Textbook cases are too clean. They are pedagogically curated, complete and unambiguous. Real cases are patchy, contradictory and evolving. The scores therefore probably overstate reliability in everyday work.

Psychotherapy is excluded. The benchmark covers adult psychiatry. Psychotherapeutic process, narrative case formulation and therapeutic alliance are explicitly out of scope, as is child and adolescent psychiatry.

The scoring is partly circular. Open-ended answers were graded by another language model, one that was itself among the systems evaluated. The authors maintain that independent validation by human clinicians remains necessary.

Acute presentations are missing. High-acuity situations such as active suicidality or presentations requiring admission are not systematically represented.

What the authors take from it

Their conclusion is notably restrained, which is what makes it usable. At the current state of the technology they see a defensible role for language models in education, documentation and preliminary formulation. Explicitly not in unsupervised decision-making where stakes are high, meaning crisis triage or initiating medication.

That boundary matches what practice suggests. The tasks where models are dependable are those involving ordering and articulating information that already exists. The tasks where they are unreliable are those where clinical judgement decides.

Three consequences for your own practice

Check what the system proposes, not whether it sounds plausible. Fluency says nothing about clinical accuracy. A well-written wrong suggestion is harder to catch than a clumsy one.

Rely on them where they measurably perform. Structuring, summarising, checking formal completeness. These are real time savings with no clinical risk attached.

Keep the judgement. Diagnosis, indication and risk assessment are your decision, not only for liability reasons but because the data shows this is precisely where the models are weakest.

Psynex is built around that division of labour. The software turns your session into structured documentation and offers suggestions you review and approve. The clinical decision stays where it belongs. Free to try.