AI in Clinical Trials: Is the Evidence Sufficiently Transparent to Appraise?

As AI interventions enter clinical trials, their evaluation requires scrutiny of both conventional trial methods and the technology being tested.

A systematic review published in The BMJ on 13 October 2026 examined 68 trial reports, 15 protocols and 378 registrations involving large language model interventions in medicine. Reporting was assessed against established guidance, including CONSORT 2025, SPIRIT 2025 and their AI extensions. Most trial reports described small, single-site randomised studies.

For readers, an important question is whether the intervention is described sufficiently clearly to understand what was evaluated. The model version, prompts, input information and extent of human involvement can affect AI outputs. These details matter when interpreting findings and considering whether they apply elsewhere.

Critical appraisal should also examine the comparator, outcome selection and clinical importance of any observed benefit. An improvement in the quality of generated text, for example, does not by itself demonstrate better patient outcomes. Similarly, performance in a tightly controlled setting may not establish effectiveness in routine practice.

Incomplete reporting creates uncertainty about these questions. It does not automatically establish that a trial was poorly conducted, but it limits readers’ ability to assess validity, reproduce the intervention and judge applicability.

For clinicians and medical affairs teams, the implication is practical: claims about AI require careful examination of study design, effect estimates, uncertainty and clinical relevance, alongside a clear understanding of how the technology was used.

Read the publication: Reporting completeness of clinical trials evaluating large language model interventions in medicine: systematic review.

To discuss bespoke training in critical appraisal of clinical trials and AI research, contact office@criticalappraisal.com.