#3-95E. AI as the Second Reviewer: Automating Systematic Reviews (CESM, 2025)

Are you overwhelmed by the tedious data extraction process required for building clinical evidence and systematic reviews? This episode explores a recent study evaluating the efficacy of large language models—specifically Elicit and ChatGPT—in replacing one of the two human reviewers traditionally required for data extraction. Analyzing 30 articles across varying review types, the researchers found that both AI tools demonstrated remarkable proficiency, achieving precision, recall, and F1-scores of approximately 90%. The most significant finding is that AI can effectively serve as a reliable second reviewer, fundamentally streamlining evidence synthesis. However, a critical limitation must be noted: error analysis revealed that AI models generated confabulations—fabricated data points—in about 4% of extractions, emphasizing that human oversight remains strictly necessary to verify and reconcile discrepancies. For physicians and clinician-researchers, this breakthrough is highly relevant; adopting AI-assisted extraction can drastically reduce the administrative burden of guideline development and academic research. By minimizing the hours spent on tedious data processing, clinicians can reallocate their valuable time toward patient care and higher-level evidence interpretation, provided they remain vigilant against AI inaccuracies.

Citation: Andersen TH, Marcussen TM, Termannsen AD, Lawaetz TWH, Nørgaard O. Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers. Cochrane Evidence Synthesis and Methods. 2025. DOI: 10.1002/cesm.70036

Disclaimer: This audio summary is based on personal interpretation and does not guarantee the exact content of the original paper. Please refer to the original article for details.