JMIR Publications

ABSTRACT

Background:

The burgeoning volume of scientific literature being generated today places a great burden on evidence reviewers. On average, only 2% to 8% of articles yielded by a search strategy are ultimately included in a systematic review. Due to the burden of increasing information loads, there is a demand for methods that improve efficiency while maintaining accuracy in performing evidence reviews.

Objective:

This systematic review aims to determine the accuracy and efficiency of AI-assisted abstract selection compared to manual abstract selection, as assessed by diagnostic performance and workload saved over sampling (WSS).

Methods:

Two reviewers searched PubMed, Proquest, and Cochrane Library for studies evaluating the diagnostic performance and/or workload savings achieved by any AI tool, whether through full or semi-automation, in the title and abstract screening phase of literature review. Variance-weighted random effects meta-analysis was done to generate univariate measures of sensitivity, specificity, and WSS for the studies using RevMan verson 4.3 and the ‘meta’ and ‘mada’ packages on R version 4.3.1. Bivariate analysis was also performed for the measures of diagnostic accuracy and a hierarchal summary operating characteristics curve (HSROC) was generated.

Results:

Twenty-two studies were included in this review, where 13 reported diagnostic performance, 14 reported WSS, and five studies reported both outcomes. In fully automated workflows, AI tools had a sensitivity of 85.6% (95% CI: 60.8%-95.8%) and a specificity of 88.7% (95% CI: 58.7%-97.7%) with considerable heterogeneity, which likely stems from the differences in the SRs and AI techniques used. In semi-automated workflows, sensitivity was 87.6% (95% CI: 77.2%-93.6%) and specificity was 94.1% (95% CI: 60.0%-99.4%) also with considerable heterogeneity. Among studies on full automation, the median workload savings for 100% recall was 50.0% (IQR: 10.2), while for studies on semi-automation, the median workload savings was 55.6% (IQR: 16.4).

Conclusions:

Given the findings of this review, the diagnostic performance of AI tools appeared to be superior when used in semi-automated workflows rather than fully automated ones. This suggest that AI tools hold great potential in augmenting the accuracy and efficiency of human reviewers during study selection in literature review.