Evaluation & error analysis of a machine learning tool to detect Randomized Controlled Trials

A pilot study using articles included in Cochrane Reviews

A machine learning classifier called Tagger has been developed to predict whether an article is an RCT, based on its title, abstract, and number of authors. In this project, we will test Tagger’s predictive accuracy on 12,000 articles taken from 843 Cochrane systematic reviews. The goal of our project is: (a) to assess Tagger performance; (b) to suggest possible improvements to Tagger’s development team (our collaborators).

Key References

• Tagger: http://arrowsmith.psych.uic.edu/cgi-bin/arrowsmith_uic/RCT_Tagger.cgi

• Tagger has already been evaluated in this previously published paper: Cohen AM, Smalheiser NR, McDonagh MS, Yu C, Adams CE, Davis JM, Yu PS. Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine. Journal of the American Medical Informatics Association. 2015 Feb 5;22(3):707-17. https://doi.org/10.1093/jamia/ocu025

Key Background

• This work is part of our NIH-funded grant project (R01LM010817), Text Mining Pipeline to Accelerate Systematic Reviews in Evidence-Based Medicine, which runs through 2021: http://ischool.illinois.edu/research/projects/text-mining-pipeline-accelerate-systematic-reviews-evidence-based-medicine

• Systematic review is a process for synthesizing literature: Hoang L, Schneider J. Opportunities for computer support for systematic reviewing-a gap analysis. In International Conference on Information 2018 Mar 25 (pp. 367-377). Springer, Cham. https://doi.org/10.1007/978-3-319-78105-1_40

• Cochrane produces systematic reviews of medical literature: http://www.cochrane.org

Data Files

Data files were deposited on a public repository and can be accessed via the following DOI links:

Data file 1: “File1 - Sampled Cochrane Reviews that Only Included RCTs.xlsx”

Kansara, Yogeshwar; Hoang, Linh; Dong, Xiaoru; Xie, Jingyi (2019): Sampled Cochrane Reviews that Only Included RCTs. University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-3285089_V2

Data file 2: “File2 - Included Articles from Cochrane reviews.xlsx”:

Kansara, Yogeshwar; Hoang, Linh (2019): Included Articles fromf Cochrane Reviews. University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-8212056_V2

Data file 3: “File3 - Articles With PubMed Identifiers.xlsx”

Kansara, Yogeshwar; Hoang, Linh (2019): Articles With PubMed Identifiers. University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-4623305_V2

Data file 4: “File4 - RCT Tagger Result.xlsx”

Kansara, Yogeshwar; Hoang, Linh (2019): RCT Tagger Results. University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-6773581_V2

Data file 5: “File5 - Error Analysis.xlsx”

Kansara, Yogeshwar; Hoang, Linh; Schneider, Jodi (2019): Error Analysis. University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-3407079_V2