Investigating the impact of trial retractions on the healthcare evidence ecosystem (VITALITY Study I): retrospective cohort study

VITALITY Collaborative Research Network

doi:10.1136/bmj-2024-082068

Investigating the impact of trial retractions on the healthcare evidence ecosystem (VITALITY Study I): retrospective cohort study

VITALITY Collaborative Research Network

Research output: Contribution to journal › Article › peer-review

Abstract

OBJECTIVE: To investigate the impact of retracted trials on the production and use of healthcare evidence in the evidence ecosystem.

DESIGN: Retrospective cohort study based on forward citation searching.

DATA SOURCES: Retraction Watch up to 5 November 2024.

STUDY SELECTION: Randomised controlled trials in humans that were retracted for any reason.

METHODS: Forward citation searching via Google Scholar and Scopus was used to identify evidence synthesis research (21 November 2024) that quantitatively incorporated retracted trials. Data were independently extracted by two groups of researchers. The results of meta-analyses were updated after exclusion of the retracted trials. The proportions of meta-analyses that changed direction of the pooled effect and/or the significance of the P value were estimated. A generalised linear mixed model was used to investigate the association between the number of included studies and the impact, measured by odds ratio and 95% confidence interval (CI). The impact of distorted evidence on clinical practice guidelines was also investigated on the basis of citation searching.

RESULTS: The searches identified 1330 retracted trials and 847 systematic reviews that quantitatively synthesised retracted trials, with a total of 3902 meta-analyses that could be replicated. After the potential clustering effects were accounted for, the exclusion of the retracted trials led to a change in the direction of the pooled effect in 8.4% (95% CI 6.8% to 10.1%), in its statistical significance in 16.0% (14.2% to 17.9%), and in both direction and significance in 3.9% (2.5% to 5.2%) and a >50% change in the magnitude of the effect in 15.7% (13.5% to 17.9%). An obvious non-linear association existed between the number of included studies and the impact on the results, with a lower number of studies having higher impact (eg, for 10 studies versus ≥20 studies, change of direction: odds ratio 2.63, 95% CI 1.29 to 5.38; P<0.001). Evidence from 68 systematic reviews with conclusions distorted by retracted trials was used in 157 guideline documents.

CONCLUSION: Retracted trials have a substantial impact on the evidence ecosystem, including evidence synthesis, clinical practice guidelines, and evidence based clinical practice. Evidence generators, synthesisers, and users must pay attention to this problem, and feasible approaches that assist with easier identification and correction of such potential contamination are needed.

STUDY REGISTRATION: Open Science Framework (https://osf.io/7eazq/).

Original language	English
Article number	e082068
Number of pages	13
Journal	BMJ (Clinical research ed.)
Volume	389
DOIs	https://doi.org/10.1136/bmj-2024-082068
Publication status	Published - 23 Apr 2025

Keywords

Retrospective Studies
Humans
Randomized Controlled Trials as Topic
Evidence-Based Medicine
Meta-Analysis as Topic

Access to Document

10.1136/bmj-2024-082068Licence: CC BY

Investigating the impact of trial retractions on the healthcare evidence ecosystem (VITALITY Study I): retrospective cohort studyFinal published version, 605 KBLicence: CC BY

Cite this

@article{296d1a8f455240cfbe2969319ea9d756,

title = "Investigating the impact of trial retractions on the healthcare evidence ecosystem (VITALITY Study I): retrospective cohort study",

abstract = "OBJECTIVE: To investigate the impact of retracted trials on the production and use of healthcare evidence in the evidence ecosystem.DESIGN: Retrospective cohort study based on forward citation searching.DATA SOURCES: Retraction Watch up to 5 November 2024.STUDY SELECTION: Randomised controlled trials in humans that were retracted for any reason.METHODS: Forward citation searching via Google Scholar and Scopus was used to identify evidence synthesis research (21 November 2024) that quantitatively incorporated retracted trials. Data were independently extracted by two groups of researchers. The results of meta-analyses were updated after exclusion of the retracted trials. The proportions of meta-analyses that changed direction of the pooled effect and/or the significance of the P value were estimated. A generalised linear mixed model was used to investigate the association between the number of included studies and the impact, measured by odds ratio and 95% confidence interval (CI). The impact of distorted evidence on clinical practice guidelines was also investigated on the basis of citation searching.RESULTS: The searches identified 1330 retracted trials and 847 systematic reviews that quantitatively synthesised retracted trials, with a total of 3902 meta-analyses that could be replicated. After the potential clustering effects were accounted for, the exclusion of the retracted trials led to a change in the direction of the pooled effect in 8.4% (95% CI 6.8% to 10.1%), in its statistical significance in 16.0% (14.2% to 17.9%), and in both direction and significance in 3.9% (2.5% to 5.2%) and a >50% change in the magnitude of the effect in 15.7% (13.5% to 17.9%). An obvious non-linear association existed between the number of included studies and the impact on the results, with a lower number of studies having higher impact (eg, for 10 studies versus ≥20 studies, change of direction: odds ratio 2.63, 95% CI 1.29 to 5.38; P<0.001). Evidence from 68 systematic reviews with conclusions distorted by retracted trials was used in 157 guideline documents.CONCLUSION: Retracted trials have a substantial impact on the evidence ecosystem, including evidence synthesis, clinical practice guidelines, and evidence based clinical practice. Evidence generators, synthesisers, and users must pay attention to this problem, and feasible approaches that assist with easier identification and correction of such potential contamination are needed.STUDY REGISTRATION: Open Science Framework (https://osf.io/7eazq/).",

keywords = "Retrospective Studies, Humans, Randomized Controlled Trials as Topic, Evidence-Based Medicine, Meta-Analysis as Topic",

author = "{VITALITY Collaborative Research Network} and Chang Xu and Shiqi Fan and Yuan Tian and Fuchen Liu and Luis Furuya-Kanamori and Justin Clark and Chao Zhang and Sheng Li and Lifeng Lin and Haitao Chu and Sheyu Li and Su Golder and Yoon Loke and Sunita Vohra and Paul Glasziou and Doi, {Suhail A} and Hui Liu",

year = "2025",

month = apr,

day = "23",

doi = "10.1136/bmj-2024-082068",

language = "English",

volume = "389",

journal = "BMJ (Clinical research ed.)",

issn = "0959-8138",

publisher = "BMJ",

}

TY - JOUR

T1 - Investigating the impact of trial retractions on the healthcare evidence ecosystem (VITALITY Study I)

T2 - retrospective cohort study

AU - VITALITY Collaborative Research Network

AU - Xu, Chang

AU - Fan, Shiqi

AU - Tian, Yuan

AU - Liu, Fuchen

AU - Furuya-Kanamori, Luis

AU - Clark, Justin

AU - Zhang, Chao

AU - Li, Sheng

AU - Lin, Lifeng

AU - Chu, Haitao

AU - Li, Sheyu

AU - Golder, Su

AU - Loke, Yoon

AU - Vohra, Sunita

AU - Glasziou, Paul

AU - Doi, Suhail A

AU - Liu, Hui

PY - 2025/4/23

Y1 - 2025/4/23

N2 - OBJECTIVE: To investigate the impact of retracted trials on the production and use of healthcare evidence in the evidence ecosystem.DESIGN: Retrospective cohort study based on forward citation searching.DATA SOURCES: Retraction Watch up to 5 November 2024.STUDY SELECTION: Randomised controlled trials in humans that were retracted for any reason.METHODS: Forward citation searching via Google Scholar and Scopus was used to identify evidence synthesis research (21 November 2024) that quantitatively incorporated retracted trials. Data were independently extracted by two groups of researchers. The results of meta-analyses were updated after exclusion of the retracted trials. The proportions of meta-analyses that changed direction of the pooled effect and/or the significance of the P value were estimated. A generalised linear mixed model was used to investigate the association between the number of included studies and the impact, measured by odds ratio and 95% confidence interval (CI). The impact of distorted evidence on clinical practice guidelines was also investigated on the basis of citation searching.RESULTS: The searches identified 1330 retracted trials and 847 systematic reviews that quantitatively synthesised retracted trials, with a total of 3902 meta-analyses that could be replicated. After the potential clustering effects were accounted for, the exclusion of the retracted trials led to a change in the direction of the pooled effect in 8.4% (95% CI 6.8% to 10.1%), in its statistical significance in 16.0% (14.2% to 17.9%), and in both direction and significance in 3.9% (2.5% to 5.2%) and a >50% change in the magnitude of the effect in 15.7% (13.5% to 17.9%). An obvious non-linear association existed between the number of included studies and the impact on the results, with a lower number of studies having higher impact (eg, for 10 studies versus ≥20 studies, change of direction: odds ratio 2.63, 95% CI 1.29 to 5.38; P<0.001). Evidence from 68 systematic reviews with conclusions distorted by retracted trials was used in 157 guideline documents.CONCLUSION: Retracted trials have a substantial impact on the evidence ecosystem, including evidence synthesis, clinical practice guidelines, and evidence based clinical practice. Evidence generators, synthesisers, and users must pay attention to this problem, and feasible approaches that assist with easier identification and correction of such potential contamination are needed.STUDY REGISTRATION: Open Science Framework (https://osf.io/7eazq/).

AB - OBJECTIVE: To investigate the impact of retracted trials on the production and use of healthcare evidence in the evidence ecosystem.DESIGN: Retrospective cohort study based on forward citation searching.DATA SOURCES: Retraction Watch up to 5 November 2024.STUDY SELECTION: Randomised controlled trials in humans that were retracted for any reason.METHODS: Forward citation searching via Google Scholar and Scopus was used to identify evidence synthesis research (21 November 2024) that quantitatively incorporated retracted trials. Data were independently extracted by two groups of researchers. The results of meta-analyses were updated after exclusion of the retracted trials. The proportions of meta-analyses that changed direction of the pooled effect and/or the significance of the P value were estimated. A generalised linear mixed model was used to investigate the association between the number of included studies and the impact, measured by odds ratio and 95% confidence interval (CI). The impact of distorted evidence on clinical practice guidelines was also investigated on the basis of citation searching.RESULTS: The searches identified 1330 retracted trials and 847 systematic reviews that quantitatively synthesised retracted trials, with a total of 3902 meta-analyses that could be replicated. After the potential clustering effects were accounted for, the exclusion of the retracted trials led to a change in the direction of the pooled effect in 8.4% (95% CI 6.8% to 10.1%), in its statistical significance in 16.0% (14.2% to 17.9%), and in both direction and significance in 3.9% (2.5% to 5.2%) and a >50% change in the magnitude of the effect in 15.7% (13.5% to 17.9%). An obvious non-linear association existed between the number of included studies and the impact on the results, with a lower number of studies having higher impact (eg, for 10 studies versus ≥20 studies, change of direction: odds ratio 2.63, 95% CI 1.29 to 5.38; P<0.001). Evidence from 68 systematic reviews with conclusions distorted by retracted trials was used in 157 guideline documents.CONCLUSION: Retracted trials have a substantial impact on the evidence ecosystem, including evidence synthesis, clinical practice guidelines, and evidence based clinical practice. Evidence generators, synthesisers, and users must pay attention to this problem, and feasible approaches that assist with easier identification and correction of such potential contamination are needed.STUDY REGISTRATION: Open Science Framework (https://osf.io/7eazq/).

KW - Retrospective Studies

KW - Humans

KW - Randomized Controlled Trials as Topic

KW - Evidence-Based Medicine

KW - Meta-Analysis as Topic

U2 - 10.1136/bmj-2024-082068

DO - 10.1136/bmj-2024-082068

M3 - Article

C2 - 40268307

SN - 0959-8138

VL - 389

JO - BMJ (Clinical research ed.)

JF - BMJ (Clinical research ed.)

M1 - e082068

ER -