A threshold analysis assessed the credibility of conclusions from network meta-analysis

TY - JOUR

T1 - A threshold analysis assessed the credibility of conclusions from network meta-analysis

AU - Caldwell, Deborah M

AU - Ades, A E

AU - Dias, Sofia

AU - Watkins, Sarah H

AU - Li, Tianjing

AU - Taske, Nichole

AU - Welton, Nicky J.

AU - Naidoo, Bhash

N1 - © 2016 The Authors. Published by Elsevier Inc.

PY - 2016/12/1

Y1 - 2016/12/1

N2 - Objective: To assess the reliability of treatment recommendations based on network meta-analysis (NMA)Study design: We consider evidence in an NMA to be potentially biased. Taking each pair-wise contrast in turn we use a structured series of threshold analyses to ask: (a) “How large would the bias in this evidence-base have to be before it changed our decision?” and (b) “If the decision changed, what is the new recommendation?” We illustrate the method via two NMAs in which a GRADE assessment for NMAs has been implemented: weight-loss and osteoporosis.Results. Four of the weight-loss NMA estimates were assessed as “low” and 6 as “moderate” quality by GRADE; for osteoporosis 6 were “low”, 9 “moderate” and 1 “high”. The threshold analysis suggests plausible bias in 3 of 10 estimates in the weight-loss network could have changed the treatment recommendation. For osteoporosis plausible bias in 6 of 16 estimates could change the recommendation. There was no relation between plausible bias changing a treatment recommendation and the original GRADE assessments.Conclusions. Reliability judgements on individual NMA contrasts do not help decision makers understand whether a treatment recommendation is reliable. Threshold analysis reveals whether the final recommendation is robust against plausible degrees of bias in the data.

AB - Objective: To assess the reliability of treatment recommendations based on network meta-analysis (NMA)Study design: We consider evidence in an NMA to be potentially biased. Taking each pair-wise contrast in turn we use a structured series of threshold analyses to ask: (a) “How large would the bias in this evidence-base have to be before it changed our decision?” and (b) “If the decision changed, what is the new recommendation?” We illustrate the method via two NMAs in which a GRADE assessment for NMAs has been implemented: weight-loss and osteoporosis.Results. Four of the weight-loss NMA estimates were assessed as “low” and 6 as “moderate” quality by GRADE; for osteoporosis 6 were “low”, 9 “moderate” and 1 “high”. The threshold analysis suggests plausible bias in 3 of 10 estimates in the weight-loss network could have changed the treatment recommendation. For osteoporosis plausible bias in 6 of 16 estimates could change the recommendation. There was no relation between plausible bias changing a treatment recommendation and the original GRADE assessments.Conclusions. Reliability judgements on individual NMA contrasts do not help decision makers understand whether a treatment recommendation is reliable. Threshold analysis reveals whether the final recommendation is robust against plausible degrees of bias in the data.

KW - mixed treatment comparison, comparative effectiveness, health technology assessment, GRADE, reliability, quality assessment, bias

U2 - 10.1016/j.jclinepi.2016.07.003

DO - 10.1016/j.jclinepi.2016.07.003

M3 - Article

SN - 0895-4356

VL - 80

SP - 68

EP - 76

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

ER -