External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb

TY - JOUR

T1 - External validation of clinical prediction models

T2 - simulation-based sample size calculations were more reliable than rules-of-thumb

AU - Snell, Kym Ie

AU - Archer, Lucinda

AU - Ensor, Joie

AU - Bonnett, Laura J

AU - Debray, Thomas Pa

AU - Phillips, Bob

AU - Collins, Gary S

AU - Riley, Richard D

N1 - © 2021 The Authors

PY - 2021/7/1

Y1 - 2021/7/1

N2 - INTRODUCTION: Sample size 'rules-of-thumb' for external validation of clinical prediction models suggest at least 100 events and 100 non-events. Such blanket guidance is imprecise, and not specific to the model or validation setting. We investigate factors affecting precision of model performance estimates upon external validation, and propose a more tailored sample size approach.METHODS: Simulation of logistic regression prediction models to investigate factors associated with precision of performance estimates. Then, explanation and illustration of a simulation-based approach to calculate the minimum sample size required to precisely estimate a model's calibration, discrimination and clinical utility.RESULTS: Precision is affected by the model's linear predictor (LP) distribution, in addition to number of events and total sample size. Sample sizes of 100 (or even 200) events and non-events can give imprecise estimates, especially for calibration. The simulation-based calculation accounts for the LP distribution and (mis)calibration in the validation sample. Application identifies 2430 required participants (531 events) for external validation of a deep vein thrombosis diagnostic model.CONCLUSION: Where researchers can anticipate the distribution of the model's LP (e.g. based on development sample, or a pilot study), a simulation-based approach for calculating sample size for external validation offers more flexibility and reliability than rules-of-thumb.

AB - INTRODUCTION: Sample size 'rules-of-thumb' for external validation of clinical prediction models suggest at least 100 events and 100 non-events. Such blanket guidance is imprecise, and not specific to the model or validation setting. We investigate factors affecting precision of model performance estimates upon external validation, and propose a more tailored sample size approach.METHODS: Simulation of logistic regression prediction models to investigate factors associated with precision of performance estimates. Then, explanation and illustration of a simulation-based approach to calculate the minimum sample size required to precisely estimate a model's calibration, discrimination and clinical utility.RESULTS: Precision is affected by the model's linear predictor (LP) distribution, in addition to number of events and total sample size. Sample sizes of 100 (or even 200) events and non-events can give imprecise estimates, especially for calibration. The simulation-based calculation accounts for the LP distribution and (mis)calibration in the validation sample. Application identifies 2430 required participants (531 events) for external validation of a deep vein thrombosis diagnostic model.CONCLUSION: Where researchers can anticipate the distribution of the model's LP (e.g. based on development sample, or a pilot study), a simulation-based approach for calculating sample size for external validation offers more flexibility and reliability than rules-of-thumb.

U2 - 10.1016/j.jclinepi.2021.02.011

DO - 10.1016/j.jclinepi.2021.02.011

M3 - Article

C2 - 33596458

SN - 0895-4356

VL - 135

SP - 79

EP - 89

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

ER -