Rigorous Large-Scale Educational RCTs are Often Uninformative: Should We Be Concerned?

TY - JOUR

T1 - Rigorous Large-Scale Educational RCTs are Often Uninformative

T2 - Should We Be Concerned?

AU - Lortie-Forgues, Hugues

AU - Inglis, Matthew

N1 - This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details.

PY - 2019/4/2

Y1 - 2019/4/2

N2 - There are a growing number of large-scale educational Randomized Controlled Trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the EEF (UK) and the NCEE (US) which evaluated interventions aimed at improving academic achievement in K-12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations (SDs). These sat within relatively large confidence intervals (mean width 0.30 SDs) which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects.

AB - There are a growing number of large-scale educational Randomized Controlled Trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the EEF (UK) and the NCEE (US) which evaluated interventions aimed at improving academic achievement in K-12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations (SDs). These sat within relatively large confidence intervals (mean width 0.30 SDs) which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects.

U2 - 10.3102/0013189X19832850

DO - 10.3102/0013189X19832850

M3 - Article

VL - 48

SP - 158

EP - 166

JO - Educational Researcher

JF - Educational Researcher

SN - 0013-189X

IS - 3

ER -