Power contours: optimising sample size and precision in experimental psychology and human neuroscience

TY - JOUR

T1 - Power contours

T2 - optimising sample size and precision in experimental psychology and human neuroscience

AU - Baker, Daniel Hart

AU - Vilidaite, Greta

AU - Lygo, Freya

AU - Smith, Anika

AU - Flack, Tessa Rosemary

AU - Gouws, Andre

AU - Andrews, Tim

N1 - This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details.

PY - 2021/8/2

Y1 - 2021/8/2

N2 - When designing experimental studies with human participants, experimenters must decide how many trials each participant will complete, as well as how many participants to test. Most discussion of statistical power (the ability of a study design to detect an effect) has focussed on sample size, and assumed sufficient trials. Here we explore the influence of both factors on statistical power, represented as a two-dimensional plot on which iso-power contours can be visualised. We demonstrate the conditions under which the number of tri- als is particularly important, i.e. when the within-participant variance is large relative to the between-participants variance. We then derive power contour plots using existing data sets for eight experimental paradigms and methodologies (including reaction times, sensory thresholds, fMRI, MEG, and EEG), and provide example code to calculate estimates of the within- and between-participant variance for each method. In all cases, the within-participant variance was larger than the between-participants variance, meaning that the number of trials has a meaningful influence on statistical power in commonly used paradigms. An online tool is pro- vided (https://shiny.york.ac.uk/powercontours/) for generating power contours, from which the optimal combination of trials and participants can be calculated when designing future studies.

AB - When designing experimental studies with human participants, experimenters must decide how many trials each participant will complete, as well as how many participants to test. Most discussion of statistical power (the ability of a study design to detect an effect) has focussed on sample size, and assumed sufficient trials. Here we explore the influence of both factors on statistical power, represented as a two-dimensional plot on which iso-power contours can be visualised. We demonstrate the conditions under which the number of tri- als is particularly important, i.e. when the within-participant variance is large relative to the between-participants variance. We then derive power contour plots using existing data sets for eight experimental paradigms and methodologies (including reaction times, sensory thresholds, fMRI, MEG, and EEG), and provide example code to calculate estimates of the within- and between-participant variance for each method. In all cases, the within-participant variance was larger than the between-participants variance, meaning that the number of trials has a meaningful influence on statistical power in commonly used paradigms. An online tool is pro- vided (https://shiny.york.ac.uk/powercontours/) for generating power contours, from which the optimal combination of trials and participants can be calculated when designing future studies.

KW - statistical power

KW - sample size

KW - neuroscience

KW - psychophysics

KW - fMRI

KW - MEG

KW - EEG

UR - https://osf.io/ebhnk/

UR - https://shiny.york.ac.uk/powercontours/

U2 - 10.1037/met0000337

DO - 10.1037/met0000337

M3 - Article

SN - 1082-989X

VL - 26

SP - 295

EP - 314

JO - Psychological Methods

JF - Psychological Methods

IS - 3

ER -