Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses

Stefano Coretta; Joseph V. Casillas; Simon Roessig; Michael Franke; Byron Ahn; Ali H. Al-Hoorie; Jalal Al-Tamimi; Najd E. Alotaibi; Mohammed K. AlShakhori; Ruth M. Altmiller; Pablo Arantes; Angeliki Athanasopoulou; Melissa M. Baese-Berk; George Bailey; Cheman Baira A Sangma; Eleonora J. Beier; Gabriela M. Benavides; Nicole Benker; Emelia P. BensonMeyer; Nina R. Benway; Grant M. Berry; Liwen Bing; Christina Bjorndahl; Mariška Bolyanatz; Aaron Braver; Alicia M. Brown; Alicia M. Brown; Alejna Brugos; Erin M. Buchanan; Tanna Butlin; Andrés Buxó-Lugo; Coline Caillol; Francesco Cangemi; Christopher Carignan; Sita Carraturo; Tiphaine Caudrelier; Eleanor Chodroff; Michelle Cohn; Johanna Cronenberg; Olivier Crouzet; Erica L. Dagar; Charlotte Dawson; Carissa A. Diantoro; Marie Dokovova; Shiloh Drake; Fengting Du; Margaux Dubuis; Florent Duême; Matthew Durward; Ander Egurtzegi; Mahmoud M. Elsherif; Janina Esser; Emmanuel Ferragne; Fernanda Ferreira; Lauren K. Fink; Sara Finley; Kurtis Foster; Paul Foulkes; Rosa Franzke; Gabriel Frazer-McKee; Robert Fromont; Christina García; Jason Geller; Camille L. Grasso; Pia Greca; Martine Grice; Magdalena S. Grose-Hodge; Amelia J. Gully; Caitlin Halfacre; Ivy Hauser; Jen Hay; Robert Haywood; Sam Hellmuth; Allison I. Hilger; Nicole Holliday; Damar Hoogland; Yaqian Huang; Vincent Hughes; Ane Icardo Isasa; Zlatomira G. Ilchovska; Hae-Sung Jeon; Jacq Jones; Mágat N. Junges; Stephanie Kaefer; Constantijn Kaland; Matthew C. Kelley; Niamh E. Kelly; Thomas Kettig; Ghada Khattab; Ruud Koolen; Emiel Krahmer; Dorota Krajewska; Andreas Krug; Abhilasha A. Kumar; Anna Lander; Tomas O. Lentz; Wanyin Li; Yanyu Li; Maria Lialiou; Jr. Ronaldo M. Lima; Justin J. H. Lo; Julio Cesar Lopez Otero; Bradley Mackay; Bethany MacLeod; Mel Mallard; Carol-Ann Mary McConnellogue; George Moroz; Mridhula Murali; Ladislas Nalborczyk; Filip Nenadić; Jessica Nieder; Dušan Nikolić; Francisco G. S. Nogueira; Heather M. Offerman; Elisa Passoni; Maud Pélissier; Scott J. Perry; Alexandra M. Pfiffner; Michael Proctor; Ryan Rhodes; Nicole Rodríguez; Elizabeth Roepke; Jan P. Röer; Lucia Sbacco; Rebecca Scarborough; Felix Schaeffler; Erik Schleef; Dominic Schmitz; Alexander Shiryaev; Márton Sóskuthy; Malin Spaniol; Joseph A. Stanley; Alyssa Strickler; Alessandro Tavano; Fabian Tomaschek; Benjamin V. Tucker; Rory Turnbull; Kingsley O. Ugwuanyi; Iñigo Urrestarazu-Porta; Ruben van de Vijver; Kristin J. Van Engen; Emiel van Miltenburg; Bruce Xiao Wang; Natasha Warner; Simon Wehrle; Hans Westerbeek; Seth Wiener; Stephen Winters; Sidney G.-J. Wong; Anna Wood; Jane Wottawa; Chenzi Xu; Germán Zárate-Sández; Georgia Zellou; Cong Zhang; Jian Zhu; Timo B. Roettger

doi:10.1177/25152459231162567

Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses

Stefano Coretta, Joseph V. Casillas, Simon Roessig, Michael Franke, Byron Ahn, Ali H. Al-Hoorie, Jalal Al-Tamimi, Najd E. Alotaibi, Mohammed K. AlShakhori, Ruth M. Altmiller, Pablo Arantes, Angeliki Athanasopoulou, Melissa M. Baese-Berk, George Bailey, Cheman Baira A Sangma, Eleonora J. Beier, Gabriela M. Benavides, Nicole Benker, Emelia P. BensonMeyer, Nina R. BenwayGrant M. Berry, Liwen Bing, Christina Bjorndahl, Mariška Bolyanatz, Aaron Braver, Alicia M. Brown, Alicia M. Brown, Alejna Brugos, Erin M. Buchanan, Tanna Butlin, Andrés Buxó-Lugo, Coline Caillol, Francesco Cangemi, Christopher Carignan, Sita Carraturo, Tiphaine Caudrelier, Eleanor Chodroff, Michelle Cohn, Johanna Cronenberg, Olivier Crouzet, Erica L. Dagar, Charlotte Dawson, Carissa A. Diantoro, Marie Dokovova, Shiloh Drake, Fengting Du, Margaux Dubuis, Florent Duême, Matthew Durward, Ander Egurtzegi, Mahmoud M. Elsherif, Janina Esser, Emmanuel Ferragne, Fernanda Ferreira, Lauren K. Fink, Sara Finley, Kurtis Foster, Paul Foulkes, Rosa Franzke, Gabriel Frazer-McKee, Robert Fromont, Christina García, Jason Geller, Camille L. Grasso, Pia Greca, Martine Grice, Magdalena S. Grose-Hodge, Amelia J. Gully, Caitlin Halfacre, Ivy Hauser, Jen Hay, Robert Haywood, Sam Hellmuth, Allison I. Hilger, Nicole Holliday, Damar Hoogland, Yaqian Huang, Vincent Hughes, Ane Icardo Isasa, Zlatomira G. Ilchovska, Hae-Sung Jeon, Jacq Jones, Mágat N. Junges, Stephanie Kaefer, Constantijn Kaland, Matthew C. Kelley, Niamh E. Kelly, Thomas Kettig, Ghada Khattab, Ruud Koolen, Emiel Krahmer, Dorota Krajewska, Andreas Krug, Abhilasha A. Kumar, Anna Lander, Tomas O. Lentz, Wanyin Li, Yanyu Li, Maria Lialiou, Jr. Ronaldo M. Lima, Justin J. H. Lo, Julio Cesar Lopez Otero, Bradley Mackay, Bethany MacLeod, Mel Mallard, Carol-Ann Mary McConnellogue, George Moroz, Mridhula Murali, Ladislas Nalborczyk, Filip Nenadić, Jessica Nieder, Dušan Nikolić, Francisco G. S. Nogueira, Heather M. Offerman, Elisa Passoni, Maud Pélissier, Scott J. Perry, Alexandra M. Pfiffner, Michael Proctor, Ryan Rhodes, Nicole Rodríguez, Elizabeth Roepke, Jan P. Röer, Lucia Sbacco, Rebecca Scarborough, Felix Schaeffler, Erik Schleef, Dominic Schmitz, Alexander Shiryaev, Márton Sóskuthy, Malin Spaniol, Joseph A. Stanley, Alyssa Strickler, Alessandro Tavano, Fabian Tomaschek, Benjamin V. Tucker, Rory Turnbull, Kingsley O. Ugwuanyi, Iñigo Urrestarazu-Porta, Ruben van de Vijver, Kristin J. Van Engen, Emiel van Miltenburg, Bruce Xiao Wang, Natasha Warner, Simon Wehrle, Hans Westerbeek, Seth Wiener, Stephen Winters, Sidney G.-J. Wong, Anna Wood, Jane Wottawa, Chenzi Xu, Germán Zárate-Sández, Georgia Zellou, Cong Zhang, Jian Zhu, Timo B. Roettger

Language and Linguistic Science

Research output: Contribution to journal › Article › peer-review

Abstract

Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions.

Original language	English
Pages (from-to)	25152459231162567
Number of pages	1
Journal	Advances in Methods and Practices in Psychological Sciences
Volume	6
Issue number	3
DOIs	https://doi.org/10.1177/25152459231162567
Publication status	Published - 2023

Bibliographical note

This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details

Access to Document

10.1177/25152459231162567

48843699_File000000_1202459385
This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details
Accepted author manuscript, 3.16 MB

https://doi.org/10.1177/25152459231162567

Cite this

Coretta, S., Casillas, J. V., Roessig, S., Franke, M., Ahn, B., Al-Hoorie, A. H., Al-Tamimi, J., Alotaibi, N. E., AlShakhori, M. K., Altmiller, R. M., Arantes, P., Athanasopoulou, A., Baese-Berk, M. M., Bailey, G., Sangma, C. B. A., Beier, E. J., Benavides, G. M., Benker, N., BensonMeyer, E. P., ... Roettger, T. B. (2023). Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses. Advances in Methods and Practices in Psychological Sciences, 6(3), 25152459231162567. https://doi.org/10.1177/25152459231162567

@article{0defb9e4a1d7497daf4e5977fc75d45f,

title = "Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses",

abstract = "Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts{\textquoteright} prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions.",

author = "Stefano Coretta and Casillas, {Joseph V.} and Simon Roessig and Michael Franke and Byron Ahn and Al-Hoorie, {Ali H.} and Jalal Al-Tamimi and Alotaibi, {Najd E.} and AlShakhori, {Mohammed K.} and Altmiller, {Ruth M.} and Pablo Arantes and Angeliki Athanasopoulou and Baese-Berk, {Melissa M.} and George Bailey and Sangma, {Cheman Baira A} and Beier, {Eleonora J.} and Benavides, {Gabriela M.} and Nicole Benker and BensonMeyer, {Emelia P.} and Benway, {Nina R.} and Berry, {Grant M.} and Liwen Bing and Christina Bjorndahl and Mari{\v s}ka Bolyanatz and Aaron Braver and Brown, {Alicia M.} and Brown, {Alicia M.} and Alejna Brugos and Buchanan, {Erin M.} and Tanna Butlin and Andr{\'e}s Bux{\'o}-Lugo and Coline Caillol and Francesco Cangemi and Christopher Carignan and Sita Carraturo and Tiphaine Caudrelier and Eleanor Chodroff and Michelle Cohn and Johanna Cronenberg and Olivier Crouzet and Dagar, {Erica L.} and Charlotte Dawson and Diantoro, {Carissa A.} and Marie Dokovova and Shiloh Drake and Fengting Du and Margaux Dubuis and Florent Du{\^e}me and Matthew Durward and Ander Egurtzegi and Elsherif, {Mahmoud M.} and Janina Esser and Emmanuel Ferragne and Fernanda Ferreira and Fink, {Lauren K.} and Sara Finley and Kurtis Foster and Paul Foulkes and Rosa Franzke and Gabriel Frazer-McKee and Robert Fromont and Christina Garc{\'i}a and Jason Geller and Grasso, {Camille L.} and Pia Greca and Martine Grice and Grose-Hodge, {Magdalena S.} and Gully, {Amelia J.} and Caitlin Halfacre and Ivy Hauser and Jen Hay and Robert Haywood and Sam Hellmuth and Hilger, {Allison I.} and Nicole Holliday and Damar Hoogland and Yaqian Huang and Vincent Hughes and Isasa, {Ane Icardo} and Ilchovska, {Zlatomira G.} and Hae-Sung Jeon and Jacq Jones and Junges, {M{\'a}gat N.} and Stephanie Kaefer and Constantijn Kaland and Kelley, {Matthew C.} and Kelly, {Niamh E.} and Thomas Kettig and Ghada Khattab and Ruud Koolen and Emiel Krahmer and Dorota Krajewska and Andreas Krug and Kumar, {Abhilasha A.} and Anna Lander and Lentz, {Tomas O.} and Wanyin Li and Yanyu Li and Maria Lialiou and {Ronaldo M. Lima}, Jr. and Lo, {Justin J. H.} and Otero, {Julio Cesar Lopez} and Bradley Mackay and Bethany MacLeod and Mel Mallard and McConnellogue, {Carol-Ann Mary} and George Moroz and Mridhula Murali and Ladislas Nalborczyk and Filip Nenadi{\'c} and Jessica Nieder and Du{\v s}an Nikoli{\'c} and Nogueira, {Francisco G. S.} and Offerman, {Heather M.} and Elisa Passoni and Maud P{\'e}lissier and Perry, {Scott J.} and Pfiffner, {Alexandra M.} and Michael Proctor and Ryan Rhodes and Nicole Rodr{\'i}guez and Elizabeth Roepke and R{\"o}er, {Jan P.} and Lucia Sbacco and Rebecca Scarborough and Felix Schaeffler and Erik Schleef and Dominic Schmitz and Alexander Shiryaev and M{\'a}rton S{\'o}skuthy and Malin Spaniol and Stanley, {Joseph A.} and Alyssa Strickler and Alessandro Tavano and Fabian Tomaschek and Tucker, {Benjamin V.} and Rory Turnbull and Ugwuanyi, {Kingsley O.} and I{\~n}igo Urrestarazu-Porta and Vijver, {Ruben van de} and Engen, {Kristin J. Van} and Miltenburg, {Emiel van} and Wang, {Bruce Xiao} and Natasha Warner and Simon Wehrle and Hans Westerbeek and Seth Wiener and Stephen Winters and Wong, {Sidney G.-J.} and Anna Wood and Jane Wottawa and Chenzi Xu and Germ{\'a}n Z{\'a}rate-S{\'a}ndez and Georgia Zellou and Cong Zhang and Jian Zhu and Roettger, {Timo B.}",

note = "This is an author-produced version of the published paper. Uploaded in accordance with the publisher{\textquoteright}s self-archiving policy. Further copying may not be permitted; contact the publisher for details ",

year = "2023",

doi = "10.1177/25152459231162567",

language = "English",

volume = "6",

pages = "25152459231162567",

journal = "Advances in Methods and Practices in Psychological Sciences",

issn = "2515-2467",

publisher = "SAGE Publications Inc.",

number = "3",

}

Coretta, S, Casillas, JV, Roessig, S, Franke, M, Ahn, B, Al-Hoorie, AH, Al-Tamimi, J, Alotaibi, NE, AlShakhori, MK, Altmiller, RM, Arantes, P, Athanasopoulou, A, Baese-Berk, MM, Bailey, G, Sangma, CBA, Beier, EJ, Benavides, GM, Benker, N, BensonMeyer, EP, Benway, NR, Berry, GM, Bing, L, Bjorndahl, C, Bolyanatz, M, Braver, A, Brown, AM, Brown, AM, Brugos, A, Buchanan, EM, Butlin, T, Buxó-Lugo, A, Caillol, C, Cangemi, F, Carignan, C, Carraturo, S, Caudrelier, T, Chodroff, E, Cohn, M, Cronenberg, J, Crouzet, O, Dagar, EL, Dawson, C, Diantoro, CA, Dokovova, M, Drake, S, Du, F, Dubuis, M, Duême, F, Durward, M, Egurtzegi, A, Elsherif, MM, Esser, J, Ferragne, E, Ferreira, F, Fink, LK, Finley, S, Foster, K, Foulkes, P, Franzke, R, Frazer-McKee, G, Fromont, R, García, C, Geller, J, Grasso, CL, Greca, P, Grice, M, Grose-Hodge, MS, Gully, AJ, Halfacre, C, Hauser, I, Hay, J, Haywood, R, Hellmuth, S, Hilger, AI, Holliday, N, Hoogland, D, Huang, Y, Hughes, V, Isasa, AI, Ilchovska, ZG, Jeon, H-S, Jones, J, Junges, MN, Kaefer, S, Kaland, C, Kelley, MC, Kelly, NE, Kettig, T, Khattab, G, Koolen, R, Krahmer, E, Krajewska, D, Krug, A, Kumar, AA, Lander, A, Lentz, TO, Li, W, Li, Y, Lialiou, M, Ronaldo M. Lima, J, Lo, JJH, Otero, JCL, Mackay, B, MacLeod, B, Mallard, M, McConnellogue, C-AM, Moroz, G, Murali, M, Nalborczyk, L, Nenadić, F, Nieder, J, Nikolić, D, Nogueira, FGS, Offerman, HM, Passoni, E, Pélissier, M, Perry, SJ, Pfiffner, AM, Proctor, M, Rhodes, R, Rodríguez, N, Roepke, E, Röer, JP, Sbacco, L, Scarborough, R, Schaeffler, F, Schleef, E, Schmitz, D, Shiryaev, A, Sóskuthy, M, Spaniol, M, Stanley, JA, Strickler, A, Tavano, A, Tomaschek, F, Tucker, BV, Turnbull, R, Ugwuanyi, KO, Urrestarazu-Porta, I, Vijver, RVD, Engen, KJV, Miltenburg, EV, Wang, BX, Warner, N, Wehrle, S, Westerbeek, H, Wiener, S, Winters, S, Wong, SG-J, Wood, A, Wottawa, J, Xu, C, Zárate-Sández, G, Zellou, G, Zhang, C, Zhu, J & Roettger, TB 2023, 'Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses', Advances in Methods and Practices in Psychological Sciences, vol. 6, no. 3, pp. 25152459231162567. https://doi.org/10.1177/25152459231162567

TY - JOUR

T1 - Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses

AU - Coretta, Stefano

AU - Casillas, Joseph V.

AU - Roessig, Simon

AU - Franke, Michael

AU - Ahn, Byron

AU - Al-Hoorie, Ali H.

AU - Al-Tamimi, Jalal

AU - Alotaibi, Najd E.

AU - AlShakhori, Mohammed K.

AU - Altmiller, Ruth M.

AU - Arantes, Pablo

AU - Athanasopoulou, Angeliki

AU - Baese-Berk, Melissa M.

AU - Bailey, George

AU - Sangma, Cheman Baira A

AU - Beier, Eleonora J.

AU - Benavides, Gabriela M.

AU - Benker, Nicole

AU - BensonMeyer, Emelia P.

AU - Benway, Nina R.

AU - Berry, Grant M.

AU - Bing, Liwen

AU - Bjorndahl, Christina

AU - Bolyanatz, Mariška

AU - Braver, Aaron

AU - Brown, Alicia M.

AU - Brugos, Alejna

AU - Buchanan, Erin M.

AU - Butlin, Tanna

AU - Buxó-Lugo, Andrés

AU - Caillol, Coline

AU - Cangemi, Francesco

AU - Carignan, Christopher

AU - Carraturo, Sita

AU - Caudrelier, Tiphaine

AU - Chodroff, Eleanor

AU - Cohn, Michelle

AU - Cronenberg, Johanna

AU - Crouzet, Olivier

AU - Dagar, Erica L.

AU - Dawson, Charlotte

AU - Diantoro, Carissa A.

AU - Dokovova, Marie

AU - Drake, Shiloh

AU - Du, Fengting

AU - Dubuis, Margaux

AU - Duême, Florent

AU - Durward, Matthew

AU - Egurtzegi, Ander

AU - Elsherif, Mahmoud M.

AU - Esser, Janina

AU - Ferragne, Emmanuel

AU - Ferreira, Fernanda

AU - Fink, Lauren K.

AU - Finley, Sara

AU - Foster, Kurtis

AU - Foulkes, Paul

AU - Franzke, Rosa

AU - Frazer-McKee, Gabriel

AU - Fromont, Robert

AU - García, Christina

AU - Geller, Jason

AU - Grasso, Camille L.

AU - Greca, Pia

AU - Grice, Martine

AU - Grose-Hodge, Magdalena S.

AU - Gully, Amelia J.

AU - Halfacre, Caitlin

AU - Hauser, Ivy

AU - Hay, Jen

AU - Haywood, Robert

AU - Hellmuth, Sam

AU - Hilger, Allison I.

AU - Holliday, Nicole

AU - Hoogland, Damar

AU - Huang, Yaqian

AU - Hughes, Vincent

AU - Isasa, Ane Icardo

AU - Ilchovska, Zlatomira G.

AU - Jeon, Hae-Sung

AU - Jones, Jacq

AU - Junges, Mágat N.

AU - Kaefer, Stephanie

AU - Kaland, Constantijn

AU - Kelley, Matthew C.

AU - Kelly, Niamh E.

AU - Kettig, Thomas

AU - Khattab, Ghada

AU - Koolen, Ruud

AU - Krahmer, Emiel

AU - Krajewska, Dorota

AU - Krug, Andreas

AU - Kumar, Abhilasha A.

AU - Lander, Anna

AU - Lentz, Tomas O.

AU - Li, Wanyin

AU - Li, Yanyu

AU - Lialiou, Maria

AU - Ronaldo M. Lima, Jr.

AU - Lo, Justin J. H.

AU - Otero, Julio Cesar Lopez

AU - Mackay, Bradley

AU - MacLeod, Bethany

AU - Mallard, Mel

AU - McConnellogue, Carol-Ann Mary

AU - Moroz, George

AU - Murali, Mridhula

AU - Nalborczyk, Ladislas

AU - Nenadić, Filip

AU - Nieder, Jessica

AU - Nikolić, Dušan

AU - Nogueira, Francisco G. S.

AU - Offerman, Heather M.

AU - Passoni, Elisa

AU - Pélissier, Maud

AU - Perry, Scott J.

AU - Pfiffner, Alexandra M.

AU - Proctor, Michael

AU - Rhodes, Ryan

AU - Rodríguez, Nicole

AU - Roepke, Elizabeth

AU - Röer, Jan P.

AU - Sbacco, Lucia

AU - Scarborough, Rebecca

AU - Schaeffler, Felix

AU - Schleef, Erik

AU - Schmitz, Dominic

AU - Shiryaev, Alexander

AU - Sóskuthy, Márton

AU - Spaniol, Malin

AU - Stanley, Joseph A.

AU - Strickler, Alyssa

AU - Tavano, Alessandro

AU - Tomaschek, Fabian

AU - Tucker, Benjamin V.

AU - Turnbull, Rory

AU - Ugwuanyi, Kingsley O.

AU - Urrestarazu-Porta, Iñigo

AU - Vijver, Ruben van de

AU - Engen, Kristin J. Van

AU - Miltenburg, Emiel van

AU - Wang, Bruce Xiao

AU - Warner, Natasha

AU - Wehrle, Simon

AU - Westerbeek, Hans

AU - Wiener, Seth

AU - Winters, Stephen

AU - Wong, Sidney G.-J.

AU - Wood, Anna

AU - Wottawa, Jane

AU - Xu, Chenzi

AU - Zárate-Sández, Germán

AU - Zellou, Georgia

AU - Zhang, Cong

AU - Zhu, Jian

AU - Roettger, Timo B.

N1 - This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details

PY - 2023

Y1 - 2023

N2 - Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions.

AB - Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions.

U2 - 10.1177/25152459231162567

DO - 10.1177/25152459231162567

M3 - Article

SN - 2515-2467

VL - 6

SP - 25152459231162567

JO - Advances in Methods and Practices in Psychological Sciences

JF - Advances in Methods and Practices in Psychological Sciences

IS - 3

ER -