ROBITT: A tool for assessing the risk-of-bias in studies of temporal trends in ecology

TY - JOUR

T1 - ROBITT

T2 - A tool for assessing the risk-of-bias in studies of temporal trends in ecology

AU - Boyd, Robin J.

AU - Powney, Gary D.

AU - Burns, Fiona

AU - Danet, Alain

AU - Duchenne, François

AU - Grainger, Matthew J.

AU - Jarvis, Susan G.

AU - Martin, Gabrielle

AU - Nilsen, Erlend B.

AU - Porcher, Emmanuelle

AU - Stewart, Gavin B.

AU - Wilson, Oliver J.

AU - Pescott, Oliver L.

N1 - Funding Information: G.D.P. and S.G.J. were supported through Natural Environment Research Council (NERC) award number NE/V006878/1 as part of the DRUID (Drivers and Repercussions of UK Insect Declines) project. R.J.B. and O.L.P. were supported by the NERC award number NE/R016429/1 as part of the UK Status, Change and Projections of the Environment (UK-SCAPE) program delivering National Capability; O.L.P. also acknowledges the support of the British Council's Alliance Hubert Curien program, award number 515719745, and the National Plant Monitoring Scheme funded by the UK Joint Nature Conservation Committee (ref. A17-0291-1205). F.D. was funded by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement number 787638, granted to C.H. Graham). E.P. and G.M. acknowledge the support of the PHC Alliance award number 44779VJ. Publisher Copyright: © 2022 The Authors. Methods in Ecology and Evolution published by John Wiley & Sons Ltd on behalf of British Ecological Society.

PY - 2022/7/1

Y1 - 2022/7/1

N2 - Aggregated species occurrence and abundance data from disparate sources are increasingly accessible to ecologists for the analysis of temporal trends in biodiversity. However, sampling biases relevant to any given research question are often poorly explored and infrequently reported; this can undermine statistical inference. In other disciplines, it is common for researchers to complete ‘risk-of-bias’ assessments to expose and document the potential for biases to undermine conclusions. The huge growth in available data, and recent controversies surrounding their use to infer temporal trends, indicate that similar assessments are urgently needed in ecology. We introduce ROBITT, a structured tool for assessing the ‘Risk-Of-Bias In studies of Temporal Trends in ecology’. ROBITT has a similar format to its counterparts in other disciplines: it comprises signalling questions designed to elicit information on the potential for bias in key study domains. In answering these, users will define study inferential goal(s) and relevant statistical target populations. This information is used to assess potential sampling biases across domains relevant to the research question (e.g. geography, taxonomy, environment), and how these vary through time. If assessments indicate biases, then users must clearly describe them and/or explain what mitigating action will be taken. Everything that users need to complete a ROBITT assessment is provided: the tool, a guidance document and a worked example. Following other disciplines, the tool and guidance document were developed through a consensus-forming process across experts working in relevant areas of ecology and evidence synthesis. We propose that researchers should be strongly encouraged to include a ROBITT assessment when publishing studies of biodiversity trends, especially when using aggregated data. This will help researchers to structure their thinking, clearly acknowledge potential sampling issues, highlight where expert consultation is required and provide an opportunity to describe data checks that might go unreported. ROBITT will also enable reviewers, editors and readers to establish how well research conclusions are supported given a dataset combined with some analytical approach. In turn, it should strengthen evidence-based policy and practice, reduce differing interpretations of data and provide a clearer picture of the uncertainties associated with our understanding of reality.

AB - Aggregated species occurrence and abundance data from disparate sources are increasingly accessible to ecologists for the analysis of temporal trends in biodiversity. However, sampling biases relevant to any given research question are often poorly explored and infrequently reported; this can undermine statistical inference. In other disciplines, it is common for researchers to complete ‘risk-of-bias’ assessments to expose and document the potential for biases to undermine conclusions. The huge growth in available data, and recent controversies surrounding their use to infer temporal trends, indicate that similar assessments are urgently needed in ecology. We introduce ROBITT, a structured tool for assessing the ‘Risk-Of-Bias In studies of Temporal Trends in ecology’. ROBITT has a similar format to its counterparts in other disciplines: it comprises signalling questions designed to elicit information on the potential for bias in key study domains. In answering these, users will define study inferential goal(s) and relevant statistical target populations. This information is used to assess potential sampling biases across domains relevant to the research question (e.g. geography, taxonomy, environment), and how these vary through time. If assessments indicate biases, then users must clearly describe them and/or explain what mitigating action will be taken. Everything that users need to complete a ROBITT assessment is provided: the tool, a guidance document and a worked example. Following other disciplines, the tool and guidance document were developed through a consensus-forming process across experts working in relevant areas of ecology and evidence synthesis. We propose that researchers should be strongly encouraged to include a ROBITT assessment when publishing studies of biodiversity trends, especially when using aggregated data. This will help researchers to structure their thinking, clearly acknowledge potential sampling issues, highlight where expert consultation is required and provide an opportunity to describe data checks that might go unreported. ROBITT will also enable reviewers, editors and readers to establish how well research conclusions are supported given a dataset combined with some analytical approach. In turn, it should strengthen evidence-based policy and practice, reduce differing interpretations of data and provide a clearer picture of the uncertainties associated with our understanding of reality.

KW - essential biodiversity variables

KW - indicators

KW - insect declines

KW - risk-of-bias

KW - species occurrence data

KW - temporal trends

KW - uncertainty

UR - http://www.scopus.com/inward/record.url?scp=85127594115&partnerID=8YFLogxK

U2 - 10.1111/2041-210X.13857

DO - 10.1111/2041-210X.13857

M3 - Article

AN - SCOPUS:85127594115

SN - 2041-210X

VL - 13

SP - 1497

EP - 1507

JO - Methods in ecology and evolution

JF - Methods in ecology and evolution

IS - 7

ER -