Abstract
Collection and analysis of self-reported information on an ordered Likert
scale is ubiquitous across the social sciences. Inference from such analyses are valid where the responses scale employed means the same thing
to all individuals. That is, if there is no differential item functioning (DIF)
present in the data. A priori this is unlikely to hold across all individuals
and cohorts in any sample of data. For this reason, anchoring vignettes
have been proposed as a way to correct for such DIF when individuals
self-assess their health (or well-being, or satisfaction levels, or disability
levels, and so on) on an ordered categorical scale. Using an example of
self-assessed pain, we illustrate the use of vignettes to adjust for DIF using the Compound Hierarchical Ordered Probit Model (CHOP IT). The
validity of this approach relies on the two underlying assumptions of response consistency (RC) and vignette equivalence (V E). Using a minor
amendment to the specification of the standard CHOP IT model, we develop easy-to-implement score tests of the null hypothesis of RC and V E
both separately and jointly. Monte Carlo simulations show that the tests
have good size and power properties in finite samples. We illustrate the
use of the test by applying it to our empirical example. The test should aid
more robust analyses of self-reported survey outcomes collected alongside anchoring vignettes.
scale is ubiquitous across the social sciences. Inference from such analyses are valid where the responses scale employed means the same thing
to all individuals. That is, if there is no differential item functioning (DIF)
present in the data. A priori this is unlikely to hold across all individuals
and cohorts in any sample of data. For this reason, anchoring vignettes
have been proposed as a way to correct for such DIF when individuals
self-assess their health (or well-being, or satisfaction levels, or disability
levels, and so on) on an ordered categorical scale. Using an example of
self-assessed pain, we illustrate the use of vignettes to adjust for DIF using the Compound Hierarchical Ordered Probit Model (CHOP IT). The
validity of this approach relies on the two underlying assumptions of response consistency (RC) and vignette equivalence (V E). Using a minor
amendment to the specification of the standard CHOP IT model, we develop easy-to-implement score tests of the null hypothesis of RC and V E
both separately and jointly. Monte Carlo simulations show that the tests
have good size and power properties in finite samples. We illustrate the
use of the test by applying it to our empirical example. The test should aid
more robust analyses of self-reported survey outcomes collected alongside anchoring vignettes.
| Original language | English |
|---|---|
| Number of pages | 46 |
| Journal | Journal of the Royal Statistical Society: Series A (Statistics in Society) |
| Early online date | 12 Nov 2020 |
| DOIs | |
| Publication status | E-pub ahead of print - 12 Nov 2020 |