Public services: Are composite measures a robust reflection of performance in the public sector

TY - BOOK

T1 - Public services: Are composite measures a robust reflection of performance in the public sector

AU - Jacobs, R.

AU - Goddard, M.

AU - Smith, P.C.

N1 - CHE RP16. © 2006 the authors. PLEASE DO NOT QUOTE WITHOUT THE AUTHOR'S PERMISSION. The full text of this report can be viewed free of charge from the Centre for Health Economics web site at: http://www.york.ac.uk/inst/che/pdf/rp16.pdf

PY - 2006/6

Y1 - 2006/6

N2 - A composite indicator is an aggregated index comprising individual performance indicators. Composite indicators integrate a large amount of information in a format that is easily understood and are therefore a valuable tool for conveying a summary assessment of performance in priority areas. This research investigates the degree to which composite measures are an appropriate metric for evaluating performance in the public sector. Do they reflect accurately the performance of organisations? To what degree are they influenced by the uncertainty surrounding underlying indicators on which they are based? Are they robust and stable over time? The construction of composite measures creates specific methodological challenges that make such questions especially pertinent. We address these through a series of quantitative analyses of panel data relating to healthcare (Star ratings of NHS acute Trusts) and local government (Comprehensive Performance Assessment (CPA) ratings of authorities) in England where composites have been widely used. The creation of a composite comprises a number of important steps, each of which requires careful judgement. These include the specification of the choice of indicators, the transformation of measured performance on individual indicators, the specification of a set of weights on individual indicators, and combining the indicators using aggregation methods or decision rules. We use Monte Carlo simulations to examine the robustness of performance judgements to these different technical choices. We show the extent to which composites provide stable performance rankings of organisations over time and assess whether variations are due to genuine performance improvement or merely the result of random statistical variation. The analysis suggests that the judgements that have to be made in the construction of the composite can have a significant impact on the resulting score. Technical and analytical issues in the design of composite indicators have important policy implications. We highlight the issues which need to be considered in the construction of robust composite indicators so that they can be designed in ways which will minimise the potential for producing misleading performance information which may fail to deliver the expected improvements or even induce unwanted side-effects.

AB - A composite indicator is an aggregated index comprising individual performance indicators. Composite indicators integrate a large amount of information in a format that is easily understood and are therefore a valuable tool for conveying a summary assessment of performance in priority areas. This research investigates the degree to which composite measures are an appropriate metric for evaluating performance in the public sector. Do they reflect accurately the performance of organisations? To what degree are they influenced by the uncertainty surrounding underlying indicators on which they are based? Are they robust and stable over time? The construction of composite measures creates specific methodological challenges that make such questions especially pertinent. We address these through a series of quantitative analyses of panel data relating to healthcare (Star ratings of NHS acute Trusts) and local government (Comprehensive Performance Assessment (CPA) ratings of authorities) in England where composites have been widely used. The creation of a composite comprises a number of important steps, each of which requires careful judgement. These include the specification of the choice of indicators, the transformation of measured performance on individual indicators, the specification of a set of weights on individual indicators, and combining the indicators using aggregation methods or decision rules. We use Monte Carlo simulations to examine the robustness of performance judgements to these different technical choices. We show the extent to which composites provide stable performance rankings of organisations over time and assess whether variations are due to genuine performance improvement or merely the result of random statistical variation. The analysis suggests that the judgements that have to be made in the construction of the composite can have a significant impact on the resulting score. Technical and analytical issues in the design of composite indicators have important policy implications. We highlight the issues which need to be considered in the construction of robust composite indicators so that they can be designed in ways which will minimise the potential for producing misleading performance information which may fail to deliver the expected improvements or even induce unwanted side-effects.

KW - performance measurement

KW - performance indicators

KW - composite indicators

M3 - Commissioned report

T3 - CHE Research Paper

BT - Public services: Are composite measures a robust reflection of performance in the public sector

PB - Centre for Health Economics, University of York

CY - York, UK

ER -