Abstract
Introduction
Policy evaluations often focus on ex-post estimation of causal effects on short-term surrogate outcomes. The value of such information is limited for decision-making as the failure to reflect policy-relevant outcomes and disregard for opportunity costs prohibits the assessment of value for money. Further, these evaluations do not always consider all relevant evidence, other courses of action and the decision uncertainty.
Methods
In this paper, we explore how policy evaluation could better meet the needs of decision-making. We begin by defining the evidence required to inform decision making. We then conduct a literature review of challenges in evaluating policies. Finally, we highlight potential methods available to help address these challenges.
Results
The evidence required to inform decision-making includes the impacts on the policy-relevant outcomes, the costs and associated opportunity costs, and the consequences of uncertainty. Challenges in evaluating health policies are described using 8 categories: i) valuation space, ii) comparators, iii) time of evaluation, iv) mechanisms of action, v) effects, vi) resources, constraints and opportunity costs, vii) fidelity, adaptation and level of implementation, and viii) generalisability and external validity. Methods from a broad set of disciplines are available to improve policy evaluation, relating to causal inference, decision-analytic modelling, theory of change, realist evaluation and structured expert elicitation.
Limitations
The targeted review may not identify all possible challenges and the methods covered are not exhaustive.
Conclusions
Evaluations should provide appropriate evidence to inform decision making. There are challenges in evaluating policies but methods from multiple disciplines are available to address these challenges.
Implications
Evaluators need to carefully consider the decision being informed, the necessary evidence to inform it and the appropriate methods.
Policy evaluations often focus on ex-post estimation of causal effects on short-term surrogate outcomes. The value of such information is limited for decision-making as the failure to reflect policy-relevant outcomes and disregard for opportunity costs prohibits the assessment of value for money. Further, these evaluations do not always consider all relevant evidence, other courses of action and the decision uncertainty.
Methods
In this paper, we explore how policy evaluation could better meet the needs of decision-making. We begin by defining the evidence required to inform decision making. We then conduct a literature review of challenges in evaluating policies. Finally, we highlight potential methods available to help address these challenges.
Results
The evidence required to inform decision-making includes the impacts on the policy-relevant outcomes, the costs and associated opportunity costs, and the consequences of uncertainty. Challenges in evaluating health policies are described using 8 categories: i) valuation space, ii) comparators, iii) time of evaluation, iv) mechanisms of action, v) effects, vi) resources, constraints and opportunity costs, vii) fidelity, adaptation and level of implementation, and viii) generalisability and external validity. Methods from a broad set of disciplines are available to improve policy evaluation, relating to causal inference, decision-analytic modelling, theory of change, realist evaluation and structured expert elicitation.
Limitations
The targeted review may not identify all possible challenges and the methods covered are not exhaustive.
Conclusions
Evaluations should provide appropriate evidence to inform decision making. There are challenges in evaluating policies but methods from multiple disciplines are available to address these challenges.
Implications
Evaluators need to carefully consider the decision being informed, the necessary evidence to inform it and the appropriate methods.
| Original language | English |
|---|---|
| Journal | Medical Decision Making |
| Early online date | 27 May 2021 |
| DOIs | |
| Publication status | E-pub ahead of print - 27 May 2021 |