Measuring Policymakers' Preferences for Evidence Quality Across Contexts

Quantitative impact evaluations –- studies which measure the effects of policy –- play an important role in government decision-making. However, the quality of the information contained within impact evaluations varies according to properties of research design, including factors that are relevant to internal and external validity, as well as other considerations such as sample size and representativeness, and levels of statistical uncertainty. As evidence is typically aggregated from multiple studies which differ both in research design and estimates of impact, how policymakers value these different dimensions of evidence quality is consequential for the decisions that they make. In this paper, we report the results of a forced-choice survey experiment which we use to measure policymakers' preferences over different dimensions of evidence quality. Our survey is fielded to decision-makers from across UK government, including social researchers, policy and spending officials, and Members of Parliament. Our findings will be relevant to a growing literature on the value that politicians and policymakers place on different types of policy-relevant evidence.