Measuring Policy Positions through the Aggregation of Survey Responses

Expert surveys provide a common means for assessing parties’ policy positions on latent dimensions. These surveys often cover a wide variety of parties and issues, and it is unlikely that experts are able to assess all parties equally well across all issues. While the existing literature using expert surveys acknowledges this fact, insufficient attention has been paid to the variance in the quality of measurement across issues and parties. In this paper, we first discuss the nature of the measurement problem with respect to expert surveys and then propose methods borrowed from the organizational psychology and medical fields to assess the ability of experts to assess where parties stand on particular dimensions. While we apply our technique to one particular study, the Chapel Hill Expert Survey, the method can be applied to any expert survey. Finally, we propose a simple non-parametric bootstrapping procedure that allows researchers to assess the effects of expert survey measurement error in analyses that use them.