Mannheim Research Colloquium on Survey Methods (MaRCS): Using Experimental Vignettes to Study how Survey Methods and Findings Affect the Public’s Evaluation of Public Opinion Polls: Considering A Dual Process Approach

Abstract:

Understanding how the public thinks about and understands the results of surveys an important part of understanding the role of surveys in policymaking and democracy more broadly. In this paper, we examine the results of three vignette experiments conducted as part of representative sample surveys in which the methodology and results of public opinion survey assessing support for specific public policy proposals was manipulated. Respondents’ beliefs about the accuracy of the survey described in the vignette experiment, and their beliefs about whether or not it should be considered by policymakers were measured. We used both manipulated (e.g., the methodological rigor of the survey described in the vignette) and measured (e.g., respondents’ opinions on the proposed policy about which public opinion was measured in the vignette) to test four different models that could be used to explain public evaluations of public opinion surveys: (1) the rational actor model, which suggests that people will evaluate more methodologically rigorous surveys more positively; (2) the science literacy model, which suggests that people high in science literacy will evaluate more methodologically rigorous surveys more positively than will people low in science literacy; (3) the motivated reasoning model, which suggests that people will evaluate surveys more positively when the survey results are consistent with their prior opinions than when they are inconsistent; and (4) a dual process model approach, which suggests that people will evaluate more methodologically rigorous surveys more positively only when they are both able and motivated to do so.  We found some support for the scientific literacy and motivated reasoning models, but these findings were qualified by an interaction between factors associated with respondent motivation, ability, and survey methodology rigor that strongly supports the dual process model perspective.

Please use this link to attend the colloquium via Zoom:

https://us02web.zoom.us/j/81804053598?pwd=WUVtaVErWklpSVBBV0N5eGo5eHFkUT09

MaRCS is a seminar series jointly organized by the Mannheim Centre for European Social Research (MZES), the University of Mannheim School of Social Sciences, and GESIS – Leibniz Institute for the Social Sciences.

You can sign-up to the MaRCS mailing list here https://lists.gesis.org/mailman/listinfo/marcs to receive invitations to the upcoming seminars.