A New Tool to Detect Low Survey Response Accuracy

The quality of survey data is quintessential for the validity and reliability of social science inferences. This quality depends to no small degree on respondents giving accurate answers. Our ability to detect low accuracy answers is however restricted by the fact that the mental steps of the process of survey response are difficult to observe empirically. To infer the degree of effort the respondents put into the survey, one common technique focuses on the response variation to question batteries (in technical parlance, the amount of \emph{non-differentiation}). This variation is captured differently by different non-differentiation indices. In a preliminary step of this project we have shown, through formal analysis and empirical analysis of ANES data, that several of the currently used non-differentiation indices yield unreliable information about the accuracy of survey responses. This paper proposes a new survey data quality indicator based on the variation in responses. The indicator, which we name the Coherence Index, improves on other current indices by a more precise operationalization of  low survey response accuracy and by more rigorous testing for such response behavior by taking into account battery-specific response strategies.