Tools to Detect Fabricated Interviews

The project investigated the performance of different tools to detect fabricated survey interviews. Using a large-scale refugee survey in Germany with known fabricated interviews, we evaluated and compared different statistical methods and falsification indicators. In addition to the evaluation of known methods, the findings contributed to developing new and enhancing old quality control systems. Hence, the project was able to address the challenges practitioners face when deciding on an appropriate detection strategy and contributed to the broader discussion of “best practices” for detecting and preventing interviewer fraud in survey research.

The first wave of the IAB-BAMF-SOEP Survey of Refugees in Germany, which includes around 600 verified falsifications—fabricated by three interviewers—for person-level and household-level interviews, served as the data basis for the study. In total, this wave included 3,554 responding households and 4,816 respondents.

First, we tested various multivariate detection strategies, including cluster analysis, as well as a newly developed detection method termed “meta-indicator”. Using various accuracy measures, we assessed the performance of these detection tools. Second, we introduced some new falsification indicators. Third, we compared the explanatory power of single indicators and tested their assumed directional implications pointing to suspicious interviewer behaviour.

Consistent with the literature, the results indicate that the different multivariate detection methods utilizing various indicators are highly effective in identifying all three confirmed falsifiers: Most falsification indicators used are successful in measuring differences between falsifiers and honest interviewers, with some newly proposed falsification indicators outperforming some existing indicators. Furthermore, the different multivariate detection methods perform similarly well in detecting the falsifiers.