Producing Questionnaire Translations: the Impact of Translator Background and Machine Translation on Translation Quality

Cross-cultural surveys rely on high quality questionnaire translations to allow valid substantive conclusions. Against this backdrop, multi-step, interdisciplinary translation methods (e.g., TRAPD, Harkness, 2003) have been developed to ensure comparability and quality in translations. Even though interdisciplinary collaboration (notably between professional translators and survey/substantive experts) has been considered to be vital, the reality in multilingual survey research often looks  different, with relying on researchers only, or on translators only (Efu Nkong 2024). In the DFG-funded project TransBack (Details), we aim to understand better the role of translator background on translation quality. Additionally, we look into the impact of machine translation, thereby acknowledging that machine translation is ‘here to stay’. In the first project phase, we conducted a survey among national coordinators of 10+ large-scale international surveys to enquire about their translation processes, the translation staff employed, their views on translation and translators, etc. (Efu Nkong 2024). In the second project phase, we conducted an experiment (English -> German) with 16 social scientists and 16 professional translators, who a) translated items from scratch and b) corrected machine translation output (deepl), where they deemed that this was necessary. In the third project phase, we used the experimental questionnaires, as well as the official German translations, and a ChatGPT version, to collect survey data (using the Bilendi panel). The survey data is the ultimate test as to what works how under real survey conditions, and what errors produced by the different groups mean. In this talk, I will focus on the project phases 2 and 3, presenting the experimental setting, first error coding results as well as first survey analyses.

Joint work with Ulrike Efu Nkong, Anke Radinger, and Chia-Jung Tsai

MaRCS is a seminar series jointly organized by the Mannheim Centre for European Social Research (MZES), the University of Mannheim School of Social Sciences, and GESIS – Leibniz Institute for the Social Sciences.