While surveys are an effective instrument to measure privacy-related attitudes, current research also uses surveys to measure participants' privacy behavior. Research shows that self-reporting behavior can suffer from recall and social desirability bias, thus potentially leading to inaccurate inference in subsequent analyses. This can be detrimental to privacy research, which typically uses privacy behavior as its dependent variable. This study investigates the extent of misreporting in survey questions about fine-grained Instagram privacy behavior by comparing it to objective measures collected via data donation. We also explore to what extent the accuracy of self-reports depends on the response format (labeled vs. numbered scale). We will receive data donations from about 400 participants in a German probability-based online panel regarding over 20 distinct privacy behaviors on Instagram, including whether participants typically post content on which faces are visible, how they manage followers, and whether they share their personal contacts with Instagram. Participants first complete survey questions on these behaviors and we vary the response format within a split-sample experiment. Respondents are then instructed to download their Instagram usage data for the last three months and donate them to our research. We analyze correlation coefficients between behavioral self-reports and donated data to assess the accuracy of self-reports in general and for specific privacy behaviors. We contribute in three key ways: First, we inform the field of questionnaire design by offering insights into how to accurately inquire about specific online privacy behaviors, which is particularly interesting for researchers who may not utilize data donation methods. Second, we examine the accuracy of self-reported data on individual Instagram privacy behavior, helping researchers assess the validity of surveying such behavior. Third, we illustrate the potential of data donation to gather detailed, fine-grained data on individual privacy behaviors.
