The COVID-19 pandemic presents a major source of fear, stress, and anxiety. This study aims to incorporate cultural differences into the debate on COVID-19 consequences for mental health. We tested the hypothesis that older adults experienced a mental burden of the pandemic in general, and of physical distancing measures in particular, that was greater in cultures of close physical contact. We defined mental burden as a self-assessed increase in loneliness, anxiety, and depression compared to before the pandemic. We used data from the SHARE Corona survey 2020 (wave 8) for 14 European countries (over 30,000 individuals) covering the population aged 50+, supplemented with data on country-specific preferred interpersonal distance and pandemic sanitary measures. Our results showed that in cultures of close physical contact the increase in loneliness was more common. The longer the stay-at-home order, the greater was the share of older adults who experienced an increase in loneliness, anxiety, and depression, and this increase was greater in cultures of close physical contact. Mental burden depended on an interplay of culturally preferred distance and the distance recommended by sanitary measures. Our results suggest that culture affects the mental burden of extreme events as the COVID-19 pandemic and its policy measures.