Methods Bites

Blog of the MZES Social Science Data Lab

Transformer-based language models

2023-05-30 1 min read Video

Transformer-based models have recently gained much attention, especially with the release of ChatGPT. Since 2017, deep learning models based on the Transformer architecture have become an important research tool. Their development and application in various fields, including the social sciences, continue to expand. In this talk, we will examine the components that make up these language models and explore how to train state-of-the-art models with HuggingFace for your research. We will also discuss these models’ limitations and open challenges, including open-source availability, the growing need for resources, responsibility, and more.


Christopher Klamm is an interdisciplinary researcher at the University of Mannheim (Germany) at the Data and Web Science Group working at the intersection of Natural Language Processing and Computational Political Science.