All time references are in CEST
Teaching Survey Methodology – Challenges and New Approaches |
|
| Session Organiser | Dr Katrin Drasch (FAU Erlangen-Nürnberg) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Many higher education curricula involve survey data collection by students, aiming to teach a complete empirical research circle from formulating a research question to finding a preliminary empirical answer. This approach is often justified because it constitutes a crucial skill for future careers in both academic and non-academic environments. However, over the past two decades, data collection has increasingly shifted from written or oral/telephone interviews to more complex methods like online surveys, longitudinal studies, and alternative data sources, making it increasingly challenging to teach survey methodology in its entirety. Current curricula often face two major issues: first, the limited time available for empirical research projects, typically a maximum of two academic terms, often necessitates shortcuts, such as skipping cognitive pretesting, to complete data analysis on time. Second, there is often little financial support for material expenses (e.g., postal charges, printing questionnaires) or technical resources (e.g., access to sampling frames). This can lead to the misinterpretation of results, such as treating non-probability snowball sampling as representative. As a result, self-collected data often suffer from quality issues and are thus not suitable for scientific publications. This session aims to discuss these challenges and explore new approaches to collecting scientifically valuable data in a teaching environment. For instance, experimental designs (e.g., factorial surveys or choice experiments) that do not require large or representative samples could be considered. Additionally, incorporating AI tools into survey methodology instruction, such as using them to generate and pretest questionnaires, could provide new opportunities. Alternative data sources (e.g., social media data or anonymized university records) and collaborations with external (non-academic) partners, along with the challenges they pose (e.g., data protection and exchange), could also be explored. These and other alternative and/or innovative topics are welcome to this session.
Keywords: higher education, teaching, data collection
Dr Ranjit K. Singh (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Understanding statistical methods in survey research is challenging. It is fundamentally challenging because many relevant pieces of information are either unobserved (such as population parameters) or even fundamentally unobservable (such as latent construct expressions). However, most methods construct their arguments for validity around causal models that depend on these unobserved or unobservable variables. Think of probability samples, psychometric measurement models, or different types of missing data. Often, these methods are taught in the social sciences with conceptual metaphors, by listing assumptions, and, optionally, with math. However, when faced with empirical data, students often struggle to bridge the gap between the abstract models and the concrete research project.
In my talk, I will make the argument that even the simplest simulations can help students bridge that gap. By tending to the small worlds of their simulations, students can apply sanity checks to their understanding and can explore the implications of methodological models in quick, interactive iterations. To illustrate this, I will present examples from the two branches of the total survey error framework. From the representation branch, a simulation that shows that random experiments in non-probability samples can still be misleading if we fail to measure all relevant moderators. From the measurement branch, I will simulate psychometric response models and their implications for measurement quality and comparability. All my simulation examples use R, but they are just as easily done in Julia or Python. I will also touch upon pragmatic issues for lecturers, such as letting students build simulations from scratch or providing a layer of abstraction by building custom helper functions.
Dr Margherita Silan (Department of Statistical Sciences, University of Padova) - Presenting Author
Dr Manuela Scioni (Department of Statistical Sciences, University of Padova)
The ‘Issues and Methods for Population and Society’ course in the Master's degree program in Statistical Sciences at the University of Padua includes a 20-hour module on social media surveys. This module aims to develop students' practical skills in conducting social media-based surveys and to enhance their critical understanding of survey methodology, including limitations and strengths. Moreover, students evaluate recruitment challenges and mitigation strategies and learn advanced techniques for mitigating selection bias in online surveys.
The Course is structured around a practical problem in collaboration with ERION, an Italian company specializing in the disposal of waste from electrical and electronic equipment (WEEE).
The module begins with the ERION research team, who introduces students to the current state of WEEE management. The students then review key concepts in survey methodology and questionnaire design, applying them to a WEEE-related survey.
Then, the course is structured around 4 experiences of Team Based Learning where students study a scientific paper on the topic and through practical activities learn:
- Pros and cons of social media recruitment.
- Technical aspects of the recruitment of respondents for online surveys using Facebook and Instagram, finally implementing their own survey.
- Basic post-stratification techniques to reduce selection bias of the sample (for instance, raking) with application on the collected data.
- More advanced post-stratification techniques (for instance quasi-randomization), comparing results with the more basic techniques.
Students run their own survey on the WEEE topic, guided step by step by the teacher, starting from the definition of the problem to the data collection, evaluating the issues encountered by the recruitment method and techniques to attempt to overcome those issues. The use of social media for respondent recruitment enables cost-effective and rapid surveys, ideal for practical student activities, including small-scale experiments.
Dr Pierre Walthéry (UK Data Service, University of Manchester ) - Presenting Author
Dr Jennifer Buckley (UK Data Service, University of Manchester )
This presentation introduces an experimental approach to teaching foundational statistical inference techniques, developed by the UK Data Service (UKDS). The aim is to bridge the gap between oversimplified introductory content and the predominantly technical material that currently dominates the teaching landscape.
Teaching statistical inference with large-scale social surveys has traditionally faced a dilemma: either separating data analysis and statistical concepts from real-world survey estimation or delivering a technical presentation of the complex statistical methods used to account for variance estimation using weights and survey design variables. This divide often results in a separation between undergraduate and postgraduate courses, raising the risk that some graduating students are underprepared for the challenges they will face in their career.
To address this gap, the UKDS has revisited some of its teaching materials to produce content that balances the need for practical, statistical inference skills with an understanding of the complex challenges users face when working with real-world survey data. The approach is built on three key pillars: clear, plain-language explanations of concepts; concrete, real-world research scenarios and the analytical dilemmas they present; and hands-on practical exercises using R and Stata.
This approach has been successfully implemented in one pilot workshop, with further pilots scheduled. It aligns with the UKDS’ broader ambition to foster communities of practice and aims at enhancing the quality of statistical training in this field.
Dr Basha Vicari (Institute for Employment Research & Statistical Office of the City of Erlangen)
Dr Katrin Drasch (Chair of Methods for Social Empirical Research (FAU Erlangen-Nürnberg)) - Presenting Author
Dr Dominik Kremer (Department Digital Humanities and Social Studies (FAU Erlangen-Nürnberg))
This presentation discusses the design of a teaching research project on suburban mobility (automobility including the parking situation, (e-)bicycle mobility, public transport, and pedestrian mobility) in a large German city. The project is embedded in a collaboration between i) the municipal statistics office which will provide cost-free access to the population registry, ii) the Department of Digital Humanities and Social Science, and iii) the Department of Sociology of the local university. Due to time (teaching environment) and financial restrictions (no third-party funding), a parsimonious strategy had to be adopted which includes getting in touch with other local institutions interested in the study's results. Two subprojects are developed: the first project is embedded in a sociology course that aims at employing a representative sample of the local population registry enhanced with a convenience sample. This project develops a standardized online questionnaire and includes a multifactorial survey experiment. It uses AI for developing the questionnaire. The second project uses an innovative technique to collect different sets of geo-tagged lifeworld data such as photos, videos, audio, and voice recordings, collected in an Android app. Following a mixed-method paradigm, this experimental approach aims at revealing problems at certain central locations (e.g., city centre, the train station) in detail but will also allow for collecting the variability of impressions concerning local mobility at other places in general. In the presentation, we will show the preliminary results of the ongoing subprojects and discuss the major challenges of collecting such data in teaching environment.