Representation side of TSE |
|
| Session Organisers | Dr Sabrina Mayer (German Center for Integration and Migration Research (DeZIM) & University of Duisburg-Essen) Dr Mariel McKone Leonard (German Center for Integration and Migration Research (DeZIM) ) |
| Time | Friday 9 July, 16:45 - 18:00 |
This panels draws together different papers dealing with consequences of biases in the sampling framework as well as the data collection process and advance our understanding of how to deal with survey biases. Papers either take a comperative or national perspective or focus on specific subgroups in the population.
Keywords: representativeness, web survey, biases, estimations
Ms Laura Scholaske (German Center for Integration and Migration Research (DeZIM-Institut)) - Presenting Author
Ms Lara Kronenbitter (German Center for Integration and Migration Research (DeZIM-Institut))
Ms Sabrina Mayer (German Center for Integration and Migration Research (DeZIM-Institut))
Conducting standardized surveys with population samples offers the chance to generalize the results for the whole population. However, the survey process becomes more prone for errors and biases, the smaller, the more vulnerable and the more hard-to-reach the population is. One example for such a group are unaccompanied minor refugees (UMR). Even though policy makers and civic administration have an on-going interest for knowing more about this group, previous studies only relied on convenience samples and qualitative interviews. However, to be able to make generalized statements about this group as well as evaluating their reactions to external changes and the on-going integration process, we would need randomized sampled survey data. But what challenges for data quality have to be faced when conducting standardized surveys based on population samples in this group? We draw here on two studies that we conducted with UMR in Germany, and systematize the different challenges, based on the Total Survey Error framework. Most of the challenges either occur on the unit nonresponse or the measurement level. Moreover, we discuss alternatives by using qualitative interviews in a mixed-mode approach to react to the different challenges which seems especially useful for overcoming problems on the measurement error level.
Mr Luke Taylor (Kantar) - Presenting Author
A major challenge when conducting political polling is deciding how to deal with respondents that do not express a firm voting preference. One approach commonly taken by pollsters is to exclude these individuals altogether from voting intention estimates. However, there is a risk this could introduce bias – if those that do not express a firm preference are systematically different from those that do provide a voting intention.
This paper outlines the methodology used in the UK by Kantar to estimate the voting intention of individuals that do not express a firm preference. First, a squeeze question is used – to get some information about the direction in which these respondents may be leaning. Second, KNN is used to impute the voting intention of those that have not stated a preference (at either the main voting intention question or at the squeeze question).
This approach is evaluated using data from the December 2019 UK General Election, in which the Conservative party achieved 44.7% of the vote in Great Britain, Labour 33.0% and the Lib Dems 11.8%. Kantar’s final pre-election poll forecast a result in Great Britain of 44% for the Conservatives, 32% for Labour and 13% for the Liberal Democrats (Mean Absolute Error across these three parties of 1.0% points). This proved to be one of the most accurate forecasts. If the final Kantar poll had excluded respondents that did not express a firm preference, the Mean Absolute Error across the three main parties would have been larger (1.5% points across the three parties).
Dr Sabrina Mayer (German Center for Integration and Migration Research (DeZIM) & University of Duisburg-Essen) - Presenting Author
Dr Jörg Dollmann (German Center for Integration and Migration Research (DeZIM) & Mannheim Center for European Social Research (MZES), University of Mannheim)
Dr Mariel McKone Leonard (German Center for Integration and Migration Research (DeZIM))
A multitude of different (often commercial) access panels provide researchers with new possibilities, e.g. to quickly survey appropriate samples in the context of external events and crises or to track long-term attitudinal developments. However, large segments of the population, such as individuals with a migration background (that migrated themselves to the host country or have at least one parent that was born abroad) or ethnic minorities, are usually underrepresented in these infrastructures.
As the number of people of ethnic minority- or immigrant-origins in most access panels is low, it is not possible to distinguish between different groups. This is problematic as we can suppose that daily life as well as racist and discriminatory experiences differ vastly between e.g. visible minorities on the one hand and people from Western European countries on the other hand. Underrepresentation of these groups also hinders the study of reactions of immigrant-origin groups towards specific external events, such as responses and threat perceptions following racist violence.
Surveying people of immigrant-origin or ethnic minorities is a specific field that deals with small, often hard-to-reach and/or hidden populations. Thus, developing an infrastructure that allows access to several of these groups would increase our understanding of the underpinnings of discriminatory experiences and the various dimensions of integration. However, not much is known about retention rates and survey behaviour these groups.
We intend to change this by setting up our own infrastructure at the German Centre for Integration and Migration Research. At the beginning of 2021, we started the setup process for the DeZIM.panel, a non-commercial online access panel that is supposed to be representative for several major groups of people of immigrant-origin as well as the native population in Germany. We focus currently on four different groups of people of immigrant-origin: people from Turkey, people from other countries that are predominantly Muslim, people from the former Soviet Union or its successor states, and people from other states with which Germany had recruitment agreements (e.g. Italy, Greece, Spain). We intend to survey 500 people per group. Later waves will focus recruitment on individuals from southeast/east Asia and additional predominantly Muslim countries.
We began with an offline recruitment wave based on a two-step sampling procedure that includes onomastic pre-classification of individuals for a push-to-web approach. We also conducted incentives experiments to test how different incentives affect retention and participation rates. After the recruitment wave, we will conduct bi-monthly online interviews for which we provide the data for secondary data analysis in our research data centre. Our paper will provide an overview of the general setup of the DeZIM.panel as well as show data for the initial wave. We intend to share best practices and discuss possible biases within the panel.
Mr Hafsteinn Einarsson (University of Manchester) - Presenting Author
Professor Natalie Shlomo (University of Manchester)
Dr Alexandru Cernat (University of Manchester)
Collecting representative data is essential to ensure that valid inferences can be drawn from social surveys. A challenging survey climate, where response rates are declining and costs are increasing over time, can affect survey organisations abilities to collect high quality survey data. Therefore, measures of representativeness such as R-indicators, have been devised to estimate whether the responding sample reflects the full sample on key characteristics. However, comparing representativeness across surveys is difficult, as accurate comparisons must be made using the same model, which can prove difficult if key variables are missing from one of the surveys. Therefore, a research gap exists in terms of cross-national comparisons of R-indicators and comparisons over time. This paper utilises the European Social Survey (ESS) contact form data to contribute to bridging that research gap. Using sample frame data and interviewer observations, a form of paradata where interviewers observe certain characteristic of the household and its neighbourhood, we construct R-indicators for each country-wave and identify variables that contribute to the lack of representativeness. We find that interviewer observations can identify nonresponse patterns that are mostly consistent across countries and time in the ESS. We conclude by discussing potential uses for these findings in survey practice, including the feasibility of using them for nonresponse bias correction.