Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
The Contribution of Paradata in Analysing Unit Nonresponse Processes and Nonresponse Bias |
|
| Convenor | Ms Verena Halbherr (ESS, GESIS-Leibniz-Institute for the Social Sciences) |
| Coordinator 1 | Ms Johanna Bristle (SHARE, Max Planck Institute for Social Law and Social Policy) |
| Coordinator 2 | Professor Annelies Blom (GIP, Mannheim University) |
Researchers are invited to submit presentation proposals at the session "The contribution of Paradata in Analysing Nonresponse Processes and Nonresponse Bias" at the European Survey Research Association conference, July, 15-19, 2013 in Ljubljana, Slovenia.
Unit Nonresponse is one of the major issues affecting data quality in surveys. Nonresponse occurs in every survey and may cause biases in estimates. Over the last decades an increase of unit nonresponse has been observed generating a growing interest in understanding nonresponse processes.
One promising way of analysing nonresponse is by the use of auxiliary variables, especially paradata. These data might include information on contact strategies, interviewer observations, interview duration and time stamps.
This session focuses on two aspects of survey nonresponse: Analysing the processes leading to nonresponse during fieldwork and the induced nonresponse bias. This might include presentations on fieldwork monitoring and responsive designs, as well as the use of response propensities and R-indicators. We specifically encourage submissions using paradata in the nonresponse analyses of face-to-face surveys based on strict probability samples.
The European Foundation for the Improvement of Living and Working Conditions (Eurofound) carries out three recurring Europe-wide surveys: the European Working Conditions Survey (EWCS), the European Quality of Life Survey (EQLS), and the European Company Survey (ECS). In recent years Eurofound has placed increased emphasis on the systematic recording and storing of paradata (i.e. auxiliary information in the form of contact data). In the last wave of the EQLS, interviewer observations of respondent gender and type of residence were also recorded. Furthermore, in both the 5th EWCS and the 3rd EQLS attempts were made to convert so-called 'soft-refusals'. The paradata allow for making comparisons between those respondents who are relatively easy and those who are relatively hard to contact (hard to find), the latter of which are assumed to be similar to the final non-contacts. They also allow for making comparisons between those respondents who agree to participate straight away and those who require more persuasion (hard to get), the latter of which are assumed to be similar to the final refusals. Using the interviewer observations from the EQLS, we are able to tentatively test these assumptions about the similarity between non-respondents and respondents that are hard to find or hard to get. First results indicate that as a result of non-contacts the 5th EWCS might slightly overestimate the fit between work-and family life and the 3rd EQLS might slightly underestimate levels of life satisfaction, social inclusion and optimism.
Across the waves of a longitudinal survey several interviewers may contact sample members to participate in a survey. One approach to identify the relative effect of various wave interviewers on current wave nonresponse is to use multiple membership (MM) models (assuming there is paradata that includes the identification codes of the interviewers allocated to each). Such models allow the effect of all distinct interviewers associated with a case to be incorporated in the model by attributing a weight to each interviewer effect. These weights represent each interviewer's relative effect. The choice of weights is either based on theory or an empirical assessment using the Deviance Information Criterion (DIC).
Fitting these models with various weights to data from the UK Family and Children Study showed no improvement in comparison to a simple multilevel logistic model that only included an effect for the current wave interviewer. This result may indicate that it is only the current wave interviewer who has an impact on current wave cooperation. Alternatively, this result may be due to a lack of power to detect the MM structure, either because insufficient cases experienced interviewer change, or because the higher-level variance is small.
Through simulation, this study assesses how accurately the MM models can identify the true weights. The quality of estimators and the power of tests are investigated when weights are chosen on an a priori theoretical basis and when weights are chosen on the basis of the DIC. The relevance to survey practice is considered.
High response rates were traditionally interpreted as an indicator of high data quality. In order to counteract increasing unit nonresponse, a number of measures to increase survey participation have been developed, mostly having little impact but requiring large (financial) effort. Moreover, recent studies have shown that blind maximization of response rates often fails to reduce nonresponse bias. In pursuit of high response rates, interviewers tend to follow the path of least resistance by concentrating on persons who are most likely to participate, thereby only reaching "more of the same" respondents. Therefore the allocation of additional fieldwork efforts on cases with low estimated response propensity has been suggested (Peytchev et al. 2010). If successful, nonresponse bias can be lowered by reducing the variance of response propensities and hence the covariance between response propensities and substantial variables. The paper demonstrates that response propensities can be estimated accurately in a face-to-face panel survey prior to data collection using both paradata (interviewer observations etc.) and data from previous waves. The paper also provides evidence from two experiments on how to intervene on low propensity cases through interviewer selection and interviewer training. The results show that these interventions successfully increased response rates for the low propensity group, reduced nonresponse bias, and even led to better data quality without increasing field costs. The paper concludes with summarizing the keys to success in case prioritization across different survey designs and modes of data collection.
This paper explores the use of ESS Round 5 paradata variables (eg. type of dwelling and neighbourhood characteristics, contact strategies and refusal information variables) to adjust survey estimates for non-response. By taking reluctant respondents, as opposed to cooperative respondents, as proxies for non-respondents, this paper obtains calibrated propensity weights in number of steps. Two types of propensity scores (i.e. focus is on ‘contactability’ and ‘survey cooperation’) are obtained. These two propensity scores are multiplied with each other to calculate weights based on a 5 class stratification. In obtaining propensity scores on survey cooperation, refusal sample is calibrated based on information for both refusers and other types of non-respondents that are neither refuser nor non-contact. Our analysis aims to pay attention on the evaluation of non-response weights on response distributions. It also aims to study its effect in substantive models such as usual prime suspect variables (e.g. political interest, immigration, etc), and work, family and well-being variables. Case studies will be carried out for countries with high number of reluctant respondents (e.g. NL and other). The paper also addresses implications of data quality in paradata and respondent sample, for the effectiveness of non-response adjustments in cross-national research.