Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Investigating non respondents: how to get reliable data and how to use them |
|
| Convenor | Mrs Michele Ernst Staehli (FORS) |
| Coordinator 1 | Mrs Hideko Matsuo (KU Leuven) |
| Coordinator 2 | Mr Dominique Joye (UNIL) |
Since response rates are not a sufficient indicator of data quality, as they are not directly linked to nonresponse bias, survey researchers have increased effort to access reliable data about nonrespondents. The purpose is to detect, quantify and ideally to adjust for nonresponse bias. But collecting such data presents several challenges. Besides the problem of their accessibility, the cost of collection and the burden for interviewers as well as for reluctant/refusing sample units, the quality of the data is not simple to achieve. There either are very basic data on the whole sample frame which mostly don't explain much of nonresponse bias, or such data are based on rough evaluation i.e. by interviewers or very short questions raising problems of data quality, or they are collected by distinct surveys raising problems of comparability with the data of respondents.
This session proposes to gather and discuss experiences about the collection of such data with, in particular, the perspective of optimizing nonresponse follow-up surveys.
The main questions we want to address are:
- which kind of data are most useful (type of collection)
- which variables/items best detect nonresponse bias (content of information)
- which is or are the best way(s) to collect reliable data
- how can the quality of such data be assured and improved and
- eventually, how can these data best be analyzed?
We especially welcome papers that compare
- different sources of data (e.g. paradata, observable data, data from nonrespondent follow-ups or surveys)
- different types of information about the nonrespondents (e.g. contextual, socio-demographic, attitudinal)
- different designs for surveying nonrespondents
- different methods of analysis and bias adjustment.
We also look forward to discover unconventional items and innovative designs and methods.
In 2010, the French national institute of demographic studies (INED) conducted a survey on end-of-life in France. The survey covers a random sample of 15 000 deaths. For each death, the physician who filled in the death certificate was asked to fill/complete a questionnaire on the circumstances of the patient's end of life, whether online or by paper and postal mode. Response rate was about 40%. A follow-up survey, with a very short questionnaire with some socio-demographic questions, was then conducted by telephone among a sample of non-respondent physicians. Its response rate was fairly good (80%).
The objective of our presentation is to study non-response biases in this survey, using different sources and strategies:
- data from the follow-up survey on non-respondents;
- data issued from the fieldwork management software (which is disconnected from the response file);
- by analysing the characteristics of those physicians who responded by internet with those who responded by paper;
- by using different weighting strategies: while calibration (Deville et al. 1993) is the most appropriate way to compute weights in such a survey, there are several methodological possibilities. One is to use only benchmark statistics on the deceased, another is to introduce also the characteristics of the physicians. We will compare these two strategies.
References
Deville, J.-C., Särndal, C.-E. and Sautory, O. (1993). "Generalized raking procedures in survey sampling", Journal of the American Statistical Association, vol 88, n°423, pp. 1013-1020
In recent literature on survey nonresponse, new indicators of the quality of the data collection have been proposed. These include indicators of imbalance and representativity (of the set of respondents) and distance (between respondents and non-respondents), computed on available auxiliary variabels. The analysis of imbalance could be obtained as coefficients of variation of the response propensity scores. This finding is used in the data collection. At suitable points in the data collection, we may wish to reduce the number of contact attempts for selected sample elements, or to stop the attempts altogether for specified units. The response propensity scores help identify the units that should be targeted in these interventions.
Both theoretical and empirical resultat are presented. We use propensity scores in conjunction with paradata from the Swedish CATI-system to examine the inflow of data (as a function of the call attempt number) for the Swedish Living Conditions Survey (LCS). Cost savings realized by fewer calls can be redirected to enhance quality of other aspects of the survey design.
In the search for alternatives to response rates as indicators of nonresponse bias, representativity indicators, or 'R-indicators' are among the most promising. The use of R-indicators depends on the availability of auxiliary data that correlate both with the response propensity and key survey variables. Their suitability for social surveys measuring a variety of subjective phenomena may therefore be limited. We investigate this problem using data from the Swiss European Social Survey (2010), including new sample data based on population registers, and data from a nonresponse follow-up survey. The R-indicators are computed with sample data and the risk of bias on a hypothetical survey variable due to differential nonresponse across population subgroups (Maximal Absolute Bias) estimated. Data from the nonresponse follow-up are used to investigate the presence of bias on selected survey variables.
Examining changes in the R-indicator and Maximal Absolute Bias finds that the risk of bias is reduced as a result of fieldwork efforts. However, the difference between estimates from the main survey and the nonresponse follow-up suggests no improvement in biases resulting from response rate increases. We find only a weak correlation between the register variables used to construct the R-indicator and variables in the nonresponse follow-up suggesting that the constructed R-indicator may not be informative about the real extent of bias from nonresponse.
We discuss the strengths and the limitations of R-indicators and nonresponse follow-ups as evidence of the presence of bias in such surveys.
Nonresponse follow-up surveys (NRS) provide information on a subsample of non-respondents. Previous research on NRS data from the ESS showed that nonresponse adjustment based on NRS makes little difference in survey estimates between weighted and unweighted respondent sample. In order to strengthen the adjustment and gain more insight into the remainder of nonrespondents, the current study uses multiple imputation to first complete the NRS for all non-respondents based on rich frame data and paradata available for all sample units. Then, we model response propensity weights using the multiply imputed NRS variables. In the final step, non-response weights for respondents are computed by averaging propensity scores from the multiple imputed data sets and applying subclassification on propensity scores.