Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Web data collection for probability-based general population surveys 3 |
|
| Convenor | Professor Peter Lynn (University of Essex) |
| Coordinator 1 | Ms Lisa Calderwood (Institute of Education, University of London) |
| Coordinator 2 | Ms Gerry Nicolaas (NatCen Social Research) |
Web survey methodology is well-established for non-probability online panels and for specialist populations where web access can be assumed to be universal and where an available sampling frame includes email addresses. However, for probability-based general population surveys experience of web data collection remains limited. Many such surveys are now considering the inclusion of web within a mixed-mode design, though few have yet incorporated a web element and there is no consensus on the best way to do this. Meanwhile there is a very small but growing number of single-mode probability-based online panels, using different methodologies.
Survey researchers anticipate several potential benefits from the use of web data collection for general population surveys, such as reduced data collection costs and faster data collection. However, there are significant challenges to be overcome. Major themes include:
Sampling and coverage: How can we design surveys, incorporating web, so that they meet the representativity requirements of general population surveys?
Participation and engagement: What must we learn and do to engage participants in web surveys so that we get high, unbiased response and good quality data? In particular, how do we engage with sub-groups crucial to the success of social surveys - those with poorer access to technology and lower skills, those from disadvantaged and minority groups, etc?
Measurement challenges: How can we best capture complex data using the web and what new opportunities (and related research implications) are there for us to capture new kinds of data?
We welcome submissions to this session that address any of the issues faced by probability-based general population surveys with respect to the inclusion of a web-based data collection element. We particularly welcome reports of findings from experimental or developmental work. We also welcome case studies of general population surveys that have added a web element.
3rd co-ordniator Dr. Caroline Roberts, University of Lausanne, caroline.roberts@unil.ch
This paper summarises findings from an experimental general population survey in England that used web as its primary data collection mode for one part of the sample, and traditional face-to-face interviewing for the other part. The purpose was to:
*compare the effectiveness of the two sample recruitment methods, and of design elements within each (e.g. the number of contacts and the value of the incentives);
*estimate the level of error under each approach given an equivalent budget;
*make judgments with regard to the data quality of each approach and the potential for time series disjuncture if the new web-based approach was adopted.
This work has parallels with work carried out in the US by Dillman and in the UK by the ONS, albeit on a larger scale than any previous single study.
In summary, we found that, if a web-based approach was adopted,
*Response rates would be lower - and a lot lower among those sub-groups that (historically) respond at low rates to interview surveys too;
*The much larger sample sizes that are affordable mean that, for some sub-groups, the total error is probably reduced, even if generously ascribing no systematic bias to the interview survey version;
*True compatibility with the interview data is unlikely to be achievable given the normative ideals implicit in many of the questions used in the survey.
This paper also highlights the implications for research commissioners when considering switching data collection modes.
Many household surveys face pressure to reduce costs by adapting their modes of data collection. This paper reports on development work for the UK Household Longitudinal Survey (UKHLS) in moving towards introducing web in addition to face-to-face interviewing. We report on an experiment carried out on the UKHLS Innovation Panel (n = 1,568 households), designed to test whether cost savings can be achieved by introducing web in a household panel, without unduly affecting attrition or measurement. A particular challenge is the need to interview each adult member (16+) of each household: significant cost savings are achieved only if everyone in the household responds by web. The experiment contrasts face-to-face interviewing with a mixed mode design, where sample members are initially invited to complete the survey by web and non-respondents are later followed up by face-to-face interviewers. Further experiments tested ways of increasing the take up of web, either by offering additional conditional incentives, or by varying the day of the week on which e-mail invitations to the web survey are sent. We report various outcomes of the experiments, focusing on the effects of web on response rates but also addressing field costs and data quality. Initial results are mixed. Nearly one quarter (23%) of households completed all interviewing by web, suggesting considerable cost-saving potential. However, overall response rates were lower with mixed modes - except amongst previous wave non-respondents - and there was more item missing data.
We analyze effects of a randomized telephone / web mode experiment in a Swiss election survey. The web sample is randomized into a group receiving an unconditional 20 Sfr. cheque and a group without incentives. The samples in all three designs are representative of Swiss adult citizens and were drawn from a register of individuals.
We examine various survey quality measures. While the response rates are comparable in the telephone and the web with incentive designs, that in the web without incentive design amounts to about half of that with incentive. Incentives increase the response rate of young people, but not so much that of older people. While young people are underrepresented in the telephone survey, and overrepresented in the web surveys, old people are underrepresented in all surveys, but more so in the web surveys. There is more item nonresponse in the telephone survey, straightlining in the web without incentive survey, and few differences with respect to primacy and recency effects. Web with incentive costs twice as much, and telephone eight times as much as web without incentive.
Using web with an incentive should be considered as an alternative to the telephone mode usually used in the survey examined, especially when taking into account its reduced costs, and its relatively high survey data quality. In the long term, while increasing coverage problems of telephone surveys speak against this mode, the opposite is true for web surveys. This might eventually solve the remaining problem of reaching older people with web surveys.
In recent years the adoption of web survey techniques has greatly increased also in the field of official statistic, spurred by the evolution of technology, the changes in lifestyles, the wide-spread use of the Internet by people and enterprises, and the development of e-Government. Last but not least a furtherfactor supporting this trend is given by the increasing pressure to find effective methods to reduce costs.
The growing role played by the web data collecting technique is stressed by the decisions of the Italian National Institute of Statistics who adopted this approach in collecting the data within the recent 2011 Censuses. For the first time, the web mode (apart from certain telephone and face-to-face remainders) was the exclusive mode used to collect data within the net of the Public Administration, whereas a mixed mode still prevailed for enterprises and non-profit organizations (even though also in these cases web use was encouraged).
The paper discuss in detail the Italian experiences in the Tuscany region, regarding the most relevant aspects of the whole process, namely: a) fine tuning, control and updating of lists of units; b) definition of the check plans during the on-line filling in of questionnaires, stressing the trade-off between data quality, easy compilation and simplicity of the form; c) continuous monitoring of the process; d) expansion to universe modalities in presence of unit non responses.