Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Innovations in measurement instrument construction for web-based surveys 3 |
|
| Convenor | Mr Simon Munzert (University of Konstanz, Germany) |
Large parts of the increasing research body on web-based surveys deal with coverage, sampling and nonresponse issues, and therefore questions of representativeness. Less frequently discussed are measurement issues which arise from the unique way web-based surveys are conducted. In comparison with other modes, web-based surveys provide a bouquet of new tools and methods which allow for previously unknown flexibility in designing measurement instruments: The respondent can be presented (audio-)visual additional to (or even as a substitute of) verbal information, question and item order can be easily randomized, and valuable paradata like response latencies, key stroke measures or server-side paradata can be collected on-the-fly. These tools may help reduce respondent's burden when answering the questionnaire, but also allow for developing completely new instruments of existing concepts (e.g., visual measures of any kinds of knowledge). Although measuring opinions, facts etc. in an online setting might induce additional measurement bias in comparison with other modes, the web survey toolbox may provide instruments which can and should be used to fight these sources of error.
The goal of this panel is to bring together scholars who make use of new web survey tools to improve existing or construct new measures of a variety of concepts. The focus hereby is not so much on purely stylistic adaptions of the questionnaire layout, but on development of new instruments with methods going beyond ordinary question wording or response scale modifications. Papers to be presented in this session might deal with one of the following topics:
- innovative adaptation of existing or development of new instruments in web-based survey setting by use of unique web survey tools
- usage of web survey paradata to reduce survey error, or as a substantive measure
- studies implementing a cross-validation or MTMM design
Many longitudinal studies are now incorporating web-based interviewing. This brings a significant challenge for one of the most important elements in a longitudinal study, the collection of event history information. In face to face and telephone interviews the interviewer plays an important role in guiding respondents through these sections, but in online interviews this is not an option.
The functionality of the web can however facilitate the use of visual and interactive event history calendars, which have previously been shown to increase the quality of reporting of events.
This paper will describe the development of an interactive visual calendar for the National Child Development Study, a longitudinal study following a cohort of British people born in 1958, which in 2013 will incorporate online interviews for the first time. The calendar collects histories across multiple domains and aids recall by providing immediate visual feedback across these domains.
The development process involved multiple stages of usability testing where respondents were videoed while using the calendar as well and interviewed by researchers about their experience of using it.
This paper will discuss the development of the interactive calendar and describe the iterative programme of usability testing. It will emphasise the importance of allowing for multiple stages of testing by showing how significantly the research instrument changed during the development process as well as sharing videos recorded during the usability testing.
In social sciences, time-use surveys or paper-and-pencil diary research is the most profound and valuable way to capture human behaviour as opposed to any other social-scientific research method (e.g. retrospective questionnaires). Time-use research derives its strength from a sequential, activity-based registration method, including the context of each activity (e.g. with whom, where, ...). There are, however, two major disadvantages: the costs and the highly intensive registration method.
These high costs stem from the need from multiple visits of interviewers, coding and conversion of paper-and-pencil diaries to a digital database and the cleaning process hereof.
The aim of the first part of this contribution is twofold: 1) we demonstrate how MOTUS reduces these costs and 2) what additional benefits MOTUS this yields. Cost reduction stems from replacing interviewers by a series of automated e-mail and text messages to guide respondents through the research process, online activity registration allows direct data-storage, and build in control scripts for inconsistencies (e.g. the identification of unspecified time-periods) allow direct "cleaning". The additional benefits are, for example, decrease of social desirable answering because of the absence of an interviewer (cf. Krauter et al. 2008), freedom for respondents to fill in their questionnaires and diaries at their own pace and moment in time (cf. Evans et al. 2005), and respondent-based cleaning which makes researcher's interference on, for example, unspecified time unnecessary (cf. Bonke et al. 2010).
The common view is that introducing CAWI in the Labour Force Survey will lead to a lack of quality concerning the measurement of the estimates. This presentation will cast light upon the quality improvements one can obtain by introducing CAWI in the LFS.
Especially with regards to measuring working hours it will be emphasized that in CATI there may be interviewing effects caused by the presence of an interviewer and timeaspects concerning the amount of time from the reference week to the interview, reducing the quality of the measurement of working hours.
The presentation will illustrate that interviewing effects and time aspects in CATI reduce the quality of the
measurement of working hours. There is a tendency to overestimate and round the actual worked hours in CATI. First of all it can be due to interviewing effects (the presence of an interviewer, the speed of the interview etc.). Secondly, the hours actually worked and the rounding effects increase the further away from the reference week, the interview takes place.
Furthermore the average working time is higher in CATI than in CAWI. However, there are two opposite effects. The interviewing effects in CATI increase the working time, while the problems with collecting interviews by CAWI in periods of holidays etc. increase the working time in CAWI compared to CATI.
The presentation concludes that in general the collection mode has more explanatory power than whether the interview takes place one or two weeks after the reference week.
The Health and Retirement Study (HRS) is the premier survey of older Americans' work and health transitions. While also covering intra-family relationships, it remains limited to reports by one side. This feature greatly limits research on intergenerational transmission, support-and-exchange, etc. Recent introduction of a web-HRS, however, opens to a unique opportunity for surveying children and parents of current respondents, while circumventing high costs and logistic difficulties associated with in-person interviews.
We pilot this idea using the American Life Panel (ALP), a panel of 5,000+ respondents targeting the adult US population and an ideal test bed:
(a) Since all members completed the 2012 web-HRS, the HRS family roster provides us with a sampling frame of their adult children, living parents, spouse and in-laws.
(b) Existing "invitation" and "subscription" protocols (from respondent-driven sampling experiments using members as seeds) provide us with a customizable framework for our application.
We design and field
(1) An "invitation module" asking a sub-sample of ALP to invite their eligible relatives from the pre-loaded family roster (a) to join the study.
(2) A "subscription module" enabling invitees to join the study.
We randomize along different invitation and subscription features, and we derive and compare estimates of inviters' invitation and invitees' participation rates by relationship type. We complement the information with questions about intergenerational use of web and communication technologies. We follow up with inviters on their experience and how that may relate to characteristics of intergenerational ties.