Managing Change in Survey Time-Series 2 |
|
| Session Organiser | Ms Daphne Ahrendt (Eurofound) |
| Time | Wednesday 17th July, 16:30 - 17:30 |
| Room | D11 |
Many survey research organisations are faced with the challenge of having to deal with change in their survey series. The impact of social media, big data, technological progress and falling response rate are forcing survey researchers to adapt more quickly to change than ever before. For organisations that have collected data over time, this need for change poses additional challenges as comparability with earlier rounds of data collection is at stake.
As an EU-Agency that has been fielding CAPI and CATI EU-wide representative surveys since 1990, Eurofound itself currently is faced by these challenges and has initiated a programme to develop a long-term vision for its surveys. In 2018, fieldwork on the European Company Survey (ECS) will move to CAWI, after previous rounds were done using CATI. The change from CAWI to CATI was preceded by several tests and a feasibility study. For the European Working Conditions Survey (EWCS) and the European Quality of Life Survey (EQLS) one of the questions to consider is how long these Europe-wide surveys can continue to be fielded face-to-face.
In this session, we wish to bring together researchers/institutes to talk about how they deal with change in their surveys. Some of the issues that could be discussed during this session include:
• How survey organisations have managed transitions from one mode to another
• How much testing is needed when considering a mode switch?
• Trends: how important are they, what constitutes a trend and when is it broken
• How much longer can we carry out f-2-f surveys in Europe?
• Experiences of sampling with mode transitions
• How important are response rates, is online a viable solution
• Change and opportunities: how can we build the need for change into improvements in data collection?
Keywords: mode transitions; cross-national surveys, time-series
Professor Jin-Shei Lai (Northwestern University) - Presenting Author
Professor David Cella (Northwestern University)
Dr Amy Paller (Northwestern University)
Dr Cindy Nowinski (Northwestern University)
Aims: Revising existing questionnaires to accommodate patients with different conditions is a common practice. Yet maintaining comparability of scores produced by these versions is challenging. Using the pediatric Neuro-QoL Stigma measure (NQ-Stigma) as an example, this study demonstrates our approach to constructing a common metric for children with various conditions who completed different versions of NQ-Stigma.
Methods: Data from 842 children ages 8-17 years were analyzed. 110 had a diagnosis of epilepsy, 140 pNF (neurofibromatosis type 1 associated neurofibroma plexform), 43 MD (muscular dystrophy), 82 cancer and 467 skin conditions (328 had atopic dermatitis/AD), with mean age (yrs)=13.5, 12.6, 14.1, 12.7 and 12.5, respectively. Children with epilepsy, pNF and MD (group 1) completed the original 18-item NQ-Stigma, while children with cancer and skin conditions (group 2) completed a revised version with 6 new items. IRT-based differential item functioning (criteria: χ2 >0.01, R2 change < 0.02), DIF, was used to evaluate measurement equivalence on group, gender, age (8-12 vs. 13-17 years), and conditions (reference group: AD). DIF impacts (IRT-scaled score differences between “all items included” versus “DIF items removed”) were evaluated to determine the inclusion/exclusion of DIF items.
Results: No items showed group, gender or age DIF. Three items showed DIF when comparing: 1) pNF and epilepsy vs. AD; 2) non-AD skin conditions and pNF vs. AD; and 3) cancer versus AD. All DIFs had minimum impact (< 0.1 IRT-scaled score). A common scoring metric between these two NQ-Stigma versions was then constructed by co-calibrating 6 new items onto the original version using an IRT-based fixed-parameter approach.
Conclusions: We used IRT techniques to successfully establish a common metric between the original and revised versions, allowing users to take into account unique and common concerns for various
Dr Z. Tuba Suzer-Gurtekin (University of Michigan) - Presenting Author
Miss Yingjia Fu (University of Michigan)
Miss Caitlin Beach (University of Michigan)
Mr Peter Sparks (University of Michigan)
Dr Richard Curtin (University of Michigan)
University of Michigan’s Surveys of Consumers (SOC) have been publishing a leading economic indicator in the U.S. since 1950s. Throughout the years, the surveys went through design revisions to adapt the changes in the technology and the society. Since 2017 the study has been running parallel Web-Mail surveys to the Telephone surveys. This effort is to meet two specific research goals: 1) Understand the change in time series by each survey error source, and 2) Reduce differential measurement differences in web, mail and telephone modes. One particular issue related to differential measurement differences across three modes is related to open-end responses. Cognitive processing, social context and burden on the respondent are factors related to response quantity and quality that vary across three modes. As a result, in the web-mail data, the length of unaided mentions is shorter overall, although the overall proportions of specific mentions follow similar trends compared to those from the telephone data. In addition to the overall comparability, the subgroup comparability needs to be established for a future mixed-mode survey design. In this paper, we present results from our experimental designs to unify the open-ended response question design across all subgroups of interest for the SOC’s news heard of recent changes in business conditions question. The experiments are conducted in phases to isolate the differential measurement error and then project to the overall estimates.
Miss Noémie Soullier (Santé publique France)
Mr Stéphane Legleye (INSEE) - Presenting Author
Mr Jean-Baptiste Richard (Santé publique France)
The Health Barometer (HB) is a repeated health phone survey of the French population, conducted since 1992 by Santé publique France, the French national public health agency. The HB was initially carried out on landline telephone (LT) numbers, benefiting from high penetration of these telephones. However, with the spread of cellular telephones (CT), sampling only LT numbers suffered from non-coverage. Thus, since 2014, the HB has used an overlapping dual-frame design of LT and CT numbers.
Nowadays, more and more telephone users are switching from LT to CT. This evolution impacts the risk-benefit ratio of interviewing LT versus CT. For instance, several limitations of CT interviews are likely to reduce progressively (poorer quality of interview, more expansive cost per interview, lower response rates, etc.). Moreover, the penetration of CT has increased, including among the elderly, reducing the undercoverage bias of a CT frame. Finally, contacting individuals on LT implies a two-stage sampling design (sampling numbers and then sampling a single respondent within the eligible persons of the household), as individual use of CT allows to adopt a single-stage design.
Consequently, we could wonder if it is time to stop contacting people on LT and what would be the impact of such a change. Whereas recent research has focused on methods allowing to include a CT sample in phone survey designs, this study aims to explore the validity of a unique CT survey.
Using the dual-frame design of the 2017 HB (the largest French health telephone survey; n=25,319), we first provide comparisons of respondent demographic characteristics between LT interviews and CT interviews. Second, we explore the similarities and differences for key health indicators before and after adjusting for the demographic characteristics. Finally, we compare the prevalence for some health conditions and their precision, estimated on the whole sample versus the cellular sample only.