Managing Change in Survey Time-Series 1 |
|
| Session Organiser |
Ms Daphne Ahrendt (Eurofound) |
| Time | Wednesday 17th July, 14:00 - 15:00 |
| Room | D01 |
Many survey research organisations are faced with the challenge of having to deal with change in their survey series. The impact of social media, big data, technological progress and falling response rate are forcing survey researchers to adapt more quickly to change than ever before. For organisations that have collected data over time, this need for change poses additional challenges as comparability with earlier rounds of data collection is at stake.
As an EU-Agency that has been fielding CAPI and CATI EU-wide representative surveys since 1990, Eurofound itself currently is faced by these challenges and has initiated a programme to develop a long-term vision for its surveys. In 2018, fieldwork on the European Company Survey (ECS) will move to CAWI, after previous rounds were done using CATI. The change from CAWI to CATI was preceded by several tests and a feasibility study. For the European Working Conditions Survey (EWCS) and the European Quality of Life Survey (EQLS) one of the questions to consider is how long these Europe-wide surveys can continue to be fielded face-to-face.
In this session, we wish to bring together researchers/institutes to talk about how they deal with change in their surveys. Some of the issues that could be discussed during this session include:
• How survey organisations have managed transitions from one mode to another
• How much testing is needed when considering a mode switch?
• Trends: how important are they, what constitutes a trend and when is it broken
• How much longer can we carry out f-2-f surveys in Europe?
• Experiences of sampling with mode transitions
• How important are response rates, is online a viable solution
• Change and opportunities: how can we build the need for change into improvements in data collection?
Keywords: mode transitions; cross-national surveys, time-series
Dr Eva Zeglovits (Institut für empirische Sozialforschung GmbH) - Presenting Author
Dr Reinhard Raml (Institut für empirische Sozialforschung GmbH)
The Quality of Life Survey in Vienna has been conducted as a CATI only survey since the 1990ies. For the 2018 survey, mixed mode combining CATI and CAWI has been introduced to overcome noncoverage and nonresponse bias. However, this possibly flaws comparisons over time. In the paper, the authors will present findings from the 2018 survey with a sample size of more than 8.000 tackling the questions:
How was the transition from CATI only to CATI/CAWI organized in the fieldwork?
Did mixing the modes help to improve data quality?
What happened to trends, and how could mode effects explain differences?
As results from this study are used as an input for evidence based policy making, it was important to report trends. Thus, the authors had to develop a strategy to correct for the mode effect to present valid trends. This strategy will be presented and discussed.
Mrs Becky Hamlyn (Kantar Public) - Presenting Author
Mrs Alice McGee (Kantar Public)
The UK Office for National Statistics (ONS) is investigating the feasibility of moving its household surveys (traditionally interviewer-led) to multi-mode data collection. As part of this, Kantar Public has been working with ONS to explore the feasibility of transitioning the questionnaire for the Crime Survey for England and Wales from an interviewer-administered to a mixed-mode instrument which can be self-completed online. This extensive development project involved 100 depth interviews, using cognitive and usability testing to iteratively develop a new approach to collecting complex crime count data on a range of devices.
The Crime Survey questionnaire is exceptionally complex and relies on trained interviewers to accurately record and count crimes. Although the survey has evolved over time, at its core, the method by which crimes are measured and counted has remained unchanged. This is both a strength and weakness of the survey. While continuity has allowed robust tracking of trends over time, there has been little scope to improve or update questions.
Although there are clear risks to the time series, a movement to online surveying creates opportunities to make the questionnaire more streamlined, tailored and user-focussed. In this paper, we will share our experiences of re-developing the survey questions for mixed-mode administration, focussing on the development of specific questionnaire features which are expected to have a wider application outside of the Crime Survey. This will include: our approach to simplifying and streamlining complex questions; creating a more tailored respondent experience by adapting order and flow; a new approach to collecting crime count data; and the feasibility of collecting detailed open-ended crime descriptions.
The paper will finally put forward a set of recommendations for best practice when designing or transitioning complex questions for online and smartphone self-completion, and we will discuss our thoughts going forward for further innovation in this space.
Mrs Katharine Sadler (NatCen Social Research) - Presenting Author
Miss Franziska Marcheselli (NatCen Social Research)
Mrs Dhriti Mandalia (NatCen Social Research)
Managing change in longstanding surveys that measure change over time is challenging. There is a tension between a) monitoring long-term trends b) meeting current data requirements and c) responding to anticipated future need. In this paper we look at the challenges we faced in maintaining comparability in measuring the prevalence of mental disorder in children and young people living in England over the past 20 years.
In each of the three surveys, the Development and Well-Being Assessment (DAWBA) was administered to a stratified probability sample of children and young people and their parents and teachers. Retaining comparability of question and mode was key to retain comparability between the surveys. However, in keeping with broadening definitions of adolescence, the 2017 sample was the first in the series to include 17 to 19 year olds. Children aged 2 to 4 were also included in the sample, offering a rare insight into the prevalence of mental disorders in preschool aged children. The 2017 survey was also the first in the series to include Body Dysmorphic Disorder, while a series of additional topics were developed reflecting large societal changes since the 2004 survey (for example social media and cyberbullying).
We discuss the strategies we adopted to manage changes in the survey design and measurement of mental disorder and assess how successful these strategies were.