An integrated tool suite for Round 10 of the European Social Survey (ESS) |
|
| Session Organisers |
Dr Brita Dorer (GESIS-Leibniz Institute for the Social Sciences) Mr Tim Hanson (European Social Survey; City, University of London) |
| Time | Friday 16 July, 16:45 - 18:00 |
The European Social Survey (ESS) is known for its strong focus on developing and applying rigorous methods for achieving a high level of cross-cultural survey data. In its 10th round, it is for the first time using an integrated survey tool suite, which includes a centralized CAPI script, a translation tool, and a fieldwork management system. Around 12 countries will use the tools for ESS Round 10, facilitating greater consistency, streamlining processes, and taking advantage of the opportunities of the digital age.
This session will zoom in on this tool suite. First the overarching idea will be discussed, technical details on the linkages between the various elements presented. Then in particular two tools will be presented in more detail: the platform for managing the complex translation processes into 30+ language versions, TranslationCTRL (TC) - formerly known as Translation Management Tool (TMT) - as well as the Fieldwork Monitoring System (FMS).
Keywords: cross-cultural surveys, survey tools, centralisation, European Social Survey (ESS)
Mr Tim Hanson (European Social Survey; City, University of London) - Presenting Author
Mr Maurice Martens (CentERdata, University of Tilburg)
The European Social Survey (ESS), conducted every two years since 2002, combines a centralised fieldwork specification, planning and oversight, with decentralised fieldwork contracting and delivery in each participating country. For Round 10 of the survey, due to be completed in 2021, a centralised survey tool suite (developed by CentERdata) has been introduced to increase consistency between countries and allow for greater central control and oversight. The tool suite also aims to remove possible barriers to entry for some countries, particularly those with more limited resources or research infrastructure.
The tool suite comprises several elements, including a central questionnaire program, a translation tool, and a fieldwork management system. It is expected that around 12 countries will use the full tool suite at ESS Round 10.
This paper will summarise the rationale for adopting the tool suite and present some of the expected benefits. An overview of the different tools will be presented, with more specific detail included in other papers in this session. We will outline some of the challenges in adopting a centralised approach for the first time on a long-running, cross-cultural survey. Finally, we will summarise learning from our experiences to date and consider next steps.
The paper is expected to be highly relevant to those either currently running or planning cross-cultural surveys, especially those that combine decentralised fieldwork with a need for a high level of central oversight and control.
Mr Maurice Martens (CentERdata, University of Tilburg) - Presenting Author
The European Social Survey (ESS), conducted every two years since 2002, combines a centralised fieldwork specification, planning and oversight, with decentralised fieldwork contracting and delivery in each participating country. For Round 10 of the survey, CentERdata has implemented the technical framework that allows countries to translate and field their questionnaires. In addition, a central system is now in place that allows for greater central control and oversight.
The tool suite comprises several elements, including a central questionnaire program, a translation tool, and a fieldwork management system. It is expected that around 12 countries will use the full tool suite at ESS Round 10.
This paper will discuss the underlying model, will discuss the workflow involved and will demonstrate and discuss the tools. In the same session, the tools will be discussed from user, and management perspective.
Mr Niccolò Ghirelli (European Social Survey; City, University of London) - Presenting Author
A key strength of the European Social Survey (ESS) is its emphasis on functionally equivalent survey designs, shared data collection, and data processing protocols. This input-harmonised approach enables robust comparisons across European countries to be drawn. However, data collection activities are conducted by decentralised national teams and survey agencies. This represents a challenge for the central monitoring of the adherence to standards and the resulting data during fieldwork in a timely, consistent, and comprehensive way. To overcome this issue, the ESS has introduced a new electronic Fieldwork Monitoring System (FMS).
From Round 9, ESS countries are required to upload weekly monitoring information to the FMS online portal using a pre-defined template. It provides all ESS stakeholders (survey agencies, national teams, and the central team) with access to timely, shared data on fieldwork progress at the case-level in a standardised format and generates pre-defined summary indicators across countries. Access to such data has the potential to make monitoring more efficient and effective and offers new opportunities to study the effectiveness of fieldwork strategies across countries in a comparable way. Additionally, it highlights the challenges inherent in translating this information into better and more responsive fieldwork monitoring across multiple countries.
In this paper, we provide an overview of the monitoring data of the FMS and the analysis it can facilitate. We present real-life examples of the use of FMS by the central team to inform fieldwork monitoring. Furthermore, we reflect on the challenges and limitations of the current tool and available indicators that have been exposed in the process of monitoring fieldwork in more than 30 countries simultaneously. We discuss the implications of these findings for fieldwork monitoring in future rounds of ESS and attempt to generalise lessons for the development of central monitoring on other cross-national surveys with decentralised data collection.
Dr Brita Dorer (GESIS-Leibniz Institute for the Social Sciences) - Presenting Author
Mrs Veronika Keck (GESIS-Leibniz Institute for the Social Sciences )
Mr Maurice Martens (CentERdata, University of Tilburg)
Mr Sebastiaan Pennings (CentERdata, University of Tilburg)
The European Social Survey (ESS) is known to apply a rigorous methodology for translating its questionnaires. While in earlier rounds, these translation processes had mainly been organized in Excel-based translation templates, the ESS has been moving more and more towards using the online translation platform TranslationCTRL (TC), formerly known as Translation Management Tool (TMT).
Now in its 10th round, this platform has become a firm element in its translation process, and is currently used for about 25 language versions.
In ESS 10, for the first time, this tool is embedded in the larger process of producing the ESS survey instruments, so for the first time, the full translation process from the initial parallel translations up to programming the survey instruments are based on TranslationCTRL.
This platform is an online service for supporting questionnaire translation processes for large multilingual surveys, originally developed for SHARE. It was adapted to be used for supporting other large-scale multilingual surveys such as the ESS and EVS.
The ESS translations are based on the TRAPD approach, complemented by translation verification by the external service provider cApStAn as well as the Survey Quality Predictor (SQP), administered by the ESS team at UPF Barcelona.
The platform has been updated after two testing rounds in the ESS 8 and 9, and further adapted to the ESS needs.
This paper will present the current version of TC, its developments compared to previous rounds, specific assets tailor-made for similar large multilingual survey projects, such as facilitating team approaches, providing a platform for Review sessions as well as international shared language processes – as well as ideas for further improvements ahead. The workflow with typical arrangements will be discussed as will specific features common to computer-assisted translation (CAT) tools, which are state-of-the-art in the broader translation business.
An important step was the inclusion of this platform into the wider suite of other tools used in the ESS as of round 10, which are the overarching topic of this session.