Questionnaire translation in theory and practice: achievements, challenges, and innovations 3 |
|
| Chair | Dr Dorothée Behr (GESIS - Leibniz Institute for the Social Sciences ) |
| Coordinator 1 | Ms Brita Dorer (GESIS - Leibniz Institute for the Social Sciences) |
| Coordinator 2 | Dr Alisú Schoua-Glusberg (Research Support Services Inc.) |
The European Agency for Safety and Health at Work (EU-OSHA) completed its second European Survey of Enterprises on New and Emerging Risks (ESENER-2) in 2014, interviewing almost 50,000 establishments across all activity sectors in 36 countries. The survey helps fill an important information gap in the world of occupational safety and health (OSH), particularly for the smallest business sizes as ESENER-2 covers establishments employing at least five people.
This presentation focuses on the main steps of the translation of the ESENER-2 questionnaire into 47 national versions:
a) A translatability assessment of the English master questionnaire version.
b) Two independent translators and one adjudicator per language version.
c) Domain expert feedback
In the translatability assessment, experienced translators from four different language families -Czech for Slavonic languages, French for Romance languages, Swedish for Germanic/Nordic languages, and Greek- elaborated rough translations of the master questionnaire in order to identify any ambiguities or other challenges for translation. Where such difficulties were identified, translators made proposals for alternative formulations for the master version. These were largely taken up and reflected in the master questionnaire.
Next, for each language version two professional translators, one adjudicator and one proof reader were selected. Training webinars were organised beforehand, involving both the translating team, the contractor in charge of the fieldwork and EU-OSHA (in order to explain specific content issues). The two independent translations were provided to the adjudicator, who produced a reconciled version that would then be discussed at a team review meeting –one for each of the 47 national versions of the questionnaire. In any case, a help desk service was offered throughout the entire translation and adjudication/adaptation process. There was a thorough documentation of each step of the translation using different worksheets of the same Excel file (one file per national versions), including hints, instructions and specific terminology.
Finally, the national versions of the questionnaires were shared with the domain experts, representatives of the national bodies in charge of OSH in each of the 36 countries covered in ESENER-2. The main aim was to ensure the appropriateness of the OSH terminology rather than a purely linguistic check and this was mostly the case. At times though there were even suggestions to improve the questionnaire, which had to be turned down (but were still recorded) for the sake of cross-national comparability. The final versions of the questionnaires were sent to proofreading, to check for correctness of the target language.
The entire process lasted from November 2013 to March 2014.
Our presentation will focus on the translation method applied to the 4th European Quality of Life Survey (EQLS). In particular, it will cover the translatability exercise that formed part of the process, cognitive testing and the application of the TRAPD model.
For the survey, a final source questionnaire in English, was developed which is now being fielded in 33 European countries (EU28 and 5 Candidate Countries). In total the questionnaire has been translated into close to 40 language versions.
As part of the questionnaire finalisation process, 30 cognitive interviews in English were carried out and the draft source English questionnaire was subjected to a translatability assessment in one Roman (French or Italian), one Germanic (Dutch or German) and one Slavic language (Czech or Polish). A senior linguist then collated the linguists’ feedback and produced a consolidated translatability report. The results of these two pre-translation exercises were discussed in a meeting between Eurofound staff, the researchers responsible for the cognitive tests and the linguists in charge of advance translation.
The EQLS TRAPD model can be outlined as follows.
Two independent translations are carried out.
For existing questions, for which translations are available from previous waves, will not need to be retranslated, translators will ensure coherence between the translations of the new and the existing questions by pointing out the key elements requiring attention. Should translators be of the opinion that coherence can only be ensured by changing an existing question, the suggested change (and accompanying argumentation) will to be provided to Eurofound for approval.
These two independent translations will be reviewed in a meeting between the two independent translators and the adjudicator. In an interactive session the two independent translations are discussed and the final translation is agreed. The adjudicator is responsible for the final decisions about the translations. The process and outcomes of decision making about the translation of each questionnaire item will be recorded systematically, distinguishing between situations where (1) translations do not differ, (2) translations differ substantively and (3) translations differ technically.
Next to a detailed discussion of EQLS TRAPD model procedures, the presentation will highlight particular questions, items and themes in the area of measuring quality of life that are susceptible for translation errors and interpretation challenges in the context of survey translator work in Europe using the findings from the translatability exercise and the cognitive tests. The session will provide an opportunity to discuss lessons learnt for achieving optimal equivalence with its measurement instruments.
The preparation of survey instrument based on the translation from other language consists of two major stages – actual translation and the verification of the translation – checking the quality of the translation. Many errors in comparative surveys are attributed to translation errors or translation differences. The verification of the translation is intended to minimize translation errors and to improve the comparability of survey instruments. But the question which verification method is the most effective still needs to be addressed. We define translation of survey instrument as the procedure of expressing stimulus (verbal or written text of question wording, question categories and other supplementary materials) in different languages. It can be used either for comparative purposes (between countries, nations or within countries or between different ethnic or linguistic groups within one country) or for non-comparative purpose - adopting research instrument from other survey and language. The requirements and the criteria of assessing the quality of the translation is similar although not identical for these two major translation tasks. In case of translation for comparative purposes the quality of the translation implies high correspondence of translation version to source text, equivalence to other languages used in the survey, satisfying the rules and demands of the language of the translation and satisfying the linguistic requirements of target group. Last two tasks are related to the data quality for particular country, first two – with the comparability of survey data. We suggest to understand the quality of the translation as the degree of semantic or grammar deviations or differences between source text and translated text. Detecting errors in the translation is not trivial task, and the effectiveness of different methods is not completely clear from the current literature. We will present the results of the experiment of using different verification methods of the same translation from English into Russians – expert review, external verification, back-translation, multiple translations and pretest on target population. We calculated the number of “issues” or “deviations” registered by each verification method and also classified what kinds of “issues” were detected. The analysis led to the conclusion that verification methods are different not only in number of detected issues, but also in types of issues found - grammar, formatting and semantics. Therefore some of these methods can be treated as complimentary rather than as alternative to each other.
Office of National Statistics has established a programme to standardise measurements that assess social and economic aspects of life in Britain. The “harmonised” concepts and questions are becoming the norm in collecting certain categories of data across national surveys in Britain, including for well-being. These questions can now be compared across years and surveys, and thus provide robust national statistics data to inform social and economic policy.
The current project is aimed at translating and testing the ONS harmonised well-being questions into various minority languages from the UK. The languages are spoken in the Bangladeshi, Pakistani and Polish communities in the UK. This project is of particular importance because it is evident from census data that the people who are not fluent speakers of English would find it difficult to understand and accurately answer survey questions. Nonetheless, they also form a vulnerable group, with less access to governmental and other resources and support, so a better measure of their well-being would be of high importance to governmental departments, local authority providers and charities.
Recent studies in translations have looked closely at linguistic and cultural equivalence. The standard approaches when translating surveys tend to be translation/back-translation and cultural checking by native speakers. Yet this can easily mask substantial differences in meaning between translations, which will reduce the power of the statistical analysis comparing or aggregating the data from different cultural and/or linguistic contexts. In order to achieve language equivalence and thus comparability between the languages, questionnaire testing work should be undertaken in all languages used. This should be a standard step in the development of the questionnaire, particularly using pre-testing methods to ensure cultural and linguistic equivalence.
This is particularly challenging when the languages used belong to groups that are culturally very different from the main language and are not fully integrated to the mainstream society. This poses two main challenges: 1) establishing a rapport and good working relationships with these groups that are sensitive to their cultural differences and 2) being mindful of how these cultural differences affect the way people respond to questions. An added complexity is that in some cases these languages can involve many dialects and may not have a standard written form.
We have devised a new approach for testing the translated questions in multiple stages:
• initial translation by two translators;
• workshop bringing together experts in well-being research, cognitive interviewers, questionnaire design experts, experts in harmonisation, translators and interpreters – the purpose of the workshop is to arrive at an accurate and culturally equivalent translation of the questions;
• cognitive testing with (almost) monolingual speakers of each language
• debriefing - the cognitive interviewer and the questionnaire design experts refine the questions to reflect the linguistic and cultural meaning, based on the results of the cognitive testing.
This approach has proven successful, bringing together the methodological expertise with a novel linguistic approach to translating survey questions. This paper will present the practical challenges of implementing this approach in the context of ONS .