Questionnaire translation in theory and practice: achievements, challenges, and innovations 2 |
|
| Chair | Dr Dorothée Behr (GESIS - Leibniz Institute for the Social Sciences ) |
| Coordinator 1 | Ms Brita Dorer (GESIS - Leibniz Institute for the Social Sciences) |
| Coordinator 2 | Dr Alisú Schoua-Glusberg (Research Support Services Inc.) |
This presentation will focus on the linguistic and cultural problems involved in the translation of answer scales from English into other languages. First, it will discuss cultural issues in responding to scales, such as politeness, social desirability, or avoidance of extreme responses. Next it will discuss linguistic issues, such as problems in maintaining scale polarity across languages, and the balance between fidelity to the source and avoiding awkwardness in the translation. Finally, the presentation will focus on a specific issue in scale translation: surveyspeak (Harkness 2000) in scales and how to best treat it in translation.
Translations of scales often try to convey the surveyspeak present in English scales. The target population for many survey translations in the U.S. are immigrants with limited prior exposure to surveys. They often lack familiarity with the task of selecting a response from a series of options the interviewer offers. They tend to treat the survey as a conversation, in which they do not carry out naturally the fourth step of Tourangeau et al.'s (2000) survey response model: formatting the response. We often see a tendency to answer in their own words instead of using the categories offered. This has implications for translation: we propose the notion that while a question stem can be interpreted as intended even if it is formulated in words the respondent understands but would not necessarily use, selecting a response is easier if the scale labels are formulated in ways that are closer to the way a respondent would answer the same question if it were open ended.
We will report on an experiment to elicit scale labels in Spanish, a card sorting exercise to group the labels and order them by intensity, and will compare ease of responding with the more traditional translated surveyspeak labels and the new labels elicited from the target population.
In cross-national survey projects, discussion often takes place when question translation requires adaptation. Adaptation refers to changes to the survey instrument that are deemed necessary to make a question more relevant or adequate to a specific group of respondents, and can be rooted in various linguistic, cultural and measurement features. Even though such changes are virtually ubiquitous in any survey translation effort, survey researchers sometimes avoid implementing them on the assumption that the closer a translated version is to the source question, the more comparable the data obtained from those questions will be. At the same time, versions tailored to the target population might lead to optimal measurement in that population and perhaps also to more comparable measurement. There is no guidance in the literature with respect to when adaptation improves or hinders quality and comparability, and how teams should resolve disagreements regarding the need for specific adaptations. This study (carried out in the context of the SERISS cluster project) intends to provide empirical evidence on best practices for adaptation in survey translation, investigating the advantages and disadvantages of two translation strategies that vary with respect to the degree of adaptation that is allowed. A selection of 40-50 questions suspected to need adaptation will be translated from the source language (English) into three target languages—Slovene, Russian and Estonian—in two different ways. One translation will be carried out to ‘stay as close as possible to the source questions’ within reasonable limits, and the other translation will follow an adaptation-driven approach where the translation team is encouraged to tailor the question as they consider most appropriate for the context in which the question will be fielded. Two independent translation teams will be set up, including professionals with similar expertise and experience who will be randomly assigned to a team. When the two approaches yield different versions, these will be tested in the CROss-National Online Survey (CRONOS) panel across three countries. Data quality and comparability will be judged by comparing the estimates obtained with each version to high quality benchmark estimates and by carrying out invariance testing analyses that compare each version to the source version. The paper will describe the final design of this experiment, the types of items selected, the instructions given to the translating teams and the analytical approach for measuring the differences between versions.
Our paper documents the introduction of a Polish language survey into the Irish European Social Survey (ESS), Round 8. The ESS quality report for the Irish ESS Round 7 suggested that the quality of the Irish data may have been comprised due to non-response from those who do not speak English (roughly about 5% of those who did not participate in the survey). As such, it was suggested that the Irish ESS survey be made more inclusive by including minority languages. Ireland has seen a large influx of migrants from across Eastern Europe and Africa since 1997. However, there is no one minority language that is spoken by over 5% of the Irish population. As of 2016, the second most commonly spoken language in Ireland after English is Polish, spoken at home by between 2-3% of the population. Since there are no data from the Irish ESS Round 7 indicating which languages were spoken by those who would not respond due to language issues, we used Polish as our best guess as to which language to target for an initial experience of integrating a minority language survey into the Irish ESS Round 8. The challenge of the process was to include a Polish survey while also abiding by the strict methodology regarding sampling, translation, and data collection required by the ESS. This paper charts the process involved in bringing in an ESS survey from another country (in this case Poland), finding translators and interviewers, localising the survey to Ireland, piloting in Ireland, and collecting the data. The paper presents an evaluation of how successful the integration of the Polish survey in the Irish fieldwork of ESS8 was, taking into consideration cost and the number of Polish speaking households ultimately included. It also provides a critical assessment of the process. This was the first time such a process was undertaken in the European Social Survey and so will offer unique insights and assessments which may inform similar processes in other countries.
Steve Dept (cApStAn) – Danina Lupsa (cApStAn)
The Double Source Double Translation Design in the HPEI
The Halin-Prémont Enneagram Indicator (HPEI) is a personality questionnaire used in leadership coaching. It is a scientifically validated tool, similar to the MBTI and PCM instruments, and it uses 52 questions to attribute a score out of 100 for each of the 9 personality types in the Enneagram.
The HPEI was first developed and validated in French. An English version was produced using a double translation and adjudication design. It was reviewed by an I/O psychologist and then tested and validated in the field. Theoretically, additional language versions could be produced using either the French source version or the English version. However, cApStAn chose to implement a double source double translation design: one translator (T1) translates from French into the target language (German in this example); another translator (T2) translates from English into German. Then a German reconciler (REC), who is proficient in both English and French, merges the two translations into a pre-final version, trying to keep the best elements from each. REC documents her work and earmarks all difficult choices and controversial issues for discussion. The reconciled version is then sent to T1 and T2, who both review it and add issues for discussion.
An online adjudication (ADJ) meeting then takes place: T1, T2, REC and a moderator examine all the issues earmarked for discussion and reach a consensus. REC functions as adjudicator when needed.
The adjudicated version is then submitted to a fourth linguist for proofreading (PRF) by a fresh eye. At this stage, no edits are made to the questionnaire: it is checked for linguistic correctness in the target language, not for faithfulness to the source. The German version (and other language versions produced using the same design) are in the process of being validated in the field.
In this session, we shall follow the entire translation history of 2 of the questions, step by step, from translation birth till adulthood, and show:
- what types of challenge were encountered;
- what decision mechanisms were used;
- what drove the choices made;
- what other options were possible;
- how the process was documented;
- what lessons were learnt; and
- what questions remain.
A variant of the double source double translation design is used in producing national versions of PISA assessments and questionnaires, but without adjudication step.