Producing Data for Comparative Social Surveys: Survey Management, Collaboration and Communication and the Impact on Data Quality |
|
| Session Organisers | Professor Jürgen H. P. Hoffmeyer-Zlotnik (Institute of Political Science, Justus Liebig University, Giessen) Dr Uwe Warner (Methodenzentrum Sozialwissenschaften, Georg-August-Universität, Göttingen) |
| Time | Friday 19th July, 13:30 - 14:30 |
| Room | D11 |
Doing comparative social survey research involves many kind of people. They fulfill different tasks and act in various roles in the process of producing comparative survey data. And they have to collaborate. The implementation and organization of this collaboration affect the collected data, the design of the fieldwork instruments, the question wordings, the measurement quality, the selection of the measurement concepts, and the choice of social theories driving the decisions during this data production process.
The large scale data collections for comparative surveys (e.g. ESS, SHARE, ISSP, WVS/EVS, Eurobarometer, EU-SILC, EU-LFS, and others) have different strategies to realize their tasks. They have different division of labor on various levels to establish the communication and cooperation between the actors involved.
Common to all forms of organizing the data collection for comparative social survey data is the workflow
- starting with the decision-making about the theory-driven research questions,
- the selection of comparable and harmonized measurement concepts applied in the study,
- the evaluation of measurable variables measuring the social facts that are intended to be measures cross-nationally, cross-culturally and over time,
- the wording of the survey question understandable and meaningful to the future respondent,
- recording the interviewee’s answer,
- preparing and providing the data and the documentation, and finally
- making the survey and fieldwork reports available to the scientific community, so that comparative use of the data becomes meaningful and relevant for the research.
Involved in this workflow are central project coordinators, national research teams, national fieldwork agencies with their interviewers and last but not least the respondents answering the survey questions.
Among others, we are interested in following topics:
- How the degrees of collaboration and communication among survey producers for comparative studies affects the data production process and the data dissemination?
- How the relationships between the actors involved hampers or fosters innovation in comparative social survey research?
- How the teamwork among the involved actors ensures or prevents from comparison of social survey data?
- How the described workflow has implications to the measurement quality of comparative survey data?
- How the harmonization, standardization and the “making the social survey measures comparable” depend on the collaboration between the involved researchers?
Keywords: survey management, harmonization, standardization, measurement, data quality
Dr Evi Scholz (GESIS) - Presenting Author
The International Social Survey Programme (ISSP) is a cross-national social science survey programme with members from around the globe who voluntarily collaborate based on a common agreement and ISSP working principles.
An ISSP questionnaire consists of two parts: the source questionnaire on the thematic module and the obligatory ISSP background variables (BV). While the source questionnaire with an input harmonisation approach is developed by all ISSP members in a joint effort, the ISSP BV result from an output harmonised approach. Since 2010, there is a set of BV on age, sex, marital status, education, work situation, religion, household composition, party affiliation, ethnicity, and geographical reference which are listed in a comprehensive guide called "ISSP BV Guidelines". The ISSP BV are implemented by individual ISSP members combining national needs of asking country-specific questions and complying with ISSP measurement requirements described in the guidelines (measurement goal, coding frame, filter and routing conditions). The production of the ISSP BV from the national BV into the harmonized ISSP BV is documented by ISSP members in an obligatory documentation which is part the data delivery.
The ISSP in contrast to other cross-national programmes with hierarchical structures is a democratic project of equal members. This particularity results in a division of labour and a special project management with country-specific elements (national survey fielding, data management and documentation) on the one side and central elements (study monitoring of national data collection, control of national data on formal compliance with ISSP requirements while integrating into common ISSP file; additional projects on adherence with measurement goals) on the other side.
The presentation will describe the concept of the ISSP BV in general, elaborates on the way the ISSP controls the data in order to guarantee the quality and give an example for the German ISSP.
Professor Jürgen H.P. Hoffmeyer-Zlotnik (Institute of Political Science, Justus Liebig University, Giessen) - Presenting Author
Dr Uwe Warner (Methodenzentrum Sozialwissenschaften, Georg-August-Universität, Göttingen)
The main tasks of the members of the national research teams are to realize the survey in their countries according to the commonly agreed rules, to implement the national particularities in the fieldwork instruments, and to collect, prepare, and transmit the national data. This means that the national experts must (a) draw the sample respecting the common requirements, (b) transform the common blueprint of the questionnaire into the national survey instrument, (c) organize and supervise the fieldwork, (d) prepare and check, the national survey data and transmit them together with the national documentation to the central project coordinators.
Of importance is the close collaboration between the central project coordinators and the national research teams while the questionnaire blueprint is being designed to elaborate the country- and culture-specific differences. The translation of the blueprint into the national questionnaires must respect the psychological model of comprehension, retrieval, judgment, and response during the interview situation.
After translation, the blueprint must be tested for cultural understanding. The sociodemographic variables must be transferred from the blueprint into a functionally equivalent national instrument. In this connection, the national contexts of the measures and the underlying social structures must be considered. Respondents must understand the questions and must be able to generate an answer for the purposes of the research question.
In our presentation we focus on mistakes appearing during the harmonization of comparative survey measurements. We discuss survey errors coming from the communication between international project coordinators, national research teams, and local fieldwork agencies.
Mrs Birgit Jesske (infas, Institute for applied Social Sciences) - Presenting Author
Requirements for uniform and comparable measurement arise in every survey, regardless of whether data shall be collected nationally or internationally. The consistent pursuit of these goals requires the standardized implementation of processes and rules for the entire research project. Studies in comparative social research focus on the questionnaire and the harmonization of the questions. However, all processes connected with the fieldwork play a role for data comparability and, ultimately, for the quality of the data.
In the social sciences, generally mandatory standards form the basis for the cooperation of all those involved – even in comparative social surveys. The rules of standardized interviewing (Fowler & Mangione 1990) are an exemplary set of rules that govern and standardize the comparability of measurement. It is not always possible to enforce standardized processes in comparative social surveys among all partners, over all countries. Quite frequently partner-specific or country-specific processes do not allow for a change towards the desired standardization (for example special survey software, sample management systems). Sometimes even cultural conditions hinder practical application (for example interviewer calls on Sunday, cash incentives).
We have gained a lot of experience at infas in working with various comparative social surveys (including SHARE, ESS, LFS, HFCS) with different constellations in those projects regarding the collaboration of all project participants.
The article shall provide insight in the processes of interviewer training, sample processing and contacting as well as fieldwork monitoring. Which processes does study design regulate or predetermine? Which country-specific margins / deviations are permitted? Are there any requirements that may counterproductively affect conducting the study? How does a survey agency deal with those guidelines and how does communication with the researchers work?