Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Quality prediction and improvement of survey question using SQP |
|
| Convenor | Dr Willem Saris (RECSM) |
In 2012 the program SQP 2.0 has been released. The program contains a data base of more than 4000 survey questions of which the quality is known on the basis of MultiTrait MultiMethod experiments.
Besides that users can introduce their own questions in the system and get a prediction of the quality of these questions after coding the characteristics of their questions. These predictions are based on the relationship between the quality of the 4000 questions involved in MTMM experiments and the characteritics of these questions.
Several courses have been given to inform people about the possibilities and the use of the program. In these courses the emphasis was on the effect of measurement errors on substantive results of survey research and on the fact that one can correct for these errors if one knows the quality of the questions.
Now that so many people are familiar with this approach, we think that it would be interesting to have a meeting with researchers who have used the program to discuss with each other the advantages and disadvantages of the present version of the program. So we invite people who have applied this approach in their research to send in an abstract of their presentation. Not only methodological papers are of interest but also applications that show the differences in results with and without correction for measurement error.
Survey translation has developed best practice procedures to translate survey instruments: adaptation should be avoided and the same stimuli and measurement properties should be provided however it is very difficult to check in a systematic way these requirements. Monitoring of the formal structure of translated questionnaires in cross-cultural surveys is challenging because one cannot know all languages. In this paper we present a procedure to check and prevent differences in the form of survey instruments using Survey Quality Prediction program (SQP). SQP asks users to code a large set of properties of a survey item to give a prediction of its quality. Deviations in translations are detected by comparing the codes of a source questionnaire and targeted languages. The paper summarizes the findings of this procedure implemented in a set of items in the Round 5 and Round 6 of the European Social Survey (ESS).
This paper aims at showing the potentiality of SQP for improving the quality of questions and data analysis concerning attitudes in the ISSP. For the sake of comparability, each repeated module in the ISSP should contain at least two thirds of the questions asked in the previous round; repeated questions should be identical in both the question wording and the asnwer categories. Then, a change in either of these two features qualifies an item as a new item. This implies that even sub-optimal questions or answer options are maintained in repeated modules.
However, in principle, this should not prevent the improvement of the quality of the data produced by the Programme and extensively used by many researchers all over the world.
The paper will focus on the ISSP Social Inequality module, which has been fielded four times (in 1987, 1992, 1999 and 2009). The strategy of analysis follows two lines. In the first place, an overall evaluation of the quality of questions included in the last round (fielded in 2009 in 39 countries) will be performed by coding each question of the source questionnaire (in British English) using SQP; the main potential improvements as devised by SQP will be highlighted. Second, the correlations between attitudes towards income inequality and taxation will be analysed with and without correction for measurement error, in order to show how a substantial improvement on the quality of results can be attained.
Does political participation make individuals more satisfied with their lives? Classical philosophy and participatory democracy authors suggest that participation in political activities is a fundamental tenet of individual wellbeing. However, more recent studies argue that it is instead wellbeing that has an impact on political participation. Even though political participation is one of the most intensively studied topics in political science, the individual consequences of taking part in political participation have seldom been approached and the relationship with individual wellbeing only recently started to gather some attention. Nevertheless, up until now, very few studies have explicitly addressed the relationships between these two concepts. In this study, we examine the relationship between political participation and individual wellbeing using longitudinal data. We also aim to show how we applied "Survey Quality Prediction" to correct for measurement error and how different results would be if correction for measurement error is ignored.
Question testing is one of the pillars of survey quality and one of the most researched topics in the field. Numerous techniques have been proposed to test and improve questions, but many of the decisions still rely on subjective analysis and expert reviews. SQP is a versatile tool that can help survey question production, systematizing the process of question development and question evaluation. SQP can also be immensely helpful to researchers as a documentation tool, allowing for version tracking and comparison, whether versions are a product of changes across time, across surveys, or across countries and languages. This presentation will illustrate how the systematization and documentation process is helping develop an index of societal wellbeing. It will also address two areas where we think SQP has room for improvement. First, some of the question characteristics to code still rely on subjective judgments (such as the extent to which the question might be subject to social desirability). We evaluate the impact this might have on the questions. Second, the strong focus on question quality in terms of reliability and validity as a means to evaluate the quality of the data to study relationships among variables seems to have silenced the discussion on bias of the estimates. We suggest that there are a number of features that have been shown to correlate highly with bias in the survey literature, and SQP could incorporate some of that knowledge to at least provide warnings to users and suggest possible fixes.