Item design and item importance |
|
| Session Organiser | Dr Anne Elevelt (CBS) |
| Time | Friday 9 July, 13:15 - 14:45 |
Recognising the detrimental effect of the measurement process on data quality, researchers focus on reducing the negative effect of questionnaire design on measurement bias in probability surveys. The issue of measurement quality is essential, especially for large-scale cross-national comparative surveys; however, even in nationally-oriented representative studies, measurement quality consumes a lot of attention.
In this session, the authors will focus on two issues: (1) how the design of particular items impacts survey responses, (2) how removing less important items boost measurement quality. Referring to the first issues, the authors will consider the effect of the scale length and its labelling on respondents understanding of the questions. In turn, the papers focused on the second issue will show how cutting the survey length can reduce the respondent burden and improve data.
Keywords: questionnaire design, measurement error, numerical scales, respondent burden, survey length
Dr Gianmaria Bottoni (European Social Survey HQ, City University of London) - Presenting Author
Dr Eva Aizpurua (European Social Survey HQ, City University of London)
Professor Rory Fitzgerald (European Social Survey HQ, City University of London)
Previous research has shown the importance of nonverbal cues and visual layouts in terms of their impact on how respondents interpret and answer survey questions (Tourangeau, Couper, & Conrad, 2007; Toepoel & Dillman, 2011). In this study, we investigate the effect of numerical values on responses to five questions measured using 11-point bipolar scales. These questions asked for evaluations of the impact that climate change will have, generally, and on different groups (e.g., people in your country, people in developing countries). Respondents were randomly assigned to scales ranging from 0 to 10, or from -5 to 5. The scales were presented horizontally and had identical verbal endpoints anchored as ‘extremely dissatisfied’ and ‘extremely satisfied’. The experiment was embedded in a cross-national face-to-face survey (CAPI and paper) conducted in Hungary (n = 948), Portugal (n = 1,200) and the UK (n = 1,276) in preparation for Round 8 of the European Social Survey (2015-2017). Consistent with studies conducted among single populations (O’Muircheartaigh, Gaskell, & Wright, 1995; Schwartz et al., 1991; Tourangeau et al., 2007), we anticipate that -5 to 5 numerical scales will result in more negatively skewed distributions than those ranging from 0-10. However, considering differences in response styles, we expect those effects to differ across countries. In addition to differences across countries, we investigate differences between the CAPI and PAPI surveys. Beyond the distributions and average scores of the five manipulated items, we analyse their predictive validity by experimental conditions. The implications of these findings and how they fit with previous work will be discussed.
Mr Farsan Ghassim (University of Oxford) - Presenting Author
Survey researchers use a variety of response scales: four labeled points, scales from 1 to 10, thermometers from 0 to 100, and so on. This paper explores six-point answer choice scales as a potential new standard for survey research in domestic and international contexts. Theoretically, a six-point scale has several advantages: First, in contrast to often used binary options or five-point scales, the six-point scale allows for better differentiation between slight, firm, or strong agreement with each side of a bipolar scale, e.g. support or opposition. Second, the six-point scale does not overwhelm respondents with too many options, which would reduce instrument reliability and validity, e.g. as in the case of 100-point scales which feature an unnecessarily high amount of choices. Third, in contrast to seven-point scales prescribed by leading survey methodologists, the six-point scale does not feature a middle response. It thus does not induce survey participants to satisfice, reduces the number of answers that are effectively equivalent to no-opinion responses, and eschews issues with subsequent dichotomization, i.e. when analysts want to collapse the various responses into two categories. Fourth, compared to even-numbered response scales with more than six choices, six-point response scales make the lack of a middle response evident to respondents. For instance, while many respondents may fail to realize the lack of a middle response on ten-point scales (i.e. erroneously treating five as the middle), such mistakes are rather rare with six-point scales, where it is clear that there are three points on each sides of a bipolar scale. Fifth and finally, in contrast to response scales with more choices, the six-point scale generally permits intuitive verbal labels in various languages, rendering it suitable for multinational survey research. In order to test the theorized benefits of six-point scales over common alternatives, this paper will report the results of online survey experiments that are scheduled to be conducted on nationwide samples of citizens across Africa, the Americas, Asia, Europe, the Middle East, and the Pacific region in February 2021.
Ms Ai Rene Ong (University of Michigan) - Presenting Author
Mr Robert Schultz (University of Michigan)
Ms Sofi Sinozich (University of Maryland)
Professor Brady West (University of Michigan)
Professor James Wagner (University of Michigan)
Dr Jennifer Sinibaldi (National Center for Science and Engineering Statistics)
Dr John Finamore (National Center for Science and Engineering Statistics)
Large-scale, nationally representative surveys serve many vital functions, but these surveys are often long and burdensome for respondents. Cutting survey length can help to reduce respondent burden and may improve data quality but removing items from these surveys is not a trivial matter. We propose a method to empirically assess item importance in national surveys and guide this decision-making process using different research products produced from such surveys. This method is demonstrated using the Survey of Doctorate Recipients (SDR) as a case study. The SDR is a biennial survey administered to a sample consisting of 80,882 individuals with a Science, Engineering, and Health doctorate. The survey is administered in three modes: mail, web, or telephone. We used three main sources to assess the usage and importance of SDR variables: 1) Congressional mandates to the sponsoring agency (the National Center for Science and Engineering Statistics) on reporting, 2) a literature review of academic research using the SDR, and 3) frequency counts of data table requests made via the SDR website. Our research team coded the bibliography for SDR variable usage and citation count. Putting this information together, we are able to identify 35 items (17% of the survey) that had not been used by any of these sources and are likely candidates to be considered for dropping or collapsing. To assess the burden of response on these same items, we use web survey paradata to calculate mean question response times and question-level break-off rates. We conclude with general recommendations for those hoping to employ similar methodologies in the future.
Dr Thorsten Heien (Kantar) - Presenting Author
Dr Dina Frommert (Deutsche Rentenversicherung Bund)
Mr Marvin Kraemer (Kantar)
Mrs Anne Langelueddeke (Deutsche Rentenversicherung Bund)
There is a new need to gather information on pension entitlements from different sources because of the now widespread multi-pillar nature of pension systems. However, getting valid information on pension entitlements is tricky. Often, respondents are not sure of the value of the entitlements. To get more reliable information it is thus common to ask respondents to refer to official documents such as annual pension information letters. But with this approach, several problems can arise which lead to missing or incorrect information: Not every pension provider sends out information letters or respondents might have lost the information letters. If an information letter can be consulted, it might be outdated and reflect pension entitlements from several years ago. But with respect to economic data like income, insurance fees, or pension contributions and entitlements, the age of the information is essential.
In the survey on “Life courses and old-age provision” (Lebensverlaeufe und Altersvorsorge; Lea) respondents were asked to provide current entitlements for every pension product they held. Additionally, they were asked to provide the year of issue if they used an official information letter as reference. The proposed paper will analyse the effect of using outdated documents by simulating the actual or “true” entitlements through adding further contributions and roughly adjusting for capital gains. Next to the overall effect, its magnitude for different socio-demographic groups is presented to identify possible structural distortions in the reporting of pension entitlements based on survey information.