Benefits and Challenges of Open-ended Questions 2 |
|
| Chair | Dr Evi Scholz (GESIS ) |
| Coordinator 1 | Mrs Cornelia Zuell (GESIS) |
The present study investigates the effects of respondent and survey characteristics on the response quality to an open-ended attitude question administered in 29 web surveys. In general, open-ended questions provide rich analytical potentials but answering them often imposes a high burden on respondents. Due to the self-administration, web surveys with open-ended questions have to be designed carefully to obtain survey data of satisfactory quality. Identifying relevant characteristics at both the respondent and the survey level should effectively support researchers in designing web surveys that generate high-quality responses to open-ended questions. Relying on satisficing theory as an overarching theoretical framework, we derived several hypotheses on how respondent and survey level characteristics as well as their interactions might affect the quality of responses to open-ended attitude questions. Satisficing theory provides us with theoretical mechanisms that link characteristics of questions and respondents with the use of response strategies which negatively affect response quality. For example, it suggests that if a question’s difficulty is high and if a respondent is low in ability and/or motivation, the respondent might decide to use satisficing as a response strategy (Krosnick, 1991, 1999). Applying this response strategy to an open-ended attitude question might result in non-interpretable responses, such as answers which do not correspond to the question, “don’t know” answers, or nonsense answers. To test our hypotheses, we draw on pooled data from 29 web surveys which were conducted between 2009 and 2015. Key questions were repeatedly asked in each survey and covered topics such as political attitudes and behaviors as well as socio-demographics. For our analyses we selected a cognitively demanding open-ended attitude question which asked for the most important problem Germany was facing at the time of the survey. Further, we measured the response quality with three different indicators: response length, response latency, and the interpretability of the answers. Applying multilevel analyses to the pooled data, we examined the effects of respondent and survey level factors on the three indicators of response quality. The findings suggest that characteristics at both levels as well as cross-level interactions influence the response quality. In particular, our empirical results supported the assumption that motivated respondents and those high in ability provided higher quality responses to open-ended attitude questions. Further, by including cross-level interactions in our models we showed that factors on different conceptual levels were not completely independent in affecting response quality. Showing that there were significant differences in the effects of questions’ position between respondents with low and high interest on giving substantive responses highlights the importance of considering respondent’s motivation in designing web surveys. In addition, our results indicate that analyzing open-ended questions’ response quality exclusively with single indicators, for instance the response length, might convey a misleading picture. Including content-related indicators such as the interpretability of responses provided us with more differentiated insights compared to the exclusive use of process-generated measures of quality.
A major problem of urban neighbourhoods, especially deprived ones, is to explore the normative structure or the acceptance of norms. Deprived neighbourhoods in particular are characterized by various forms of deviant behavior, ranging from loud noise to physical aggression. Our paper addresses this problem: the acceptance of norms and reactions to deviant behavior. Backed up by the definition of situations by Goffman, we try to understand which dynamics influence the decisions of acting within situations. The innovative method is to present a series of conflict situations each followed by a set of open-ended questions.
The theoretical reasoning is a rational choice stating that the actors’ propensity to sanction will vary by the perceived cost of the action, and these, in turn by the perceived attitude and sanctioning willingness of other residents. The more other residents are assumed to share the actor’s attitude, the higher is the willingness to sanction. Perceived legitimation thus is a major determinant of costs.
We choose a two-step design, with an integrated perspective: First, we present data of the perception of situations of residents of two lower-middle class neighborhoods in a face-to-face survey (N = 376). Second, we report the results from N = 1,557 structured participant observations within the public space in a large deprived housing estate N=1,557 structured participant observations. The findings substantiate the assumptions and indicate in case of low perceived consensus (or heterogeneous acceptance) a withdrawal from the neighborhood. Moreover, the situation method proves to be a valid instrument. It requires open-ended questions to elicit the different aspects of norms: the views of the actor, her/his perceptions of the other residents’ views and reactions.
Young Life and Times (YLT) is an annual cross-sectional postal survey of 16-year olds undertaken in Northern Ireland since 2003. It is one of the longest-running surveys of this kind in Britain and Ireland. YLT is run by ARK – a joint initiative between the two universities in Northern Ireland and widely used by government and voluntary sector organisations to monitor policy and young people’s attitudes on a wide range of issues.
Open-ended questions have always been a feature of the YLT surveys. Every year YLT offers 16 year olds an opportunity to expand on specific issues covered in that particular year. These have included various topics such as mental and sexual health, education, culture and so forth. However, one issue remains constant, that of community relations, with an open-ended question included every year since 2003. As a result, we have data on attitudes to cross-community relations in Northern Ireland spanning a 14 year period.
In 2003 Northern Ireland was emerging from over 30 years of conflict. A lot has happened since then ensuing quite dramatic political, economic and social change. In political terms, intermittent periods of divisive events and deadlocks have threatened to destabilise the peace accord. In this presentation we focus on how this series of responses has benefited our understanding of the long-term quantitative trend data. We discuss the challenges of managing the data. This includes the impact that external events exercise on the nature of responses to open-ended questions and the importance of appreciating the contextual setting during which the data were collected.