Response format & response behaviour |
|
| Chair | Dr David Richter (DIW Berlin ) |
Question order effects refer to the phenomenon that different orders in which questions (or response options) are presented may influence respondents’ answers in a systematic way. Past empirical research documents considerable order effects, casting doubts on the general validity of survey reports. In a classic example cited by Schuman & Presser (1981), questions regarding freedom of the press are answered very differently if the order of the two countries mentioned in the questions was switched (first question on freedom of the press in the USA, second question on freedom of the press in Soviet Russia vs. first question on freedom of the press in Soviet Russia, second question on freedom of the press in the USA). Another frequently cited example of order effects are questions on satisfaction with life in general and satisfaction in different life domains (Strack, Martin, & Schwarz, 1988).
However, experimental studies conducted in psychological laboratories are usually designed and tweaked to bring about and maximize effects. This practice can be problematic and is seen as one reason for the rather low replicability of psychological studies (Open Science Collaboration, 2015). Consequently, there is increasing skepticism about whether comparable effects – in prevalence and magnitude – emerge if data is collected in real-world settings.
For our analyses, we draw on three large panel surveys –the German Socio-Economic Panel Study (SOEP; 28,000 respondents in 16,000 households; waves 2005-2015) and its related studies, the SOEP-Innovation Sample (SOEP-IS; 5,500 respondents in nearly 3,500 households; waves 2011-2015) and the Program for the International Assessment of Adult Competencies (PIAAC-L; 6,200 respondents surveyed in 2014), in which the order of questions in the questionnaire often changes in an essentially random fashion over time. This may be due to the implementation of new survey items, rotating modules of the questionnaire, or changes in filter questions. Our analysis focuses on survey questions on attitudes, beliefs, and opinions, which have been shown to be particularly plagued by order effects.
Initial analyses showed that distributions, means, and standard deviations of the responses to the various questions were highly comparable and almost identical across the different panels and across different survey years. Question order effects were non-existent or trivial. Even variations that could be assumed to yield strong effects showed only small effects: first question on ‘loneliness’ asked after question on ‘hours of sleep per day’ vs. question on ‘number of friends’: d = 0.45, first question on ‘locus of control’ asked after question on ‘risk aversion’ vs. question on ‘self-esteem’: d = 0.34.
We conclude that the data collection in household surveys is robust with regard to question order effects and that the existing literature reporting strong effects in (social-psychological) experiments should be reexamined. Future studies should further examine our initial results and vary question order in household panel surveys systematically within the panel study and the same survey year.
The paper aims at considering a new technique for price sensitivity data collection using complex laddering designs. Price sensitivity data collection methods (Stoetzel, 1954; Gabor and Granger, 1961) are crucially important for market research but are also too rudimentary compared to the progresses happened in data collection tools. These methods can be upgraded and can also be cleansed by a questionable operational and theoretical revision: the Price Sensitivity Meter (PSM) introduced by van Westendorp (1976). Indeed PSM is long far away from being a scientifically grounded technique not only for the scenic but meaningless criss-cross of lines but because of a serious problem of conceptual indetermination of the "economic" and "expensive" price levels, in fact they are simply subjective ranges and not points dividing areas.
The present research underlines the importance of measuring the ceiling reservation price, the intermediate reservation price (novelty) and, when it is applicable, the floor reservation price. The psychological aspects and processes that make it difficult the measuring purpose are thoroughly analyzed: acquiescence, problem of comparisons, artificial setting, overestimation and also underestimation of price sensitivity, use of an answer scale for data collection. It is then argued that laddering is a good proposition to explore price sensitivity even if it suffers of a first price level effect.
The research proposes the Complex Laddering Path (CLP) as an alternative model for price sensitivity data collection, that is able to reduce the bias. CLP is based on intelligent paths that use previous answers deciding what price level is asked in the second, third and fourth step. Paths are programmed to change direction upward and downward at the end of the sensitivity exploration for a higher accuracy overcoming the original limit of a small set of price levels. Finally this technique could be used not only for price sensitivity but also for measuring similar quantitative variables (e.g. quantity sensitivity).
In Germany, working time conditions are a hot topic in both political and economics debates (BMAS 2015). One big problem is the accurate capturing of actual work hours and preferred work hours. The estimation of over- and underemployment in Germany is usually based on data from the Mikrozensus (Ehing 2014: 4) or the Socio-Economic Panel (SOEP). However, the results differ significantly and may lead to different political conclusions. A recent publication of the Statistisches Bundesamt (Rengers 2015) which is based on the Mikrozensus, points out that for 9.3 percent of the overall labor force, actual work hours and preferred work hours differ. The majority would prefer an increase in their actual work hours. In contrast, estimations based on SOEP data (Holst/Wieber 2014) show that 71 percent of all employees face a mismatch between actual and preferred work hours, with the majority preferring a reduction in their working hours.
The present paper examines in a stepwise approach possible causes for these substantially diverging findings. Replicating the Rengers (2015) study using SOEP data, i.e. accounting for the operationalization as well as the structure of the work force, reveals only a little convergence. The "Families in Germany" (FiD) subsample of the SOEP contains similar questions to the Mikrozensus concerning the desired working time. Using this data indicates that divergences in the questionnaires of both the SOEP and the Mikrozensus underly such differences. We assess that overemployment is underestimated in the Mikrozensus. The SOEP – on the other hand - rather provides an upper limit of the rate of employees with working time discrepancies. Regarding the questions on desired working time in both surveys further basic methodical research is needed about to what extent (1) income consequences of working time changes are really taken into account; (2) future perspectives are included when answering, and (3) which anchor points are chosen: the contracted or the actual working time.
Literature:
Bundeministerium für Arbeit und Soziales (BMAS) (2015): Grünbuch „Arbeiten 4.0“. In:
http://www.arbeitenviernull.de/dialogprozess/gruenbuch.html [last access 07.April 2016]
Ehing, Daniel (2014): Unter- und Überbeschäftigung in Deutschland: Eine Analyse wesentlicher Zusammenhänge für die Unterbeschäftigung in Teilzeit, Zeitschrift für Sozialreform, Nr. 53, S. 247-272.
Holst, Elke/Wieber, Anna (2014): Arbeitszeit und Erwerbstätigkeit: Bei der Erwerbstätigkeit der Frauen liegt Ostdeutschland vorn, DIW Wochenbericht, H. 40, S. 967-975.
Rengers, Martina (2015): Unterbeschäftigung, Überbeschäftigung und Wunscharbeitszeiten in Deutschland. Ergebnisse für das Jahr 2014. In: Statistisches Bundesamt (Hrsg.): Wirtschaft und Statistik, H. 6, S. 22 – 42.
ELIPSS is an online survey instrument for the scientific community; it allows researchers to innovate both in terms of content and of method. Among the recent surveys, the “Mobilities and spatial experiences over the life course (Mobilities)” one proposes a new way for outlining national and international mobility behaviors. To achieve its ambitious goals, ELIPSS devised an original design strategy based on the use of maps. In the present paper we will present the rationale for the survey design and critically address its results.
The Mobilities project proposes a new theoretical construct, the ‘space-set’, to better account for behavioral and attitudinal differences in an age of increased spatial mobility. Space-sets designate the entanglement of geographical places where individuals spend or have spent their social life. They are composed by people’s objective and subjective memories and experiences stemming from past and present practices of spatial mobility. A space-set can also be conceived as a network qualified by its structural features: size, range and focus.
To operationalize the space-set concept, a survey has been conceived and administered to the probability-based ELIPSS panel. For its pilot phase, the panel sample was composed of 1,000 individuals representative of the French population aged 18-75. The panel members have been equipped with a 7’ tablet and with a 3G subscription to answer a monthly questionnaire.
To make space-sets emerge, part of the survey consisted in asking the panelists to identify the places they visited and thus constitute an exhaustive networked catalogue. Some major limitations and constraints due to the ELIPSS panel specificities played a major role in designing the interface. Besides the 30 minutes maximal duration of the survey and the heterogeneity in digital proficiency of the panelists, a main constraint was their geographical knowledge. To overcome these limitations, we leveraged on the visual appeal of a map: the interface invited the users to select the French national departments (for nation-wide space-sets) and the countries (for world-wide space-sets) visited.
By adopting this strategy we tried to mitigate the memory efforts demanded to the user, leaving open to him/her the way to elicit the places he/she visited during his/her life. The respondent could select directly on a map the visited places and was asked to characterize them afterwards. This response format has the advantage of bringing playfulness in the survey settings, creating a more robust engagement of respondents.
To fully address the efficiency of our map based enquiry method, we will present and discuss, in the last part of the paper, the navigation and behavioral data we collected. For example, we extensively followed the user journey in managing the interface: from the use of features like the search engine for retrieving a specific country to the collection of trials and errors. Furthermore, a qualitative analysis of users’ comments has been carried out.
The discussion we are proposing can be valuable for future research projects and interface implementations of innovative quantitative survey.