Different Methods, Same Results? – How Can We Increase Confidence in Scientific Findings 1? |
|
| Session Organisers |
Dr Thorsten Kneip (Max Planck Institute for Social Law and Social Policy, MEA) Dr Gerrit Bauer (LMU Munich) Professor Elmar Schlueter (Justus-Liebig-University Giessen ) Professor Jochen Mayerl (Chemnitz University of Technology) |
| Time | Friday 19th July, 09:00 - 10:30 |
| Room | D12 |
This session follows up on earlier discussions at last year's ESRA conference on the fruitful use of multiple methods. We are interested in how to increase confidence in scientific findings in the light of mixed evidence on the one hand and busted seemingly established findings on the other ("replication crises"). While we have seen an ever increasing proliferation of methods for survey data collection and analysis in recent years, there is still a lack of standards in how to aggregate findings. Too easily, convergence of findings is seen as indicative for a "true" effect, while it may well reflect repeated systematic errors. While replicability and reproducibility are fundamental for empirical research, replication studies may not only confirm true but also false results. In a similar vein, diverging results when using different methods are often to be expected, as they aim for the identification of different effects or rely on different assumptions, the violations of which lead to different forms of bias.
The common problem seems to be rooted in a lack of awareness and transparency regarding implicit decisions made in the process of analysis and the lack of explication and discussion of model assumptions. We invite researchers to submit papers discussing the consequences of applying alternative methods of survey data analysis addressing the same research question. A focus should be given on making explicit all assumptions related to the chosen method(s). Examples would be:
Studies comparing at least two different estimation approaches, addressing different potential sources (and directions) of bias; extensive robustness checks varying theoretically undetermined parameters (e.g. functional form of control variables, definition of analytic sample); replication studies critically reflecting or challenging decisions made in the entire research process; crowd research.
Mr Thomas Hinz (University of Konstanz) - Presenting Author
Mr Thomas Wöhler (University of Konstanz)
In many countries, central registers from which researchers are able to draw random samples of the resident population simply do not exist. In order to reach a random sample of the population, often a complicated and costly cluster sampling based on cooperative municipalities is used. Meanwhile, some municipalities charge immense fees for such random samples which are, however, still required for a many survey studies. In our paper, we test a geo-sampling strategy that is common in environmental, biological or geological sciences as an alternative to come up with a random sample of residents in two smaller cities in Germany. We discuss the statistical features of geo-sampling, describe how the strategy can be applied in practice and evaluate results from our empirical application. The evaluation is based on a comparison of geo-sampling and register samples.
Ms Jill Darling (University of Southern California) - Presenting Author
Dr Arie Kapteyn (University of southern California )
This paper provides new information regarding an innovative method of pre-election polling. We used probabilistic methods with an internet tracking poll to forecast the “generic” (national major party) vote in the 2018 U.S. Midterm Election. We also measured vote using a traditional categorical method for comparison. We previously tested probability-based forecasting in the election for U.S. President in 2016 without the categorical comparison. Probabilistic polling (Delavande & Manski, 2010) provides an alternative to traditional polling methods. This approach asks respondents to provide a percentage likelihood of voting for each presidential candidate, well as likelihood of voting. In a series of surveys that began in fall of 2017, respondents who were members of the USC Center for Economic and Social Research’s Understanding America Study internet panel answered questions about voting in the midterm elections. They answered both a probability vote question and a traditional categorical vote question, with order randomized. We obtained votes cast in the election via a post-election survey among the same respondents. This work is part of an ongoing exploration of the utility of methods that may help address problems facing the field of election polling, so we have made our microdata and methods available to other researchers for analysis. Probability methods successfully predicted the 2012 outcome (Gutsche, Kapteyn, Meijer, & Weerman, 2014; Kapteyn, Meijer, & Weerman, 2012). The method also accurately modeled the outcome of the unusual 2016 Presidential election, after adjustments to the panel's weighting procedures made post-election. This presentation focuses on a longitudinal analysis of self-reported estimation of likelihood of voting in 2016 and 2018, and a comparison of the predictive power of probability and categorical vote estimates tested in 2018. We will also touch on lessons learned, potential contributions to the field of election polling, and next steps.
Dr Marcin Kocor (Jagiellonian University in Krakow) - Presenting Author
Dr Szymon Czarnik (Jagiellonian University in Krakow)
The social phenomenon of over/undereducation and over/underskilling is one of the most popular topics in research on labor market problems. For the last few decades, those problems were investigated with regard to casual mechanisms and consequences of labor market mismatches. However, a recent comparison of results from different countries and studies, reveals great disparities between particular estimations of overeducation or overskilling. The main reason behind the observed differences is the lack of common approach to the measurement of mismatches. Hence, in our paper we describe various methods used to estimate overeducation and overskilling, highlighting their particular advantages and disadvantages, based on relevant studies. Finally, we want to propose a method which provides more precision and which is more intuitive in an application.
Mr Diego Montano (Ulm University) - Presenting Author
Absenteeism is one of the most relevant indicators of organizational performance. Important antecedents of absenteeism rates are illnesses, working conditions, tasks definitions, and organizational structures and processes. In large population surveys, information on absenteeism is collected by asking individuals the number of days they were absent from work for reasons of health problems. However, these type of open-ended items seeking numerical answers are prone to rounding and scale contraction effects. Previous findings have suggested that the inaccuracy of absenteeism rates is associated with cognitive processes related to uncertainty in the internal representation of the numerical value, and/or difficulties in mapping the numerical value onto a numerical response. Consequently, the observed distribution of responses is expected to deviate from theoretical distributions of count random variables. In addition, different model specifications and assumptions are also expected to have an impact on the magnitude of the parameters of interest in empirical research. Hence, the aims of this presentation are to assess (1) the extent to which different model specifications modify the parameters of interest, and (2) how the statistical analysis may take into account the cognitive processes biasing the absenteeism rates.
These aims are investigated by using data from the European Working Conditions Survey. As an illustrative example, the presentation will focus on how the associations between illness-related absenteeism and health problems are influenced by different model specifications. The statistical analysis will be performed by means of hierarchical Bayesian regression models estimated under four different distribution assumptions: logistic, Poisson, zero-inflated Poisson, and hurdle Poisson. The random effects part of the models will include a nested structure of individuals within European Union regions (NUTS 2) and countries.