Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Natural Experiments in Survey Research |
|
| Convenor | Dr Henning Best (GESIS - Leibniz Institute for the Social Sciences) |
| Coordinator 1 | Dr Gerrit Bauer (LMU Munich) |
Experiments are generally regarded as the royal road to causal inference. Yet, social science research often cannot make use of research designs based on randomized laboratory experiments. This is, in part, due to the very nature of social inquiry, which generally is concerned with society. Consequently, critics blame the (alleged) low external validity of lab experiments in the social sciences. Natural experiments can help to reduce these problems as they are set in a real societal context, and external validity can be enhanced. They do, however, face serious problems as well: endogeneity, insufficiencies in standardizing treatment- and control conditions, and self-selection into study- and control group. Advances in data analysis have tackled these problems, and methods such as IV-regression, conditional fixed-effects models and propensity score matching help in identifying unbiased treatment effects.
In this session we are particularly interested in papers on identification of treatment effects in natural experiments, research combining surveys with natural-experimental designs, papers that employ multiple methods of treatment estimation, and innovative ways to design or analyze natural experiments in cross-sectional and especially panel surveys.
The American Community Survey (ACS) was launched following the 2000 Census to replace the decennial census long-form that administered a detailed survey to approximately 15% of all U.S. households. The ACS interview obtains similar information about socioeconomic status, well-being, migration, and other topics from approximately three percent of U.S. households each year, and has been in the field since 2005. In contrast to the once-every-decade survey that the decennial census long-form represented, the ACS is fielded continuously. This design allows researchers to use ACS data to examine the effects of various unanticipated events, policy changes, and other natural experiments. In this paper, we provide an introduction to the ACS, and describe how it can be used to assess the causal effects of unanticipated events. We focus on one specific example: using the ACS to evaluate the effects of Hurricane Katrina on the population of New Orleans. The design of the ACS allowed us to identify a representative sample of people interviewed in the year following Katrina who resided in New Orleans before Hurricane Katrina, regardless of where in the U.S. they were when interviewed. We improved the comparability between this sample and an independent ACS-based pre-Katrina sample of city residents using a propensity score weighting procedure that allowed us to assess causal effects of the disaster using marginal structural models. We used this approach to assess the effects of Hurricane Katrina on labor market behavior, health status, and migration. We found major negative effects of the disaster on these outcomes.
Despite the extensive literature on propensity score (PS) methods there are still several open questions for their implementation. Based on the results of an extensive simulation exercise, we try to address some of these questions and provide guidelines for applicants.
The first question we consider is what should be the preferred method to estimate the PS. We compare machine learning techniques (MLT) with standard logit models. Using Monte Carlo simulations, Setoguchi et al (2008) found that in the estimation of the PS, Neural Networks outperform other MLT (Cart, Pruned Cart, etc.) and logistic regression in terms of bias reduction of the ATT. Lee et al (2009) found that Boosted Regression outperforms other MLT (Cart, Pruned Cart, Random Forest) and logistic regression. The results in the two papers are only partially comparable because outcome of different nature have been used and the estimated propensity scores were used differently: Setoguchi et al (2008) used a matching technique, while Lee et al (2009) considered a weighting estimator. Moreover, the best performing techniques were not used in both papers. By focussing on matching techniques, the first goal of our paper is to let the same technique compete in the same setting. In particular, we assess the robustness of different MLT to the inclusion of irrelevant covariates (i.e., variables that are not confounders). Finally, we assess the efficacy of several measures of balancing as a tool to predict the quality of the propensity matching estimators in terms of ATT bias reduction.
Previous research has found income inequality and religiosity are positively associated across nations, where religiosity is measured using surveys of people's attendance at services and self-reported beliefs. Studies highlighting this association have interpreted it as evidence that inequality fosters religiosity. There is also good reason to think, however, that religiosity contributes to inequality; the association could thus be due to a causal relationship running in either direction. This paper presents the results of two empirical analyses testing more robustly whether inequality genuinely causes religiosity.
First, I use Easterly's (2007) instrument for inequality: the proportion of a country's arable land suitable for wheat relative to the proportion suitable for sugarcane, given that wheat production has historically been organised in smaller farms and sugar production in plantations, leading to divergent patterns of land concentration. Using this instrument, I rule out the possibility of any reverse-causality, and find broadly consistent evidence that inequality does indeed lead to religiosity.
Second, in longitudinal analyses, I find that recent changes over time in inequality have been correlated with changes over time in religiosity. However, the enduring cross-national differences in inequality rooted in land suitability for wheat versus sugar production are not related to recent rates of change in religiosity.
Historically rooted differences in countries' levels of inequality have therefore shaped countries' relative levels of religiosity, but are no longer the sources of countries' differing rates of secularization today.
As a consequence of the ongoing theoretical exchange between social movement and organization theory, Fligstein and McAdam (2011, 2012) developed the unifying concept of strategic action fields (SAF). SAFs allow for a more complex understanding of changing organizational fields in a multilayered system. However, this adolescent approach lacks broad empirical testing of central assumptions and needs further elaboration - a fact being even more important for parts of the concepts concerned with rare occurring events such as field crisis.
German reunification in 1990 represents a natural experiment of merging SAFs into an existing system after a field crisis: Organizational fields of eastern Germany are put into a set of dominant western SAFs, while their western counterparts stay in their familiar web of SAFs. Hypotheses provided by the SAF approach are tested by using data of the Establishment History Panel (BHP) from 1972-2010 (east: 1992). The method of this paper is presented as an innovative way to analyze the effect of merging different SAFs together: Determinants of organizational failure are estimated for both German regions. In a second step, predicted probabilities are calculated for different settings of organizational characteristics for East and West Germany. Finally, the treatment effect of merging SAFs is discussed as east-west discrepancies of predicted probabilities for each setting.