Failures in Survey Experiments 2 |
|
| Coordinator 1 | Dr Kristin Kelley (WZB Berlin Social Science Center) |
| Coordinator 2 | Professor Lena Hipp (WZB Berlin Social Science Center/University of Potsdam) |
What can failures and unexpected results tell us about survey experimental methodology?
In recent years, social scientists have increasingly relied on survey experiments to estimate causal effects. As with any experimental design, survey experiments can fail or yield results that researchers did not anticipate or preregister. Researchers find themselves in situations with null, unexpected, inconsistent, or inconclusive results and must then decide whether results reflect on the theory being tested, the experimental design, or both. Usually, the insights gained from such failures are not widely shared, even though they could be very useful to improve the design of experiments, quality of research, and transparency.
We propose a session in which scholars present insights from survey experiments that failed or led to unexpected results. We believe that sharing the design and results from failed survey experiments, carefully considering their possible flaws, and talking about unexpected findings is useful to the development of theory (e.g., identifying scope conditions) and methods, and contributions to the transparent research practices.
We invite contributions that address the following: What’s the appropriate response to a “failed” experiment or to deal with unexpected results and null findings? More specifically, the objective of the session will be to reflect on the definition of a “failed survey experiment” (e.g., null findings, unexpected findings, problems during the conduct of the survey experiments, findings that contradict field-experimental evidence and theoretical/pre-registered predictions). We will discuss why some experiments fail (e.g., poor treatments/manipulations, unreliable/invalid dependent measurements, underdeveloped theory and/or hypotheses, underpowered), whether and how to interpret and publish results from failed survey experiments, what was learned from failed survey experiments, and recommendations for survey experiment methodology.