All time references are in CEST
Historical and philosophical reflections on survey sampling 1 |
|
| Session Organisers |
Mr Lukas Griessl (University of Essex) Professor Nick Allum (University of Essex) |
| Time | Thursday 20 July, 09:00 - 10:30 |
| Room | U6-07 |
The practice of survey sampling has a long history. After having been first suggested in one way or another during the 17th and 18th centuries, its ‘official’ history began at the end of the 19th century, when Anders Nicolai Kiær suggested sampling as an alternative to full enumeration and the controversies that it sparked at the International Statistics Institute (ISI) (e.g., Kruskal and Mosteller, 1980; Lusinchi, 2021) during the first third of the 20th century. In the course of several conferences at the ISI, supporters of sampling step-by-step convinced the international community of its advantages. After first being met with disdain from statistical communities, survey sampling evolved to become the standard process to understand populations based on a smaller part. Throughout the 20th century, also after the famous misprediction in the context of the 1936 and 1948 US elections, survey sampling thus became more sophisticated, professional, and widespread and is now a ubiquitous practice in modern societies. In the 21st century, the discipline, however, faces new challenges. Response rates declined and costs increased. New modes of data gathering and analysis through online sources and big data challenge the underlying philosophy of survey sampling. The field of survey methodology thus finds itself amidst fundamental challenges that could benefit from revisiting the history, philosophy and sociology of survey sampling, to help find a way through the present challenges. In this panel, we thus want to explore the historical development of survey sampling, and welcome contributions on, but not limited to, the history of major developments and controversies in the discipline, sociological and philosophical reflections on the practice of survey sampling and possible paths to take in the future.
Keywords: history and sociology of survey research, the past and future of sampling, philosophical considerations, polling
Dr Julian Molina (University of Bristol) - Presenting Author
The British Crime Survey marked an “empirical breakthrough” for understanding patterns of crime and victimisation. Initiated by researchers in 1981, the survey drew upon already existing survey techniques and expertise to address varied policy and political challenges to British crime policy. The survey has subsequently become, in the words of one of its original designers, a “juggernaut mainly concerned with administrative processes.” However, the survey’s early history offers insights into the pragmatic actions taken by policy and research communities to develop a “more realistic picture” of crime through the use of sampling techniques. The paper revisits the origins of the first survey to examine the methodological disputes within the Home Office over the value of sample surveys and police-recorded crime statistics. This involved professional tensions between the Home Office Statistical Department and Research Unit over the use of sampling methodologies in the late 1970s and early 1980s, the culture of research and statistics within government departments, “efficiency reforms” to statistical services which challenged the professional epistemic authority of police statistics, and introduced a customer model for government statistical products. The paper also outlines the role of international networks of survey experts who lent their expertise to support the development of sample crime surveys during the 1970s in Canada, Netherlands, Germany, the USA and the UK.
Dr Tom W. Smith (NORC at the University of Chicago) - Presenting Author
Cross-national, survey research emerged out of and developed along with many of the seminal megatrends of the 20th century including globalization and democratization. It was also shaped in important ways by such major historical events as World War II, the advent of post-bellum collective multilateralism, and the spread and collapse of Communism.
The development of cross-national, survey research is an example of what Rogers (2005) calls the diffusion of innovation. Public opinion polls were created in the United States in the mid-1930s and spread to other countries (Bulmer, 1998; Bulmer, Bales, and Sklar, 1991; Heath, Fischer, and Smith, 2005; Lagos, 2008; Livingston, 2003; Norris, 2009 Oberschall, 2008; Rokkan, 1955; Smith, 2010; Verba, 1993; Zetterberg, 2008). As Verba (1993) has observed, “Survey research has been developed largely in the United States, and has been transferred from there to other western democracies and more recently to developing states,” Like all diffusions, its development and trajectory was innovation specific and was both aided and hindered by the particular characteristics of survey research itself.
Its expansion was part of the more general process of globalization (Heath, Fisher, and Smith, 2005). Of course in the case of survey research, globalization involved considerable interaction between the global product (survey research) and the local markets and cultures. Thus, as Heath, Fisher, and Smith (2005) note, “Globalization of public opinion polls has not entailed a straightforward spread of a standardized ‘product’ throughout the world in terms of survey conduct.”
Additionally, “(t)he expansion of surveys in general and public opinion polling in particular was part of the general growth of democracy within and across societies (Oberschall, 2008).” Surveys in general and public opinion polls in particular typically develop and only thrive in open, democratic societies (Butler, Penniman, and Ranney, 1981). They are rarely allowed in authoritarian regimes and seldom flourish in colonies. Democratization in general and decolonialization in particular opened up more countries to surveys.
Besides being shaped by these overarching megatrends, the development of cross-national, survey research was also influenced by important, historical events. Chief among these were the impact of World War II, the advent of post-war collective multilateralism and the founding of the United Nations, and the emergence of the Cold War and the imposition of the Iron Curtain across Europe.
This paper examines 1) the emergence of cross-national, survey research including the role of early adopters - Gallup, the National Opinion Research Center (NORC), other survey-research organizations, and Public Opinion Quarterly; 2) the initial diffusion of survey research by Gallup, International Research Associates, Inc., and others, 3) foundational survey-research meetings and associations, 4) the impact of World War II, 5) the role of the United Nations and other international organizations including its collaboration with the World Association for Public Opinion Research, 6) the first comparative surveys, and 7) the contributions of international exchanges and immigrations.
Professor Michael Traugott (University of Michigan) - Presenting Author
The importance of polling in the United States at its inception was justified by Gallup and Rae (1940) as a way to achieve a higher form of democracy. A central element of the method’s advantages lay in sampling to represent the views of those who might otherwise not be able to participate in other forms of political expression. Over time, declining response rates and concerns about nonresponse bias have raised questions about such claims of representativeness, as the analysis of data came to rely on various forms of statistical modeling rather than the simple aggregation of expressed views from a random sample of citizens.
At the same time, during a recent period of declining trust in a variety of institutions, trust in polls has declined as well. Research to understand this phenomenon has identified motivated reasoning as one possible explanation – the finding that perceptions of the accuracy or trustworthiness of a specific poll finding is related to whether the results agree with an individual’s own views on the topic. Individuals exposed to results showing a majority of respondents support a view contrary to their own find them less accurate or trustworthy compared to equivalent results that support their personal view. This research suggests a diminished role for public acceptance of two of the main roles of polls in a democratic society – conveying the public’s views to their elected representatives and an interest in learning what their fellow citizens think about important issues.
Dr Maria Mercedes Henriquez de Urdaneta (Universidad de Piura) - Presenting Author
The study exposes the methodological design challenges of research projects to qualify for the Master’s degree in Marriage and Family at a Peruvian University. Furthermore, as an interdisciplinary degree in its study framework and the academic origin of teachers and students, it turns the research design into a complex study where it is necessary to interrelate various disciplines, questions, and research contexts, acting the methodological design as the basis for optimum scientific level.
Two critical considerations were added to the experience: the first is that it was the first master’s degree in marriage and family-approved in Perú, so there were no previous experiences, being the case reported the first promotion. The second is that the projects began their development before the pandemic, having the additional challenge of redesigning some methodological frameworks. The Peruvian case is worth highlighting because it was one of the countries with the most extensive mandate to dictate and develop all academic activities 100% online for two years (2020 and 2021), fully returning to regular activities in mid-2022.
In this way, the work summarizes the methodological challenges, including considerations for instrument design and data collection techniques in quantitative, qualitative, and mixed models. In the same way, it contemplates planning strategies for the accompaniment and training of teachers and students, considering that some researchers were vulnerable by dealing with sensitive issues and being in a pandemic context.
The work uses the anthropological theories of Polo (2015, 2016, 2017, 2018) and Sellés (2013, 2011), which allow considering the researcher as the center, generating a triangulation design that helps adapt the project, using operationalization of concepts as a primary tool for its practical measurement design, which impacted that more than 80% of the projects completed successfully, an achievement never seen before in the institution.
Dr Jon Krosnick (Stanford University) - Presenting Author
This session will occur at the perfect time in the evolution of modern survey research methodology, to help researchers grapple with an unfolding controversy regarding how to produce accurate survey measurements while minimizing costs. For decades now, loud voices have advocated for the abandoning of random sampling and the abandonment of telephone interviewing, which they say are dead methodologies, despite their success and centerpiece visibility in the field for many prior years. The newly advocated methods are essentially convenience sampling with demographic weighting, a method that was tried and abandoned 80 years ago as unreliable, and nearly entirely automated internet data collection. This presentation will review the body of existing evidence on the accuracy of these various methods in attempting to forecast the outcomes of American elections in recent years, as well as an in depth investigation of the accuracy of one of the least expensive and most visible modern methods: river sampling. The conclusions show that long-standing, tried and true methods (random sampling and telephone interviewing) remain reliable, whereas the abandonment of these methods so far has been yielding notably less accurate results and compromising the public's perceptions of the accuracy of survey findings.