Reflections on Mixed and Multimethod Research 2 |
|
| Session Organisers |
Dr Susanne Vogl (University of Vienna) Dr Andrea Hense (Sociological Research Institute Göttingen) Dr Leila Akremi (German Statutory Pension Insurance Scheme) |
| Time | Tuesday 16th July, 14:00 - 15:30 |
| Room | D26 |
For a few decades now, mixed (qualitative and quantitative) and multimethod research (methods of the same tradition) has become fashionable. Beyond a period of developing design terminologies and discussing epistemological backgrounds, the community is now paying more and more attention to practical issues and a greater variety of methodological combinations. Arguably, there are different ways of „mixing” different research methods in mixed methods or multimethod research. The mixing can occur at different stages of the research process like sampling, data collection, data analysis, and interpretation. This implies that different approaches and data can be integrated at various levels or stages and strategies or data can stem from the same (multimethod) or different (mixed methods) research traditions.
One of the most important problems of mixed methods and multimethod research is how to integrate different approaches and their results to generate “meta-inferences” (Teddlie/Tashakkori 2009: 300). Some solutions for dealing with this “integration challenge“ (Freshwater/Fetters 2015; Bryman 2007) are broadly discussed in handbooks (Cresswell/Plano Clark 2011; Tashakkori/Teddlie 2010), but in practical research applications specific difficulties arise which are not covered by existing methods literature.
In this session, we invite presentations on different research designs in mixed and multimethod research and their purposes. We encourage a critical reflection on strengths, weaknesses, and practical problems of the “mixing” and the solution thereof. Presentations that combine survey research or survey data with qualitative methods are specifically welcome. Furthermore, they can focus on various types of sampling, data collection methods, data, and analytical strategies, e.g. verbal and visual data, big data, surveys, experiments, ethnography, qualitative and/or quantitative observations, network or discourse analysis, text mining, content analysis and so on.
We aim to stipulate a discussion on types of research designs and combinations of different data types in a thoughtful and innovative way. We encourage critical and reflective presentations of research practices in mixed and multimethod research (instead of mere presentations of what has been done or two loosely linked studies) to advance understanding and practice of mixed and multimethod research.
Keywords: mixed methods, multimethod, research design, types of data
Miss Aida Montenegro (University of Bonn) - Presenting Author
Within the current discourse on lectures as passive modes of learning or authoritarian modes of teaching, measuring student engagement in large lecture-based courses (LLBCs) provides insight into the appeal that LLBCs have for students. Moreover, given the large numbers of university students, empirical studies on students’ engagement in large learning environments are important for improving teaching conditions and learning outcomes. This paper explains how quantitative and qualitative data can be integrated within a single study in order to investigate students’ engagement in LLBCs.
To gauge the quality of students’ engagement in LLBCs, this research project utilizes questionnaire items based on observational data, as well as interviews and observations on students’ reactive and proactive forms of engagement. The questionnaire on student engagement was designed for first-semester students in introductory-level sociology courses. In total, 18 items were employed to measure behavioural, emotional, cognitive and agentic engagements. Four groups of students (n=340) at four German universities were asked to indicate strength of agreement on engagement during lectures. The results are discussed in terms of implications for practice and the utility of the methodological approach for evaluating the complexity of students’ engagement in LLBCs. Semi-structured interviews with the lecturers were conducted by exploring four mechanisms of conceptual models, namely: construction, differentiation, reorganization and refinement. The interviews focused on student engagement in LLBCs were revised in order to examine connections and supporting statements related to students’ (dis)engagement, and then compared to the data gathered from questionnaires and observations.
Self-determination and achievement goal theories were used for the theoretical foundation of this study.
Mrs Lorie Dudoignon (Médiamétrie)
Mrs Aurélie Vanheuverzwyn (Médiamétrie)
Mrs Fabienne Le Sager (Médiamétrie) - Presenting Author
Founded in 1985 in response to new demands in the French audiovisual sector, Médiamétrie is now the benchmark when it comes to audience measurement of audiovisual and digital media. Médiamétrie’s main mission is to measure and analyse the behavior of the audience and the trends in the market. In a context of ever-wider media offerings and increasingly complex media consumption modes, Médiamétrie is always on the lookout for new trends to design and develop new methodologies and technologies for audience measurement.
Audience measurement was originally based on panels of individuals. The emergence of return path data has provided the media ecosystem with new sources of data that are comprehensive and available in near real time. Rather than oppose big data and survey data, Médiamétrie has chosen to develop hybrid approaches that mix the different data sources to create a new, richer or more detailed one.
Two types of approach have been developed.
The first one is based on a basic principle in survey sampling theory: the use of any auxiliary information related to variables of interest improves the statistical precision of the results. Big data is considered here as auxiliary information in the same way as Insee census data.
The second approach responds to a different need. Actually, the survey sample may be too small to estimate audience results of small TV channels or websites. Big data then brings the necessary deepness and becomes the reference data and survey data is used as training data set in the construction of a qualification model.
The purpose of the presentation is to discuss the strenghts and weaknesses of each source, especially with regard to the new European General Data Protection Regulation, and to provide an overview of the hybrid methods developed by Médiamétrie since 2012.
Mr Felix Knappertsbusch (Helmut-Schmidt-University, Hamburg) - Presenting Author
The methodological (self-)reflection of mixed methods and multimethod research (MMMR) has produced a new line of research investigating the prevalence of integrative designs. This “prevalence rates literature” (Alise & Teddlie, 2010) has produced estimations of the current proportion of MMMR ranging from 5% in sociology (Alise, 2008) to 14% in educational research (Truscott et al., 2010).
However, few studies investigate the different “styles” (Brewer & Hunter, 2006) of MMMR research practice: What methods and designs are used? Which methodological frameworks are applied? Further investigating the methodological details of integrative research studies is an important advancement of the prevalence literature for two reasons.
Firstly, there is an abundance of methodological frameworks proposed to guide method integration, but their relevance for research practice is not self-evident. Prevalence studies repeatedly find that a significant proportion of MMMR-studies are not labeled as such (Gambrel & Butler VI, 2013; Truscott et al., 2010), which may be indicative of the practical impact of methodological literature.
Secondly, previous research indicates that there are considerable differences regarding the prevalence and “style” of MMMR in different areas. However, systematic comparisons of research-practice between fields remain scarce.
One reason systematic reviews have not advanced beyond prevalence rates is their usually somewhat narrow definition of research practice. The prevalence literature has mainly relied on published methodological accounts, despite research suggesting they provide a fragmented and potentially biased picture.
In line with basic assumptions of MMMR-methodology, I instead propose (1) a three-dimensional conceptualization of research practice, adding researchers’ individual characteristics (methodological beliefs), and the “logic-in-use” of everyday research practice (interaction routines) to the “reconstructed logic” of published accounts.
Secondly, this conceptualization is operationalized through multiple methods in a sequential multi-phase mixed methods design, combining a comparative systematic review of publications, a survey of methodological beliefs among the sampled authors, and qualitative interviews regarding everyday research practice.
Ms Tania Gutsche (University of Southern California)
Mr Bas Weerman (University of Southern California) - Presenting Author
In collaboration with the Consumer Payments Research Center of the U.S. Federal Reserve Bank (formerly based in Boston, now in Atlanta), we have been surveying between one and three thousand Understanding America Study (UAS) respondents each October (since 2015). Respondents are asked to consent each year to participate in a multi-step financial baseline survey followed by three days of payment tracking. The UAS respondents are randomly assigned days from October 1st to October 31st. Though the majority of surveys are completed online, various phone interviews have been conducted as follow up interviews to the diary experience or to question outlying data. In addition, two paper memory aids are sent by FedEx or priority mail each year along with a receipt pouch and FAQ document, as well as multiple email reminders. In most years an additional hurricane effect survey is fielded to those in the hurricane zone, as well as a request for respondents to consent to a soft credit pull. This year we conducted a number of interviews specially addressing the user experience with the study over their three-day cycle.
Our paper will document a history of survey refinements for the baseline, pre-diary and three-day diary surveys, and our sampling and consent process. We will explore findings from our past experience running the survey which included requesting the return of paper diaries to check against online entries as well as running the diary twice in one month. We will share insights gleaned from the post-survey phone interviews and survey comments. Finally we will discuss past experiments with value look-ups and screen display options and explore the challenges of designing a mobile app for future diary studies.