Mixed Methods and the Limits of Social Research 2 |
|
| Chair | Dr Andrea Hense (Sociological Research Institute Göttingen (SOFI), Germany ) |
| Coordinator 1 | Professor Udo Kelle (Helmut Schmidt University Hamburg, Germany) |
| Coordinator 2 | Dr Felix Knappertsbusch (Giessen University, Germany) |
Previous research with mixed-methods designs has shown several alternatives how to combine qualitative and quantitative approaches. Nevertheless, the debate needs to focus on precise, functional and application-orientated advice to mix both types of data. We intend to contribute to this issue by discussing one way to combine unexpected results from quantitative research with qualitative interview material. In this regard, we would like to present chances and difficulties which occur when displaying these results in segment or quote matrixes.
Our approach is based on Leech und Onuewegbuzie (2009) and is most similar to a mixed concurrent dominant status design with a dominant quantitative strand. We use the panel study “Labour Market and Social Security” (PASS) as quantitative data source and 12 guided interviews with long term unemployed people as qualitative data source to investigate the association of stigma consciousness and sanctions by the German employment service. Initially, combining different methods was applied for providing contextual understanding and “putting ‘meat on the bones’ of ‘dry’ quantitative findings” (Byrman 2006: 106-107). By using side by side joint displays to integrate both results and the unexpected results of the quantitative analysis it became more appropriate to understand contradictions between both strands and the conceptual framework of the study. In general, with these joint displays the results of the quantitative examination strand can be seen in a particular and useful way. The same applies to the linkage to the theoretical framework of the study.
Even if scrutinizing the quantitative results works very well, some questions will still remain open. Discussing the integrated results is suitable for bridging, but also bears the risk of driving apart quantitative and qualitative research again. This attempt clarifies the well-known problems of compatibility and the specific and converse shortcomings of both approaches such as the poor quality of some variables, missing reality, etc. on the one hand and e.g. generalizability on the other hand. In addition, other issues emerge which are mainly related to the communication and the presentation of the results, such as different nomenclature, how to structure the manuscript or which journal to choose for publication, etc.
However, quote matrixes serve as suitable tools for analyzing the inferences in our mixed-methods design. Notably analytical depth and transparency can be increased and further research issues developed. However, the limitations of this approach need to be addressed.
The challenge in data analysis often lies in taking account of the multi-dimensionality and complexity of the data and at the same time discovering patterns, which requires reduction and simplification. Quantitizing qualitative data can serve as an additional step during data analysis to broaden the perspective and collect complementary views. Examples from research illustrate how information gathered during one data collection can generate different types of data or how qualitative data can be transformed to be analysed statistically. Practi-calities of integrating qualitative and quantitative analyses are illustrated to offer good-practice examples for transformation designs. With the methodological reflection of research practice we evaluate consequences for the field of mixed methods research, in which qualitative and quantitative analyses are usually independent of each other and the statistical analysis of codes created during an interpretive phase plays a minor role.
Within mixed methods research (MMR) two principal approaches can be identified. The first one aims to a logic of complementarity, under which bringing qualitative and quantitative strategies together may be seen as an attempt of information enrichment, to obtain a more comprehensive understanding of a phenomenon [1]. The second one concentrates on a convergence perspective that focuses on the possibility to overcome single methods biases through data integration [2]. The assumption behind this second approach is that qualitative and quantitative methods have specific strengths and limitations and merging them may increase the overall validity of a study. While the first objective appears rather unproblematic both theoretically and empirically, the second one poses some challenges, especially in relation to the elicitation of “meta-inferences”.
The purpose of this work is to explore how the two perspectives of complementarity and convergence are intended and endeavored by MMR studies within the social sciences. In order to provide an answer to this issue, research synthesis seems particularly appropriate. We will conduct, indeed, a research synthesis using a mixed strategy to investigate those studies located in the MMR field. We partially refer here to the framework mixed methods research synthesis (MMRS), which is “a systematic review applying the principles of mixed methods” [3, p. 662]. Still, this study would only consider mixed researches from a methodological point of view and it is not intended to synthesize findings from mixed, qualitative and quantitative researches.
We will realize a literature dépistage on MMR in social research, identifying significant articles from scientific journals databases. Here the WoS (Web of Science) and Scopus databases are explored from an exhaustive perspective, retrieving all articles published in international journals (2003-2016). The phase of evaluation would regard the screening of the retrieved mixed studies, according to relevant criteria. Analyses are conducted following three main data analysis techniques: automated content analysis, citation network analysis and scholars interviews. The first analysis would help to distinguish between theory and praxis, underling eventual discrepancies or concurrences between the declared theoretical approach of the article and the actual solution implemented in the research design, thanks to a coding protocol. The second one would identify nodes and relationships among the authors of these articles to identify clusters of citations/authors. Then, on the basis of pinpointed clusters, interviewees will be selected among researchers, so that actors’ perspectives may emerge, enriching the analyses and being compared with outcomes from the text and citation analyses.
Finally, we expect the approach of complementarity to show some consistency in respect to the two level of theory and praxis, while it is likely that convergence poses differences in ways it is theoretically considered and modalities it is implemented empirically.
[1] Jick T., (1979), Mixing qualitative and quantitative methods: Triangulation in action, in “Administrative Science Quarterly”, 24:4.
[2] Erzberger C., Prein G., (1997), Triangulation: Validity and empirically-based hypothesis construction, in “Quality&Quantity”, 31.
[3] Heyvaert M., (2013), Mixed methods research synthesis: Definition, framework and potential, in "Quality&Quantity", 47:2.
New sources of data such as, for example, learning analytics (LA) provide opportunities for addressing new research questions as well as alternative approaches to answer the old ones. It has been argued that these new sources of data (known as found or organic data) are not able to replace traditional forms of data and that they should instead be viewed as a complimentary source of information. While survey researchers have been actively discussing integration strategies with respect to found or organic data, discussion on the role and advantages of qualitative data given these new challenges is at the outset. Drawing on two pilot studies, we illustrate how qualitative data can be integrated with LA and survey self-reports for two purposes: evaluation of data quality and answering substantive research questions. While in the first approach qualitative data help interpret discrepancies found while comparing survey and LA data, in the second application it is used to provide addition insights missed by the other two methods. Two pilot studies were implemented during two online courses conducted between February and September 2016 as part of an online program for working professionals at the University of Mannheim. The survey and LA data stem from 16 participants observed over 12 weeks of the 1st course (Nit=192) and 15 participants in the 2nd course (Nit=180).
The first approach of integrating qualitative data is centered around the problem of student workload measurement. As self-reported data obtained from tradition survey instruments can suffer from various problems, LA can offer an alternative way of measuring workload in online or blended learning environments. We compared results provided by video watching logs and a weekly evaluation survey. We found that survey self-report data and LA result in different estimates of video watching workload. Given that neither of the two methods is free from errors, understanding of the observed differences called for additional data. To provide further insights for the observed differences, we have conducted a series of qualitative interviews with 13 study participants. The interviews included two components. The first one included cognitive interview questions aimed at evaluation of the survey data. Furthermore, we included questions addressing how students interacted with videos, which helped us understand the data generating process of LA data.
The second application of qualitative data included the evaluation of two design changes for the purpose of increasing student engagement: implementation of additional video material and comparison of synchronous and asynchronous communication techniques with students. Following the design-based research perspective popular in the learning science, we integrated LA, weekly survey data as well as qualitative semi-structured interviews in order to draw a more complete picture of students’ engagement. Qualitative interviews were conducted at the start (N=16) and at the end of the courses (N=13).
We discuss differences in integration strategies arising from the two distinct purposes. We conclude with a discussion of ethics and data protection implications.