Mixed Methods and the Limits of Social Research 1 |
|
| Chair | Dr Andrea Hense (Sociological Research Institute Göttingen (SOFI), Germany ) |
| Coordinator 1 | Professor Udo Kelle (Helmut Schmidt University Hamburg, Germany) |
| Coordinator 2 | Dr Felix Knappertsbusch (Giessen University, Germany) |
Confronted with the task to describe cultural-normative conceptions of family, we combine data from a quantitative and a qualitative instrument. Quantitative data come a survey (‘Family Leitbild Survey’, FLB 2012), conducted in Germany in 2012 with n=5000 participants, representative for the population, aged 20 to 39. We use a question and an item set in which seven different living arrangements are described, asking for each of them whether or not the respondent would consider it a family (e.g. ‘a gay or lesbian couple with own children’). We study the patterns of answers across the seven items, looking for underlying principles that could be interpreted as understanding of what family means to the respondent.
Qualitative data come from a study (‘Family in Pictures’) combining drawings with subsequent semi-structured interviews. We recruited n=101 participants (16+) in Germany in early 2015 in a quota sample. Each participant was sent a kit by mail that included coloured pencils, paper and the instruction to draw a ‘real’ family. Participants also were asked to fill out a short standardised questionnaire regarding their socio-demographic characteristics, so that the composition of the sample could be monitored and basic comparisons between social groups were enabled. Each drawing then was interpreted in itself by a group of researchers in an ad hoc interpretation. Interpretations were recorded and transformed into questions in an interview guide. In the semi-structured phone interview, participants were, at first, asked to describe their basic thoughts and intentions when reading the instructions and starting to draw. This description served to consider to what degree the drawing actually represented a cultural conception or merely the participant’s personal family situation. The participants then were asked about each detail of their drawing to confirm or correct the preceding ad hoc interpretations. In as much as participants were open to describe own nonreflective stereotypes, their self-interpretation is taken as accurate interpretation of the drawing. In as much as they seemed to filter their answers and report ‘politically correct’ perceptions, the ad hoc interpretation is taken as more probable. This way, the participant’s personal conception is captured of how a ‘real’ family looks like. Motifs that are found in several drawings indicate collectively shared cultural conceptions.
The combination of the two methods allows combining advantages of both approaches: a fairly complete identification of existing family conceptions, including those not represented in the survey items, a fairly detailed and illustrative description of each conception, evidence regarding the prevalence of each conception and statistical evidence regarding significant differences between social groups. Also the combination increases the reliability of results since findings from one instrument can be confirmed by the other. A methodological limitation lies in the restricted comparability: The qualitative study encouraged participants to draw one rather than several pictures of a family, and it asked for a picture of a ‘real’ family. Accordingly this method rather captures the core of family understandings while the quantitative method rather reveals their openness and boundaries.
In Germany welfare benefit recipients subject to benefit sanctions within the Social Code II live below the subsistence level for a fixed period of time. Individuals with low education bear an increased risk of being sanctioned. Furthermore, our quantitative analyses of linked administrative and survey data using bivariate probit models indicate that this increased risk cannot be attributed to a lower motivation to work or a lower willingness to make concessions regarding job offers. Our content and documentary analyses of qualitative interviews with jobcentre employees and sanctioned unemployed and their individual case files reveal complex processes: Low cultural capital when dealing with administrative matters, a habitual distance vis-à-vis jobcentre employees as well as previous negative attributions in individual case files can advance the sanctioning of low-skilled welfare recipients. In this way, sanctions contribute to the (re)production of social inequality.
These findings are based on secondary analyses of different quantitative and qualitative data sets. In our presentation we firstly would like to discuss the advantages of combining these diverse datasets in order to reduce the limitations of a mono-methodological approach. Thus, the administrative data produced by the jobcentres provide big data with reliable information on sanctions. Through record linkage with survey data of welfare recipients, information on motivation to work and willingness to make concessions regarding job offers can be added to the multivariate model to reduce omitted variable bias. But the finding of an increased risk of being sanctioned of low qualified welfare recipients persists. For a better empirically based understanding of these central quantitative findings and for a complementary design, qualitative data that represent perspectives of jobcentre employees and sanctioned unemployed and their individual case files have been analysed as well. Six possible reasons could be identified; together with the quantitative findings they have built the basis for policy advice and recommendations for (further) training of case workers and executive managers of the jobcentres.
However, our mixed methods approach had to deal with methodological problems too, e.g. with a different terminology and conceptualization of categories in our quantitative and qualitative research. For example, education as a central category has been defined and operationalised by formal educational attainment in our quantitative approach, while our qualitative approach has used a broader and more open concept – because the interviewees seldom used terms of formal attainment and because of a specific theoretical background (Bourdieu's concept of cultural capital). In our presentation we secondly would like to discuss those inconsistencies and how to deal best with them.
There is a longstanding debate in survey research methodology about how the wording of questions can lead to validity threats and substantial measurement error. This methodological problem area is highly interesting for the mixed-methods-community, since mono-method approaches are in most cases not sufficient to cope with the respective validity problems whereas the combined use of qualitative and quantitative methods can help to identify and to solve problems related to question wording.
We will give empirical examples for this by drawing on statistical analyses using items for the measurement of religiosity from different comparative large scale survey data sets (namely the “International Social Survey Programme” and the “European Social Survey”): with regard to religious and cosmological beliefs as well as concerning the self-assessment of religiosity and spirituality we can identify striking inconsistencies and peculiar distributions of responses, especially if we look at particular religious or non-religious sub-populations. These statistical results raise the question why members of certain religious subgroups of the general population (e.g. decidedly secular or religious, lukewarm religious or religiously disinterested respondents) show specific response patterns and whether question wording leads to unob-served heterogeneity (i.e. systematic differences between different groups regarding the un-derstanding of certain items). We tried to answer these questions by using a mixed methods design: in the quantitative strand of the study we tested different hypotheses about possible reasons for the observed inconsistencies. In the qualitative part of our research project we conducted qualitative in-depth interviews especially suited for the reconstruction of the re-spondents´ understanding of certain questions and terms contained in these questions (“cogni-tive interviews”). Our qualitative and quantitative results show that items developed for measuring religiosity often contain ambiguous terms and concepts which are difficult to un-derstand for members of some sub-populations while they carry well-defined (but culture-specifically different!) meanings for others. In terms of measurement-theory the resulting high variance of responses may in many cases still be interpreted as a sometimes high but accepta-ble (random) measurement error – however, it can also strongly restrict or even distort relia-bility of scales and, even worse, lead to systematic biases.
In the final part of the paper we will show that the presented mixed-methods-strategy is not only helpful to identify validity threats arising from question wording, but can also foster a deeper understanding of the use and meaning of specific religious languages in different social groups and milieus.
The websites of in-vitro fertilization (IVF) clinics have become privileged vehicles for search and provision of information by donors and recipients looking for fertility treatments with donated eggs and sperm. To monitor the quality of contents available on these websites is essential for the development of people-centred care and improve health literacy. There are different guidelines to assess the formal quality of these sources of information, such as: the Discern tool; the Health On Net code; eEurope guidelines for health-related websites; the AMA guidelines for medical and health information sites; and the recent Disease Prevention and Health Promotion Website Information Evaluation Instrument (ODPHP). However, such instruments are only based on quantitative approaches and do not evaluate the accuracy of the information on specific topics. Such an evaluation needs to be conducted case by case through a multi-method approach, taking into account national regulatory frameworks and health policies, as well as local, cultural, socioeconomic and clinical dimensions of reproductive medicine. We aimed to discuss the prospects and limits we have experienced when using a multi-method approach to assess the quality of the contents on gamete donation in the websites of the Portuguese IVF-clinics that import eggs and sperm from abroad or are available to provide cross-border reproductive care. On November 2016, the web page of the National Council for Assisted Reproductive Technologies was searched to identify the 13 eligible clinics.
The first challenge was to choose the most adequate quantitative instrument to assess the websites’ reliability (accuracy and credibility of website content, as well as transparency in its purpose and ownership) and usability (website architecture, overall design and content design). Our choice was the ODPHP mainly due to its completeness. Afterwards, we faced several dilemmas during the ongoing preliminary analysis: the quality of the websites could be overestimated; it only provides a partial analysis of the contents related to the circulation of recipients, eggs and sperm.
A complementary qualitative in-depth analysis of the contents of the websites is required, covering the following issues: risks and benefits of gamete donation (e.g., accuracy and robustness of the information about success rates, costs, and donor’s anonymity); circumstances under which fertility clinics recruit and select donors and match donor-recipient; protection of personal data and user involvement on balancing public/private and individual/collective interests; how patients’ expectations and doubts may be motivated by the displayed contents; the cultural norms that frame gender and parenthood relationships. We also realised that a critical analysis of the blogs included in these websites could be useful for gathering information about experiences, perceptions and feelings of recipients and donors.
Our preliminary assessment of the contents on gamete donation in the websites of the Portuguese fertility IVF-clinics that import eggs and sperm from abroad and or are available to provide cross-border reproductive indicate the need to use a multi-method approach to assure the quality of the websites when promoting fair, equal and sustainable people-centred healthcare systems.