MJA
MJA InSight
MJA Careers
Review

Quality in qualitative research

Simon C Kitto, Janice Chesters and Carol Grbich
Med J Aust 2008; 188 (4): 243-246.
Abstract
  • Qualitative research most commonly involves the systematic collection, ordering, description and interpretation of textual data generated from talk, observation or documentation.

  • A report of qualitative research should address the following criteria:

    • Clarification and justification;

    • Procedural rigour;

    • Representativeness;

    • Interpretative rigour;

    • Reflexivity and evaluative rigour; and

    • Transferability.

  • Because of the limitations on article length for the Medical Journal of Australia, authors should focus on only a couple of aspects of the research, rather than trying to present a simplified description of multiple aspects.

Research in health care, especially clinical medicine, is an increasingly complex field that ranges from small-scale, cutting-edge benchtop science to large-scale population studies. Recently, there has been an increasing recognition by evidence-based clinical researchers that it is important to look towards qualitative research. In addition, an expanding interest in examining the attitudes, beliefs and experiences of those involved or affected by the delivery of health care have brought qualitative research into the spotlight.1 The increased interest in this form of research has led to concerns by readers and reviewers about the assessment of quality.2-10 In this article, we provide definitions and standards for qualitative research that can be used for assessment. We expect that authors who submit work for publication that adheres to the suggestions in this article will increase the likelihood of acceptance for publication and, more importantly, will enhance the transferability of findings into policy and practice in the future.

Qualitative research

Most commonly, qualitative research is concerned with the systematic collection, ordering, description and interpretation of textual data generated from talk, observation or documentation. Qualitative research methods include the techniques of interviewing, observation, and document analysis. Its goal is to explore the behaviour, processes of interaction, and the meanings, values and experiences of purposefully sampled individuals and groups in their “natural” context.11-14 The capacity to make conceptual generalisations from the local context of a qualitative study to other settings is the desired outcome.

In contrast, most quantitative research is concerned with measuring the magnitude, size or extent of a phenomenon. Data collection is by formal rules of procedure and verification, analysis is through the use of standardised statistical formats, and prediction and empirical generalisation are the desired outcomes.15,16 The conventional methodological criteria of quantitative research — validity, reliability and empirical generalisability — are generally not directly applied to qualitative research because of the different frameworks, sampling approaches, size of sample and goals of qualitative research. Instead, terms such as rigour (thoroughness and appropriateness of the use of research methods), credibility (meaningful, well presented findings) and relevance (utility of findings) are used to judge the quality or “trustworthiness” of a study (see Box 1 for definitions of some common terms in qualitative research).17

MJA guidelines for assessing qualitative research

There has been a strong move towards standardising research reporting in quantitative research, as can be seen through the development of standard reporting mechanisms like the CONSORT (Consolidated Standards of Reporting Trials) statement.18 However, some question the appropriateness of using checklists to assess qualitative research because of the diversity of approaches in collecting, analysing and interpreting qualitative data.4,19 Furthermore, the sheer number of checklists can prove overwhelming and confusing for new researchers. A recent review of criteria for assessing interview and focus group studies revealed more than 22 checklists actively being used, and subsequently resulted in the formulation of a new 32-item guideline.20

Nevertheless, it is possible to develop clear and useful generic guidelines for assessing and presenting qualitative research.2,7,17,21 Box 2 outlines some criteria that have been constructed with the MJA readership in mind that can be used to assess and enhance the rigour of qualitative studies. In themselves, these criteria do not ensure rigour. However, they can strengthen rigour if they are used in concordance with a broader understanding of qualitative research design, data collection and analysis.19

Clarification and justification

As in all forms of research, clarity of research question reflected in the aims of the study is essential for evaluating results and their interpretation. The demonstration of theoretical rigour (referring to the soundness of fit of the research question, aims and the choice of methods appropriate to the research problem11) is extremely important.

There is a wide variety of named qualitative approaches that are underpinned by particular theoretical perspectives. In addition, the researcher may use basic field research (question, investigation, interpretation). Regardless of the theoretical approach used, the choice requires justification in reference to the research question of the study.

Procedural rigour

Procedural, or methodological, rigour concerns the transparency or “explicitness” of the description of the way the research was conducted. It involves detailing issues of accessing subjects; development of rapport and trust; how data are collected, recorded, coded and analysed; and accounts of the manner in which errors or subject refusals are dealt with.4,11,22 In this regard, readers and reviewers may ask the following questions while examining descriptions of qualitative methods: How were participants/settings accessed? Who was interviewed/observed? How often? For how long? What interview questions were asked? What was the purpose of any observation? Which policy documents/case notes were accessed? How were they assessed? How was collected data managed?

Representativeness

There are a number of commonly available, non-probability sampling approaches. Maximum variation sampling seeks representativeness of all aspects of the topic in terms of participants. Homogenous sampling consists of the selection of a group fitting specified criteria. Snowball sampling involves networking from one difficult-to-access type of participant to a wider range of participants. Finally, convenience sampling involves studying easily accessed individuals or groups. This last technique obviously presents its own ethical dilemmas of the “insider” type and is possibly the weakest form of sampling in terms of allowing conceptual generalisability.4,15,16,22,23 Maximum variation is the ideal when a holistic overview of the phenomenon is sought; for instance, the question of how a particular hospital department operates may involve sampling in the wider organisation as well as within the individual department and among recipients of services.

Simply mentioning the sampling strategy in the methods section of a qualitative research paper is not sufficient. The key findings of the research need to be evaluated in reference to the diverse characteristics of the research subjects. Through constantly comparing the experiences and responses of the participants against each other, subtle but significant differences can be uncovered that can generate profound insights into the phenomena under study.19

Interpretative rigour

Interpretative rigour relates to as full as possible a demonstration of the data/evidence. In qualitative research, a commonly used concept is inter-rater reliability. This refers to using a type of researcher triangulation by which multiple researchers are involved in the analytical process. This is an attempt to increase the validity and reliability of the study19 through the provision of a more complex and nuanced understanding of the possible interpretations of the objects of the research.11 In contrast to the quantitative research paradigm, what is important in this process is not the level of consensus, but the opportunity for discussion among analysts to provide opportunities for developing further coding.19

A related technique is that of respondent validation, or member checking. This entails offering subjects interviewed the opportunity to view and amend their transcripts as a type of validity.12 However, this approach does have limitations due to the evolution over time of the positions and purposes of the researchers and participants, thereby potentially affecting interpretations and accounts. Respondent validation should be thought of as part of a process of reducing error, which involves the generation of further original data, which then requires interpretation.8

Other techniques that enhance interpretative rigour are the differing forms of triangulation: data (multiple evidentiary sources; ie, documents, interviews, survey data, observation), methods (multiple methods), and theory (multiple theoretical and conceptual frames applied to the research to enhance insights into phenomena). Using these forms of triangulation allows the development of a comprehensive understanding of the phenomena and can ameliorate the potential bias of simply using one method.4,5,8,11,16,22

In the interpretive process, accounts of “negative” or “deviant” cases are especially important. These are explanations pertaining to data or evidence that contradicts the researchers overall explanatory account of the phenomena.5

In sum, a clear description of what forms of analysis were used, the process and what were the major outcomes of the analytical process in terms of findings is needed to ensure quality for the author, and to enable an assessment to be made in terms of the analytical quality of the research by the reader.

Reflexivity and evaluative rigour

Reflexivity is where researchers openly acknowledge and address the influence that the relationship among the researchers, the research topic and subjects may have on the results.4,11,13 Fundamentally, reflexivity requires a demonstration by the researchers that they are aware of the sociocultural position they inhabit and how their value systems might affect the selection of the research problem, research design, collection and analysis of data.15 It also refers to an awareness by the researchers of the social setting of the research and of the wider social context in which it is placed.4

Evaluative rigour refers to ensuring that the ethical and political aspects of research are addressed. Typically, this refers to proper ethics approval from appropriate committees covering confidentiality, informed consent and steps to avoid possible adverse effects on the subjects. Importantly, where appropriate, relevant community leaders should be consulted in the design and conduct of the research.11 Researchers should revisit their actions and interactions within the research process to ensure as “accurate” as possible portrayal of the production of their findings.

Transferability

Conceptual generalisability and transferability refer to how well the study’s findings inform health care contexts that differ from that in which the original study was undertaken.4 For example, a review of data from qualitative studies was conducted on a wide variety of doctor–patient interactions about medication compliance.24 The authors examined barriers to patients taking prescribed medication as directed by their doctors and found that patients were often inclined to resist taking medicines, not because of problems with the patients, doctors or systems, but because patients were concerned about the medicines. This type of study allows for the construction and transfer of general policy on medicine-taking (through, for example, less emphasis on patient behaviour modification and more emphasis on production of safer medicines) and practice (suggesting, for instance, that doctors should assist lay evaluations through provision of more information, support, feedback and safe prescribing practices).

The parameters for presenting qualitative research in the MJA

The formats of biomedical journals such as the Medical Journal of Australia and BMJ provide particular challenges for comprehensive analysis and reporting of qualitative studies.4,11,22 The house style of biomedical journals involves tight restrictions on word length and particular forms of presentation. For instance, lengthy ethnographic studies of health issues can require 6000–8000 words to successfully elucidate the phenomena.25 Obviously, this is not tenable within most biomedical journals, where article lengths of 2000–3000 words are more common. These raise significant limitations to an exhaustive analysis of the volume of data that is commonly produced within qualitative studies.5

When writing for the MJA, we suggest that qualitative authors focus on only a couple of aspects of their research findings and use visual displays strategically (ie, charts, quotes and tables).22 We suggest that authors avoid an overabundance of simplified descriptions of multiple aspects of the phenomena under study. This detracts from a focused, transparent and considered analysis of the core issues relevant to the objectives of the research.

Conclusion

We have set out some useful general rules that, if followed, will allow for concise and informative assessment of qualitative findings. These assessment criteria will be particularly relevant to articles that use the most common qualitative data collection techniques: interviews, focus groups, document analysis, or observation techniques (alone or in combination). We suggest that potential MJA authors also consider using the article format and distribution of word count presented in Box 3 to maximise the rigour and clarity of their research articles. Using this outline should allow the authors to provide the reviewers and readers with enough information on each aspect of the research to engender a sense of research rigour and the trustworthiness of research findings.

1 Glossary of terms

Credibility: refers to whether the findings are well presented and meaningful

Evaluative rigour: the transparent description of ethical and political aspects of the conduct of research

Procedural rigour: the transparent description of the conduct of research

Reflexivity: open acknowledgement of the complex influences among the researchers, the research topic and subjects on the research results

Transferability or relevance: refers to how useful the findings are to the context and phenomenon under study

Triangulation: a comprehensive approach to the conduct of research using multiple theories, data and methods

2 Criteria for assessing qualitative research

Clarification

  • What are the aims of the research?

  • What is the research question?

Justification

  • Why is a qualitative approach the best option to answer this question?

  • Why was the particular qualitative research design chosen?

Procedural rigour

  • Have the techniques of data collection been clearly documented?

  • Are the forms of data analysis completely transparent?

Representativeness

  • What sampling techniques have been used to answer the research question?

  • Do the sampling techniques support conceptual generalisability?

Interpretation

  • Has a more conceptual discussion of the results and linkage to existing theory or new theory been developed to explain the relevance of findings to a targeted audience or discipline?

  • Have any negative cases been included and discussed?

Reflexivity and evaluative rigour

  • Has a clear statement of the effect on the data of the researcher’s views and the methods chosen been included?

  • Has an explicit evaluation of the relationship between the researcher and those under research, addressing any ethical issues, been discussed?

  • Has ethics approval been obtained from an appropriate institution?

Transferability

  • Has a critical evaluation of the application of findings to other similar contexts been made?

  • Has the relevance of these findings to current knowledge, policy, and practice or to current research been discussed?

3 Format of qualitative research articles

Section

Word count


Abstract

200–250

Aims, Methods, Results, Conclusion

Introduction and review of literature

250–300

Methods

Approach

100–150

Setting and sampling strategy

75–100

Data collection techniques

75–100

Analysis

100–150

Results

750–900

Discussion

400–450

Implications and limitations

50–100

Total

2500

Received 
4 Feb 2007
accepted 
24 Sep 2007
Simon C Kitto, PhD, Senior Lecturer1
Janice Chesters, PhD, Associate Professor1
Carol Grbich, PhD, Professor2
1 School of Rural Health, Monash University, Moe, VIC.
2 Flinders University, Adelaide, SA.
Competing Interests: 

None identified.

Reference Text: 
Green J, Britten N. Qualitative research and evidence based medicine. BMJ 1998; 316: 1230-1232.
Reference Order: 
1
PubMed ID: 
9583929
Reference Text: 
Boulton M, Fitzpatrick R, Swinburn C. Qualitative research in health care: II. A structured review and evaluation of studies. J Eval Clin Pract 1996; 2: 171-179.
Reference Order: 
2
PubMed ID: 
9238586
Reference Text: 
Daly J, Willis K, Small R, et al. A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol 2007; 60: 43-49.
Reference Order: 
3
PubMed ID: 
17161753
Reference Text: 
Green J, Thorogood N. Qualitative methods for health research. London: Sage, 2004.
Reference Order: 
4
PubMed ID: 
Reference Text: 
Mays N, Pope C. Qualitative research: rigour and qualitative research. BMJ 1995; 311: 109-112.
Reference Order: 
5
PubMed ID: 
7613363
Reference Text: 
Mays N, Pope C. Qualitative research: observational methods in health care settings. BMJ 1995; 311: 182-184.
Reference Order: 
6
PubMed ID: 
7613435
Reference Text: 
Mays N, Pope C. Quality in qualitative health research. In: Pope C, Mays N, editors. Qualitative research in health care. London: BMJ Books, 1999: 89-101.
Reference Order: 
7
PubMed ID: 
Reference Text: 
Mays N, Pope C. Qualitative research in health care: assessing quality in qualitative research. BMJ 2000; 320: 50-52.
Reference Order: 
8
PubMed ID: 
10617534
Reference Text: 
Pope C, Mays N. Qualitative research: reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ 1995; 311: 42-45.
Reference Order: 
9
PubMed ID: 
7613329
Reference Text: 
Pope C, Ziebland S, Mays N. Qualitative research in health care: analysing qualitative data. BMJ 2000; 320: 114-116.
Reference Order: 
10
PubMed ID: 
10625273
Reference Text: 
Liamputtong P, Ezzy D. Qualitative research methods. Melbourne: Oxford University Press, 2005.
Reference Order: 
11
PubMed ID: 
Reference Text: 
Lincoln YS, Guba EG. Naturalistic inquiry. Newbury Park, Calif: Sage, 1985.
Reference Order: 
12
PubMed ID: 
Reference Text: 
Malterud K. Qualitative research: standards, challenges, and guidelines. Lancet 2001; 358: 483-488.
Reference Order: 
13
PubMed ID: 
11513933
Reference Text: 
Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. 2nd ed. Thousand Oaks, Calif: Sage, 1994.
Reference Order: 
14
PubMed ID: 
Reference Text: 
Grbich C. Qualitative research in health: an introduction. London: Sage, 1999.
Reference Order: 
15
PubMed ID: 
Reference Text: 
Grbich C. Qualitative data analysis: an introduction. London: Sage, 2007.
Reference Order: 
16
PubMed ID: 
Reference Text: 
Critical Appraisal Skills Programme (CASP). 10 questions to help you make sense of qualitative research. Oxford: Public Health Resource Unit, 2006. http://www.phru.nhs.uk/Doc_Links/Qualitative%20Appraisal%20 Tool.pdf (accessed Oct 2007).
Reference Order: 
17
PubMed ID: 
Reference Text: 
Moher D, Schulz K, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001; 285: 1987-1991.
Reference Order: 
18
PubMed ID: 
11308435
Reference Text: 
Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ 2001; 322: 1115-1117.
Reference Order: 
19
PubMed ID: 
11337448
Reference Text: 
Tong A, Sainsbury P, Craig JC. Critical appraisal criteria for interviews and focus groups. In: Qualitative Research in Evidence-based Healthcare — an Exploration of Scope and Methods. 2006; Jul 10–11; Adelaide.
Reference Order: 
20
PubMed ID: 
Reference Text: 
Blaxter M. Criteria for qualitative research. Med Sociol News 2000; 26: 34-37.
Reference Order: 
21
PubMed ID: 
Reference Text: 
Hansen EC. Successful qualitative health research: a practical introduction. Sydney: Allen and Unwin, 2006.
Reference Order: 
22
PubMed ID: 
Reference Text: 
Rice PL, Ezzy D. Qualitative research methods: a health focus. Melbourne: Oxford University Press, 1999.
Reference Order: 
23
PubMed ID: 
Reference Text: 
Pound P, Britten N, Morgan M, et al. Resisting medicines: a synthesis of qualitative studies of medicine taking. Soc Sci Med 2005; 61: 133-155.
Reference Order: 
24
PubMed ID: 
15847968
Reference Text: 
Moore D. Governing street-based injecting drug users: a critique of heroin overdose prevention in Australia. Soc Sci Med 2004; 59: 1547-1557.
Reference Order: 
25
PubMed ID: 
15246182