Medical education research: aligning design and research goals

Given that the training of doctors started some 2000 years ago, one might think that everything that needs to be known about medical education is long known. Not so. Times change and so do practice, society, health care systems and patient expectations. There are also shifts in how we deliver education, and in training linked to changing health care practices and systems (eg, limits on hours of training).1 Moreover, our knowledge of what constitutes good clinical practice is constantly evolving.2

Medical education must therefore prepare today’s medical students and doctors in training to work in very different ways from those of the past.3 One way to address this is to emphasise the use of evidence from well designed — and well conducted — medical education research. Almost any gap between best practice and what actually happens in medical education can be addressed by encouraging thinking, discovery, innovation and improvement through well planned research.

Developing an educational research question

The first step in medical education research is the process of developing an education research question. This is no different from developing a clinical research question. The process typically begins with a problem, an issue you would like to know more about or a situation that needs to be improved. For example, do you want to understand why students who fail are less likely to seek feedback than those who perform well? Or do you want to bring in a new approach to teaching laparoscopic techniques, then see if this change accelerates residents’ skills acquisition? Once you are clear about the problem, the next step in the process is to read the relevant literature, to identify what is already known about the problem, the gaps in knowledge, and why addressing a particular gap is important.4 This last point is important — it is the “who cares” or “so what” aspect of the process of developing a research focus, and is what shows other people (other educators, and journal editors) that your study is interesting and relevant.

Assumptions about the nature of reality (ontology) and the nature of knowledge (epistemology) will influence how you perform your review of the literature and how you formulate your research question. Do you believe there is an absolute truth, that knowledge is objective and reality is tangible and measurable (a positivist perspective)? This approach underpins most laboratory and clinical research and hence is familiar to medical doctors. Or do you hold that reality is subjective, existing as perceived by people and constructed by social, historical and individual contexts (a social constructivist or interpretivist perspective)? In reality, educational research initiatives may contain both positivist and social constructivist domains, and where this is the case, the approach to research planning and data interpretation will carefully integrate and consider the relative contributions of both perspectives and how they complement each other.

The philosophical stance you adopt will influence the nature of your research question. Positivist research questions typically focus on testing and confirming a provisional (null) hypothesis. For example, you might want to examine whether early performance predicts later performance,5,6 whether students spend more time preparing web-based compared with paper-based portfolios,7 or how much variance in Mini-Clinical Evaluation Exercise (Mini-CEX) ratings is attributable to raters’ social judgments.8 On the other hand, social constructivist research questions tend to be about hypothesis generation — they are open and exploratory, seeking to understand phenomena such as student culture in a medical school,9 the unintended consequences of a curriculum reform,10 or how residents learn.11 (Interestingly, the study by Gingerich and colleagues8 is an example of a quantitative study which starts from a social constructivist perspective.)

If positivist philosophy is concerned with establishing casual connections while social constructivism is concerned with describing phenomena in their natural setting, then different methods of enquiry and data collection are needed for each.12 These are typically categorised into quantitative (explanatory) and qualitative (exploratory) methods. Each approach has its own means of designing research, as well as of collecting and analysing data (Box).

Study design: quantitative v qualitative

There are four broad approaches to study design within quantitative research: descriptive, correlational, quasi-experimental and experimental.13 The randomised controlled trial (RCT) is an example of experimental research with which most readers will be familiar. RCTs have been adopted to examine how innovations in medical education compare with existing alternatives with respect to their effectiveness and costs.14 RCTs are central to hypothesis and guideline development. Conducting well designed RCTs in medical educational research is of great importance because they allow causal conclusions to be drawn.15 This information can then be used to generate questions and later investigations in everyday (uncontrolled) educational settings such as medical schools and hospitals. Cook16 provides a comprehensive overview of the role of RCTs in medical education.

The study design of qualitative research is also predominantly determined by the research question. Creswell17 provides a popular and helpful categorisation of qualitative research designs: ethnographic, narrative, phenomenological, grounded theory, and case study. While the five methods generally use similar data collection techniques (observation, interviews, and reviewing documents or visual material), the purpose of the study differentiates them. Phenomenology describes experiences as they are lived, ethnography describes a culture or context, while a case study may describe in depth the experience of one person, family, group, community or institution.12,17,18 Designing qualitative studies requires pre-planning in terms of scoping the project and considering the goal of the study (including what data are required in advance, which will in turn dictate the study design). However, the activities of collecting and analysing data, developing and modifying theory, and elaborating or refocusing the research questions usually occur more or less simultaneously, each influencing all of the others. In this way, qualitative research design is less linear than quantitative research, which is much more step-wise and fixed, so the study can be replicated or repeated using the same protocol at another time or by a different researcher.

There are several other fundamental differences related to quantitative and qualitative study design. We discuss some of the main ones below.

The first is the role of the investigator in the research process. Where the goals of knowledge are to describe, observe and measure something tangible (ie, objectivity), the investigator and the focus of the research are independent of each other. The investigator has no influence on the research process. On the other hand, qualitative research and analysis are dependent on the relationship between the investigator and the research process.19 The specific ideas and beliefs predominant in the society and groups to which the investigator belongs will affect or even determine what she or he discovers and concludes. This is akin to the real world of clinical medicine, where diagnosis and management are positioned within a human and personal approach. This can be addressed by investigators reflecting on their position in the research, making explicit the assumptions on which their studies are based via self-reflection or reflexivity.19 There are recommended techniques for doing just this through the process of checking credibility, confirmability, transferability and dependability (eg, triangulation, audit trails).20

The second difference is the use of theory or conceptual frameworks in medical educational research. In positivist, quantitative research, theory is often seen as something from which to derive a hypothesis, a tentative explanation that accounts for a set of facts and can be tested by further investigation. The use of theory is much more explicit in the qualitative tradition. Here, it typically has the purpose of providing a framework to organise and interpret the data in such a way as to highlight commonalities and patterns and generate conceptual generalisability.21 These can then be assessed by others for their transferability and potential for applicability to other situations.22

The third area of difference is data collection. The goal of qualitative studies is to achieve depth of understanding whereas that of quantitative studies is to achieve breadth of understanding.23 Quantitative study samples need to be of sufficient size to enable statistical analysis and to demonstrate associative or causative relationships between variables. One of the ultimate aims of quantitative research is generalisability; representative sampling is therefore critical, as is probabilistic or random sampling to minimise the potential for bias in selection and to control for confounders. In contrast, in qualitative inquiry, samples are selected purposefully to yield cases that are information rich.23 Purposive sampling involves identifying and selecting individuals or groups of individuals who are especially knowledgeable about or experienced with the phenomenon of interest.24 For example, if your problem area is the quality and quantity of feedback at medical school, you may want to sample clinical teachers who differ in seniority, active participation in teaching, and clinical specialty, as well as medical students from different year groups. Qualitative methods place primary emphasis on saturation (ie, continuing to sample until no new substantive information is acquired), whereas quantitative methods place primary emphasis on generalisability (ie, ensuring that the knowledge gained is representative of the population from which the sample was drawn). Thus, quantitative studies tend to involve relatively large numbers compared with qualitative studies.

The final difference is perhaps less about design and more about the management and presentation of data. Different research methods generate different types of data, and these different types of data require different analytical approaches. Quantitative analysis presents and interprets numerical data typically via descriptive and inferential statistics. The process of qualitative data analysis is a little different. It usually follows an inductive approach where textual, observational or visual data are scrutinised for conceptual categories and descriptive themes, which are then organised by meaning and used for drawing conclusions.12,13,17,18,25 Quotes (eg, from interviews) are used to clarify links between data, interpretation and conclusions, and they help the reader assess face and content validity. In this way, quotes are evidence for data interpretation in the same way as tables of statistical data.

Both quantitative and qualitative research should have a logical chain of reasoning behind their choice of study design, methods, tools and analytical approaches. This allows researchers to rule out rival hypotheses and explanations with convincing arguments and solid data, and for the research outcomes to be judged on explicit quality criteria.26

Conclusion

As with all research, medical education research requires a philosophical stance, a research question, study design, data collection methods and data analysis. Incongruence across the different stages of any research project is very obvious to those reading and judging research. All designs have their strengths and weaknesses, and it is critical to be aware of these when thinking about how best to address a particular research goal.