INTRODUCTION

Medical professionals are always learning, whether in a formal sense via continuing medical education (CME) or as a result of their regular interaction with clinical cases [1-3]. The variable of ongoing informal learning can be a confounding factor that complicates determining the effectiveness of CME because it is difficult to control for in statistical analysis. Despite this, frameworks through which to explore the impact of educational interventions exist and are consistently being developed and refined. In this review article, I would like to propose a model of learning evaluation named the ‘7Is framework’ based on a new paradigm of evaluation using a linear approach rather than a vertical or hierarchical approach. This framework was developed on the basis of the recommendations identified from a literature review. Using the new paradigm to better understand the learning process and to inform educational interventions may enable better management of patients by physicians.

MODELS OF LEARNING EVALUATION

A frequently cited model of learning evaluation is that developed by Kirkpatrick [4-6]. The model, a four-stage approach to evaluation, was originally developed for business training directors. It comprises a hierarchy described in Table 1, starting with ‘reaction’ and progressing through ‘learning’, ‘behaviour’, and ‘results’. Kirkpatrick himself did not originally use the term ‘levels’ although links between levels are supported by Cook and West [7] as a means to understanding the bridge between the original intervention and the overall outcome (Fig. 1).

The use of the levels concept implies that the Kirkpatrick framework is about a product, that is, the aspect of a particular learning outcome of greatest interest, rather than a process, which would be the quality of the instruction needed to achieve that outcome [8]. Conceptual frameworks are often built on previous theories and revisions of original models. One of the most commonly cited variations of the Kirkpatrick approach was suggested by Barr et al. [9]. In a review of inter-professional education, subtypes of levels 2 and 4 of the Kirkpatrick framework were added (Table 2). This approach added more detail to the levels and provided researchers with a focused area of study. Although this approach is pragmatic, it has not been validated by either the original paper or subsequent work. The Best Evidence in Medical Education group [10,11] follows this framework; however, no formal study has compared a traditional Kirkpatrick level 2 with Barr’s version.

Another approach has been to focus purely on the learner, that is, the health care professionals themselves. Acknowledging Kirkpatrick as a source, Belfield et al. [12] proposed a five-level hierarchy: healthcare outcomes; healthcare professionals’ behaviour, performance or practice; learning or knowledge; reaction or satisfaction of participants; and participation or completion. Belfield et al. [12] highlighted the difficulty of outcomes-based research in a medical education setting, in particular, the problem that patient outcomes may only become apparent over a protracted period of time due to the time needed for the learner to acquire and implement new skills. They emphasised that levels of effectiveness, the outcome measures used, should be clearly reported and justified in study reports. This is of relevance to the utilization of Kirkpatrick, as much terminology in medical education such as ‘appraisal,’ ‘assessment,’ ‘evaluation,’ and ‘competency’ may carry different meanings across disciplines and specialties [12]. This lack of terminology standardization may mean that despite the fact that an evaluative framework such as Kirkpatrick’s is in place, the measures or metrics used to assess outcomes in each of the domains may not be reliable between, and even within, studies.

A model based on Kirkpatrick’s framework but also incorporating theories specific to medical education has been devised by Moore et al. [13] (Table 3). They clearly point out that their approach is a conceptual framework rather than a fully validated model. The incorporation of Miller’s pyramid [14], a theoretical approach to learning in which the learners move from ‘knows’ and ‘knows how’ to ‘shows how’ and ‘does’ is synergistic with the learning and performance domains. Similar to the approach adopted by Barr et al. [9], the essential Kirkpatrick framework is not changed but the detail provided at each level is greatly expanded. Again, although not directly validated, this approach has received endorsement by the Royal College of Physicians and Surgeons of Canada [15,16], where the knowledge and behaviour domains are separated into self-reported and observed elements.

Ultimately, in all the models discussed above, the hierarchal nature of the Kirkpatrick levels is present, and the flow from reaction to the intervention, followed by learning and implementation into behaviour and then a translation into patient benefit is still present. Given the specific lack of validation of these models, it is difficult to know whether they are enhanced versions of the Kirkpatrick model with tighter definitions of the domains or novel concepts in their own right. In an attempt to avoid the hierarchical structure inherent in Kirkpatrick’s framework, Hakkennes and Green [17] used five categories in no particular order to measure three separate outcome domains (Table 4). The Hakkennes model provides clear metrics, with examples, in which to measure the effectiveness of the intervention. Uniquely to this model, the concept of actual versus surrogate measures is utilized. The Kirkpatrick model allows the end-user to decide the outcome measure at each particular level. Hakkennes and Green [17] provide a framework for strengthening validity by differentiating between actual or gold-standard outcomes and those that are perceived or potentially confounded.

Increasing the level of detail in each domain or level raises a challenge to the epistemological approach used by developers of outcomes-based systems. In the business-oriented model put forward by Kirkpatrick, a clear connection between learning and outcome is inherent in the structure, despite the absence of objective descriptors of each level. Conversely, the increasing detail provided by Hakkennes and Green [17] implies a more dissociated structure where information must be collated and analysed at each level. These approaches represent complex methodological debates and form part of a likely spectrum in paradigms for evaluating medical interventions. On one side is a discrete hierarchal approach, as demonstrated by Lemmens et al. [18], who describe an evaluation model for determining the effectiveness of disease management programs. This model has a sequence of knowledge acquisition, leading to behaviour intention and then change—ultimately leading to improved clinical outcomes. A case study supports their proposition with neither Kirkpatrick nor Barr cited as sources or alternatives to this evaluative approach. On the other side of the spectrum is a less prescribed approach espoused by Yardley and Dornan [19] in which the researcher assesses the complexity of the outcomes and applies an appropriate framework dependent on the processes that may occur from intervention to outcome (Fig. 2).

However as Yardley and Dornan [19] argues, it is important to ask not, “Did a specific outcome occur?” but “What were the outcomes of this intervention?” as the latter may aid in the determination of the underlying cause of the outcomes. In this respect, the Kirkpatrick level-based approach does aid reasons for success or failure of a particular project to be elucidated. For example, in a large study of the effect of an evidence-based care model to aid management of the febrile child, a number of quality measures, including laboratory testing, admission rates, length of stay and costs, demonstrated significant improvements [20]. However, as there was no evaluation of individual performance, it is difficult to know whether it was the processes in place that resulted in change or learning at an individual level. This may have consequences for implementation at other sites and makes it difficult to determine the most influential shifts in practice that will require validation in future studies.

PROBLEMS WITH KIRKPATRICK’S DESIGN

Commentaries on concerns with Kirkpatrick’s design predominantly relate to industries outside of health care. Bates [21] raised three concerns with the Kirkpatrick approach: It is an incomplete model, there is an unproven assumption of causality, and the higher outcome levels do not necessarily imply higher levels of information. Despite Kirkpatrick not initially using the term “level” himself, his design implicitly suggests that level-four outcomes are of greater importance than those of level one (Fig. 3). The hierarchy of organizational outcomes, or return on investment, having more ‘value’ than initial participant reactions has been challenged in the business community for some time [22]. Despite Kirkpatrick believing, in respect of evaluating level two (knowledge change), that “No change in behaviour can be expected unless one or more of these learning objectives have been accomplished” [6], research in the business sector has failed to confirm the hierarchical relationship the Kirkpatrick model suggests [23,24]. These concerns surrounding the Kirkpatrick model have introduced uncertainty in the medical education academic community. In 2012, Yardley and Dornan published a systematic analysis of the use of Kirkpatrick in the context of medical education. The work analysed articles utilising a Kirkpatrick evaluative approach and also used a case-control journal review to determine methodologies the Kirkpatrick system may miss [19]. They also identified a collection of commentaries highlighting how the very hierarchical nature of Kirkpatrick could bias the outcomes the evaluation was aiming to examine. Alliger et al. [23] and Abernathy [25] have previously argued from a business perspective that the very notion of levels of evaluation influences the academic approach the evaluator may take. Yardley and Dornan [19] suggested that medical education is a much more complex system than business, as the stakeholders include patients and doctors but also the patients’ families, the health systems and the communities around the health systems.

A NEW CONCEPTUAL MODEL FOR EVALUATION

The model put forward by Kirkpatrick has not gained universal traction throughout the academic medical educational community. Yardley and Dornan [19] felt that following Kirkpatrick blindly would be akin to “performing research on a clinical drug and not assessing the potential side effects.” Those utilising the Kirkpatrick Framework in the healthcare setting are assuming a linear sequencing of acquisition and subsequent use of knowledge, skills and attitudes. Medical interventions do not have a binary transactional nature and have unpredictable outcomes. In the same way, educational or other practice-changing interventions are also heterogeneous: Learners modifies some aspects of their behaviour but not others in response to the intervention. A framework acknowledging this effect would be beneficial. Furthermore, the continuing review of evidence, rather than the improvement of it, has been persistent in medical education evaluative literature. Buckley et al. [26] felt a level-based system of evaluation was neither intuitive nor theoretically sound, citing the need for a continuum of change rather than discrete levels. Despite this systematic review, many continue to endorse the Kirkpatrick framework and researchers continue to build upon the Kirkpatrick model [27]. Models that aim to enhance Kirkpatrick add to the levels, rather than redefining the concept entirely. The linear nature of individuals’ learning, leading to behavioural change and finally patient outcomes is implicit in all such systems.

DERIVING A NEW MODEL

As demonstrated by the variety of models discussed and described, many approaches to evaluation exist. It is likely that no one system is best for all situations; rather, the best approach varies according to factors such as the intent of the evaluation, the nature of key stakeholders, and available resources [28]. It is for this reason that evaluative frameworks are created. A framework has been defined as a set of steps to operationalise a particular evaluation model or approach. However, often the terms ‘approach’, ‘framework’, and ‘model’ are used interchangeably. The use of frameworks as conceptual models is a pragmatic way of simplifying potential complex methodological approaches. However, there is a risk that the designs in themselves lack epistemological rigor. It is argued from an information systems perspective that this could lead to problems either regarding applicability of outcomes in certain contexts or the actual feasibility in delivering certain evaluations [29].

In order to create a valid framework, the literature on the work of Kirkpatrick and others was utilised to derive core needs as follows: First, measures should be obtained from a variety of sources: Kirkpatrick is an incomplete model and a new framework needs to allow a broad range of evaluative methodologies. Second, the domains (or measures making up those domains) are independent of each other: The assumption of causality is a core feature of the original Kirkpatrick model, and this assumption has not been proven. Third, the framework does not explicitly suggest a hierarchy: A hierarchical approach gives an undue weight to certain parts of the framework. In clinical medicine, this may well be justified – the primary clinical outcome to the patient being a clear and transparent end point. In education, however, although patient benefit is of importance, not all educational intervention is necessarily related to patient experience or outcomes. Given the significant confounding influences on determining how participants may learn and apply their knowledge, concentrating on patient outcome may leave out information that determines the effectiveness of the intervention. The framework brings together a chain of methods which in totality deliver an overall effect rather than relying on just one component. Fourth, patient outcomes are considered: Although patient outcomes may not be the highest rung of outcome, they are a vital part of the evaluation process and must have internal validity with respect to the intervention used; that is, the intervention would plausibly result in the outcome being measured. Fifth, engagement with the intervention itself is considered: Previous models have only measured satisfaction with the model, not the actual uptake or utilisation itself.

The underlying tenet of the proposed structure was to demonstrate a process of evaluation. The model aimed to include some of the missing components of Kirkpatrick and other frameworks in creating domains encapsulating the interaction with the intervention, the intervention itself and the outcomes of the intervention. The purpose of including the intervention itself is to ingrain the principle that the evaluation requires information taken before, during and after the intervention.

The basic constructs of assessing learning and behaviour remain as in Kirkpatrick but are considered in parallel rather than on top of each other. The model proposed here uses the concepts of ‘ideation’ (what you think you have learned) and ‘integration’ (what you have shown you have learned) to ensure these two evaluative approaches are included. This brings together the learning and behaviour sections, not as a hierarchy but as a common domain. In recognition of the complexity of healthcare, the concept of ‘results,’ which correspond to level four of Kirkpatrick’s framework, becomes ‘implementation’ and ‘improvement’. This enables patient benefit to be encapsulated by direct clinical effects but also by experience outcome measures as well. The conceptual framework, provisionally entitled ‘the 7Is framework’, is shown in Fig. 4 and Table 5. The alliteration is coincidental but may aid memorability of the framework.

This framework has been recently utilised in a study examining outcomes of an e-learning package [30]. An audio-visual intervention in paediatric fever was designed, delivered and tested against the new system. Interaction with the intervention was variable; only 28.7% completed the post-learning section, and issues were identified with accessing the video cases. Although measures of ideation significantly increased and there was a trend towards behaviour change, full implementation of the guidance did not occur and overall admission rates increased. This work demonstrated that the 7Is framework allows the various effects of an intervention to be conceptualised, promoting the development of a set of valid and specific outcome measures, ultimately leading to more robust evaluation. The next steps are to validate the original domains further (Table 6).

CONCLUSION

Traditionally, evaluative frameworks have concentrated on knowledge, behaviour or system change as discreet entities. The 7Is framework may contribute a new paradigm to the literature on evaluating practice changing interventions: the recognition of the importance of interacting and interfacing with the intervention and removing a hierarchal structure. The 7Is framework can be used to improve understanding of why interventions are effective and will hopefully promote the development of improved outcome measures.

Acknowledgments

SUPPLEMENTARY MATERIAL

Fig. 1.

The bridge of levels from reaction to results. This shows the process of understanding as the ‘bridge’ between the original intervention and the overall outcome [7]. Diagram was drawn by Damian Roland based on licence-free clip art.

Fig. 2.

The range of epistemological approaches to evaluation. The researcher assesses the complexity of the outcomes and applies an appropriate framework dependent on the processes that may occur from intervention to outcome [19]. Diagram was drawn by Damian Roland.

Fig. 3.

The hierarchal nature of the Kirkpatrick evaluation framework where level-four outcomes are of greater importance than those of level one [21]. Diagram was drawn by Damian Roland.

Fig. 4.

A theoretical schema for evaluating outcomes of practice-changing interventions – The 7Is framework. Diagram was drawn by Damian Roland.

References

• 1. Nylenna M, Aasland OG. Doctors’ learning habits: CME activities among Norwegian physicians over the last decade. BMC Med Educ 2007;7:10. http://dx.doi.org/10.1186/1472-6920-7-10 ArticlePubMedPMC
• 2. Roberts T. Learning responsibility?: exploring doctors’ transitions to new levels of performance. ESRC End of Project Report 2009.
• 3. Graham D, Thomson A, Gregory S. Liberating learning [Internet] Milton Keynes: Norfolk House East; 2009 [cited 2015 Apr 24] Available from: http://www.nact.org.uk/getfile/2173
• 4. Kirkpatrick D. Evaluation. In: Craig RL, Bittel LR, editors. Training and development handbook. American Society for Training and Development. New York: McGraw-Hill; 1967.
• 5. Kirkpatrick D. Evaluation of training. In: Craig RL, editor. Training and development handbook: a guide to human resource development. New York: McGraw-Hill; 1976. 317 p.
• 6. Kirkpatrick D, Kirkpatrick J. Evaluating training programs: the four levels. 3rd ed. San Francisco: Berrett-Koehler Publishers Inc.; 2006.
• 7. Cook DA, West CP. Perspective: reconsidering the focus on “outcomes research” in medical education: a cautionary note. Acad Med 2013;88:162-167. http://dx.doi.org/10.1097/ACM.0b013e31827c3d78 ArticlePubMed
• 8. Morrison J. ABC of learning and teaching in medicine: evaluation. BMJ 2003;326:385-387. http://dx.doi.org/10.1136/bmj.326.7385.385 ArticlePubMedPMC
• 9. Barr H, Freeth D, Hammick M, Koppel I, Reeves S. Evaluations of interprofessional education: a United Kingdom review for health and social care [Internet]. London: Centre for the Advancement of Interprofessional Education; 2000 [cited 2015 Apr 24]. Available from: http://caipe.org.uk/silo/files/evaluations-of-interprofessional-education.pdf
• 10. Association for Medical Education in Europe. Best Evidence Medical Education [Internet]. Dundee: Association for Medical Education in Europe; 2013 [cited 2015 Apr 24]. Available from: http://www.bemecollaboration.org/
• 11. Association for Medical Education in Europe. BEME coding sheet [Internet]. Dundee: Association for Medical Education in Europe; 2005 [cited 2015 Apr 24]. Available from: http://www.bemecollaboration.org/downloads/749/beme4_appx1.pdf
• 12. Belfield C, Thomas H, Bullock A, Eynon R, Wall D. Measuring effectiveness for best evidence medical education: a discussion. Med Teach 2001;23:164-170. http://dx.doi.org/10.1080/0142150020031084 ArticlePubMed
• 13. Moore DE Jr, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof 2009;29:1-15. http://dx.doi.org/10.1002/chp.20001 ArticlePubMed
• 14. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65(9 Suppl):S63-S67. ArticlePubMed
• 15. Horsley T, Grimshaw J, Campbell C. Maintaining the competence of Europe’s workforce. BMJ 2010;341:c4687. http://dx.doi.org/10.1136/bmj.c4687 Article
• 16. Campbell C, Silver I, Sherbino J, Cate OT, Holmboe ES. Competency-based continuing professional development. Med Teach 2010;32:657-662. http://dx.doi.org/10.3109/0142159X.2010.500708 ArticlePubMed
• 17. Hakkennes S, Green S. Measures for assessing practice change in medical practitioners. Implement Sci 2006;1:29. http://dx.doi.org/10.1186/1748-5908-1-29 ArticlePubMedPMC
• 18. Lemmens KM, Nieboer AP, van Schayck CP, Asin JD, Huijsman R. A model to evaluate quality and effectiveness of disease management. Qual Saf Health Care 2008;17:447-453. http://dx.doi.org/10.1136/qshc.2006.021865 ArticlePubMed
• 19. Yardley S, Dornan T. Kirkpatrick’s levels and education ‘evidence’. Med Educ 2012;46:97-106. http://dx.doi.org/10.1111/j.1365-2923.2011.04076.x ArticlePubMed
• 20. Byington CL, Reynolds CC, Korgenski K, Sheng X, Valentine KJ, Nelson RE, Daly JA, Osguthorpe RJ, James B, Savitz L. Costs and infant outcomes after implementation of a care process model for febrile infants. Pediatrics 2012;130:e16-e24. http://dx.doi.org/10.1542/peds.2012-0127 ArticlePubMedPMC
• 21. Bates R. A critical analysis of evaluation practice: the kirkpatrick model and the principle of beneficence. Eval Program Plann 2004;27:341-347. http://dx.doi.org/10.1016/j.evalprogplan.2004.04.011 Article
• 22. Holton EF. The flawed four level evaluation model. Human Resour Dev Q 1996;7:5-21. http://dx.doi.org/10.1002/hrdq.3920070103 Article
• 23. Alliger GM, Tannenbaum SI, Bennett W, Trave H, Shotland A. A meta-analysis of the relations among training criteria. Person Psychol 1997;50:341-358. http://dx.doi.org/10.1111/j.1744-6570.1997.tb00911.x Article
• 24. Alliger GM, Janak EA. Kirkpatricks’s level of training criteria: thirty years later. Person Psychol 1989;42:331-342. http://dx.doi.org/10.1111/j.1744-6570.1989.tb00661.x Article
• 25. Abernathy D. Thinking outside the evaluation box. Train Dev 1999;53:19-23.
• 26. Buckley LL, Goering P, Parikh SV, Butterill D, Foo EK. Applying a ‘stages of change’ model to enhance a traditional evaluation of a research transfer course. J Eval Clin Pract 2003;9:385-390. http://dx.doi.org/10.1046/j.1365-2753.2003.00407.x ArticlePubMed
• 27. Tian J, Atkinson N, Portnoy B, Gold R. A systematic review of evaluation in formal continuing medical education. J Contin Educ Health Prof 2007;27:16-27. http://dx.doi.org/10.1002/chp.89 ArticlePubMed
• 28. Kahen B. Excerpts from review of evaluation frameworks Regina: Saskatchewan Ministry of Education; 2008.
• 29. Recker J. Conceptual model evaluation: towards more paradigmatic rigor. In: Castro J, Teniente E, editors. Proceedings of the CAISE’05 Workshops 2005 Jun 13-17 Porto, Portugal. Porto: Faculdade de Engenharia da Universidade do Porto; 2005
• 30. Roland D, Charadva C, Coats T, Matheson D. Determining the effectiveness of educational interventions in paediatric emergency care. Emerg Med J 2014;31:787-788. http://dx.doi.org/10.1136/emermed-2014-204221.25 Article

Figure & Data

References

Citations

Citations to this article as recorded by  

• Redefining the Role of Medical Affairs Professionals as Innovators and Leaders in Industry-Led Medical Education
  Sajita Setia, Elliot Loo, Salil Prakash Shinde, Manmohan Singh, Chew Hooi Wong, Karan Thakkar
  Pharmaceutical Medicine.2024; 38(3): 167.     CrossRef
• From not knowing to doing: An interprofessional approach to clinician training in use of Goal Attainment Scaling (GAS) as a recovery‐oriented outcome measure in a rural mental health service
  Kate Furlanetto, Rajlaxmi Khopade, Vivek Phutane, Ravi Bhat, Paul Stolee
  International Journal of Mental Health Nursing.2024; 33(5): 1471.     CrossRef
• Sharpening the lens to evaluate interprofessional education and interprofessional collaboration by improving the conceptual framework: a critical discussion
  Florian B. Neubauer, Felicitas L. Wagner, Andrea Lörwald, Sören Huwendiek
  BMC Medical Education.2024;[Epub]     CrossRef
• Healthcare decision-makers’ perspectives on evaluating conflict management training in paediatric healthcare: a utilisation-focused qualitative study
  Juliette Phillipson, Sarah Barclay, Esse Menson, Oscar Lyons
  BMJ Paediatrics Open.2024; 8(1): e003047.     CrossRef
• Integrating planetary health education into tertiary curricula: a practical toolbox for implementation
  Zerina Lokmic-Tomkins, Liza Barbour, Jessica LeClair, Jeneile Luebke, Sarah L. McGuinness, Vijay S. Limaye, Parvathy Pillai, Maxfield Flynn, Michael A. Kamp, Karin Leder, Jonathan A. Patz
  Frontiers in Medicine.2024;[Epub]     CrossRef
• A Conceptual Framework for Continuing Medical Education and Population Health
  Abhimanyu Sud, Kate Hodgson, Gary Bloch, Ross Upshur
  Teaching and Learning in Medicine.2022; 34(5): 541.     CrossRef
• Educational programs and teaching strategies for health professionals responding to women with complex perinatal mental health and psychosocial concerns: A scoping review
  Louise Everitt, Virginia Stulz, Rakime Elmir, Virginia Schmied
  Nurse Education in Practice.2022; 60: 103319.     CrossRef
• A systematic scoping review of teaching and evaluating communications in the intensive care unit
  Elisha Wan Ying Chia, Huixin Huang, Sherill Goh, Marlyn Tracy Peries, Charlotte Cheuk Yiu Lee, Lorraine Hui En Tan, Michelle Shi Qing Khoo, Kuang Teck Tay, Yun Ting Ong, Wei Qiang Lim, Xiu Hui Tan, Yao Hao Tan, Cheryl Shumin Kow, Annelissa Mien Chew Chi
  The Asia Pacific Scholar.2021; 6(1): 3.     CrossRef
• 198 A Novel ‘Virtual Simulation’ for The Advanced Life Support Group, Making a Dream a Reality: A Beginner’s Guide
  Carl Leith Van Heyningen, Kate Denning, Sibi Joseph
  International Journal of Healthcare Simulation.2021;[Epub]     CrossRef
• Changes in academic performance in the online, integrated system-based curriculum implemented due to the COVID-19 pandemic in a medical school in Korea
  Do-Hwan Kim, Hyo Jeong Lee, Yanyan Lin, Ye Ji Kang
  Journal of Educational Evaluation for Health Professions.2021; 18: 24.     CrossRef
• In situ simulation and its effects on patient outcomes: a systematic review
  Daniel Goldshtein, Cole Krensky, Sachin Doshi, Vsevolod S. Perelman
  BMJ Simulation and Technology Enhanced Learning.2020; 6(1): 3.     CrossRef
• Critical analysis of evidence about the impacts on surgical teams of ‘mental practice’ in systematic reviews: a systematic rapid evidence assessment (SREA)
  Huon Snelgrove, Ben Gabbott
  BMC Medical Education.2020;[Epub]     CrossRef
• Interprofessional communication (IPC) for medical students: a scoping review
  Chermaine Bok, Cheng Han Ng, Jeffery Wei Heng Koh, Zhi Hao Ong, Haziratul Zakirah Binte Ghazali, Lorraine Hui En Tan, Yun Ting Ong, Clarissa Wei Shuen Cheong, Annelissa Mien Chew Chin, Stephen Mason, Lalit Kumar Radha Krishna
  BMC Medical Education.2020;[Epub]     CrossRef
• Have we forgotten to teach how to think?
  Damian Roland
  Emergency Medicine Journal.2017; 34(2): 68.     CrossRef
• Implementation of effective practices in health facilities: a systematic review of cluster randomised trials
  Emma R Allanson, Özge Tunçalp, Joshua P Vogel, Dina N Khan, Olufemi T Oladapo, Qian Long, Ahmet Metin Gülmezoglu
  BMJ Global Health.2017; 2(2): e000266.     CrossRef
• A qualitative study of self-evaluation of junior doctor performance: is perceived ‘safeness’ a more useful metric than confidence and competence?
  Damian Roland, David Matheson, Timothy Coats, Graham Martin
  BMJ Open.2015; 5(11): e008521.     CrossRef