Outcomes-based education in the health professions has emerged as a priority for curriculum planners striving to align with societal needs. However, many struggle with effective methods of implementing such an approach. In this narrative, we describe the lessons learned from the implementation of a national, needs-based, outcome-oriented, competency framework called the CanMEDS initiative of The Royal College of Physicians and Surgeons of Canada.
We developed a framework of physician competencies organized around seven physician "Roles": Medical Expert, Communicator, Collaborator, Manager, Health Advocate, Scholar, and Professional. A systematic implementation plan involved: the development of standards for curriculum and assessment, faculty development, educational research and resources, and outreach.
Implementing this competency framework has resulted in successes, challenges, resistance to change, and a list of essential ingredients for outcomes-based medical education.
A multifaceted implementation strategy has enabled this large-scale curriculum change for outcomes-based education.
Significant impediments to effective emergency department (ED) teaching compromise what could otherwise be an excellent learning milieu. There is little literature to guide faculty development around specific emergency medicine teaching techniques. We determine what recognized experts in emergency medicine teaching consider to be the important clinical teaching behaviors that make them good teachers, the main impediments to good teaching in EDs, and important prerequisites for a good ED teacher.
This was a structured telephone survey with qualitative grounded-theory analysis. Participants were current Canadian emergency medicine teaching faculty who have won awards, been promoted, or received persistent excellent evaluations according to their ED teaching. Participants underwent a 45- to 60-minute standardized structured telephone interview. Interviews were transcribed and independently coded by 2 investigators using a grounded-theory approach. The codes were merged by consensus, and the data were recoded. Twenty percent of data were then coded by both investigators to estimate interrater reliability of final coding. Discrepancies were resolved by agreement.
Of 43 potential participants, 33 were still in practice, available, and willing to participate. Twelve ED-specific, practical, implementable strategies representing the general themes of learner-centeredness, active learning, individual relevance, and efficiency emerged. Participants collectively identified 6 significant impediments to teaching and 9 prerequisites to being an effective ED teacher.
Accomplished emergency medicine teaching faculty identify with common impediments to ED teaching yet are able to describe practical, easily implemented strategies that they believe make them good teachers. They also take advantage of basic prerequisites for good teaching.
One of the impediments to teaching professionalism is unprofessional behavior amongst clinical teachers. No method of reliably assessing the professional behavior of clinical teachers has yet been reported. The aim of this project was to develop and pilot such a tool. Thirty-four desirable professional behaviors in clinical teachers were identified. Medical students (n = 13) and medical educators (n = 30) rated their importance and validity. Based on the ratings, 16 behaviors in 4 dimensions were included in an assessment form that was piloted in the Department of Pediatrics at McGill University, with medical students (n = 94) rating the professionalism of their clinical faculty (n = 20). One hundred and ninety forms were returned with between 1 and 22 evaluations per faculty member. Scores ranged from 25 to 48 (maximum rating = 48, mean score = 42.7, SD = 6.29). A generalizability analysis was conducted; internal consistency was 0.89, and reliability for a mean of 8.23 ratings per faculty member was 0.53. Inter-rater reliability for one item was 0.11, potentially due to the context specificity of behavior or low frequency of unprofessional behaviors. Exploratory factor analysis revealed 3 factors with eigen values over 1. Assessment of the professionalism of clinical teaching faculty appears to be feasible, acceptable and reasonably reliable. The explicit evaluation of professional conduct in clinical faculty could encourage the maintenance of professional behaviors and potentially decrease the effects of negative role modeling and positively affect the hidden and informal curricula.
Assessment of clinical teaching by learners is of value to teachers, department heads, and program directors, and must be comprehensive and feasible.
To review published evaluation instruments with psychometric evaluations and to develop and psychometrically evaluate an instrument for assessing clinical teaching with linkages to the CanMEDS roles.
We developed a 19-item questionnaire to reflect 10 domains relevant to teaching and the CanMEDS roles. A total of 317 medical learners assessed 170 instructors. Fourteen (4.4 %) clinical clerks, 229 (72.3%) residents, and 53 (16.7%) fellows assessed 170 instructors. Twenty-one (6.6%) did not specify their position.
A mean number of eight raters assessed each instructor. The internal consistency reliability of the 19-item instrument was Cronbach's a?=?0.95. The generalizability coefficient (Ep(2)) analysis indicated that the raters achieved Ep(2) of 0.95. The factor analysis showed three factors that accounted for 67.97% of the total variance. The three factors together, with the variance accounted for and their internal consistency reliability, are teaching skills (variance?=?53.25s%; Cronbach's a?=?0.92), Patient interaction (variance?=?8.56%; Cronbach's a?=?0.91), and professionalism (variance?=?6.16%; Cronbach's a?=?0.86). The three factors are intercorrelated (correlations?=?0.48, 0.58, 0.46; p?
Dr. Ginsburg is professor, Department of Medicine, and scientist, Wilson Centre for Research in Education, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada. Dr. Eva is professor, Department of Medicine, and senior scientist, Centre for Health Education Scholarship, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada. Dr. Regehr is professor, Department of Surgery, and associate director, Centre for Health Education Scholarship, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada.
Although scores on in-training evaluation reports (ITERs) are often criticized for poor reliability and validity, ITER comments may yield valuable information. The authors assessed across-rotation reliability of ITER scores in one internal medicine program, ability of ITER scores and comments to predict postgraduate year three (PGY3) performance, and reliability and incremental predictive validity of attendings' analysis of written comments.
Numeric and narrative data from the first two years of ITERs for one cohort of residents at the University of Toronto Faculty of Medicine (2009-2011) were assessed for reliability and predictive validity of third-year performance. Twenty-four faculty attendings rank-ordered comments (without scores) such that each resident was ranked by three faculty. Mean ITER scores and comment rankings were submitted to regression analyses; dependent variables were PGY3 ITER scores and program directors' rankings.
Reliabilities of ITER scores across nine rotations for 63 residents were 0.53 for both postgraduate year one (PGY1) and postgraduate year two (PGY2). Interrater reliabilities across three attendings' rankings were 0.83 for PGY1 and 0.79 for PGY2. There were strong correlations between ITER scores and comments within each year (0.72 and 0.70). Regressions revealed that PGY1 and PGY2 ITER scores collectively explained 25% of variance in PGY3 scores and 46% of variance in PGY3 rankings. Comment rankings did not improve predictions.
ITER scores across multiple rotations showed decent reliability and predictive validity. Comment ranks did not add to the predictive ability, but correlation analyses suggest that trainee performance can be measured through these comments.
The interassessor reliability of faculty assessments was evaluated in a study aimed at enhancing the educational value of marking medical student case reports. After criteria for case report assessment were developed, three student write-ups were sent for evaluation to 17 volunteer faculty members with extensive teaching and assessment experience. Content analysis of their narrative comments and analysis of their global ratings of the student reports revealed marked disparities in the criteria the different assessors felt were important and an unacceptable spread in the ratings assigned to the reports. The study showed that the current case report assessment method was unreliable and that it should be replaced by a standardized assessment instrument.
The Teaching Scholars Program for Educators in the Health Sciences at McGill University, in Montreal, Quebec, was designed to promote the professional development of health science educators by increasing their expertise in developing and implementing educational programs and taking on leadership roles in education. This program, which was initiated in 1997 and is tailored to the individual needs of the participants, consists of participation in: two university courses; a monthly seminar; a research study or an educational project, consisting of curriculum design and evaluation; and faculty-wide faculty development activities. As of 2006, 34 scholars have completed this program. Outcome data indicate that the majority of teaching scholars have taken on new roles and responsibilities in medical education; maintained the changes implemented in their teaching practices; continued to participate in faculty development activities; and presented their work at educational meetings. A number of scholars have also applied successfully for educationally related grants and have published their educational projects. Five of the scholars have pursued advanced studies. This program, which aims to move beyond the improvement of teaching skills by providing a foundation for educational leadership and scholarship, resembles many others in its emphasis on independent study, peer support, and the maintenance of ongoing responsibilities. It is innovative in that scholars participate in university courses and are encouraged to attend an "outside" conference or course. The overall benefits of this program, as noted by the scholars, include increased knowledge and skills, introduction to a "community of practice," and new career paths and opportunities.
To establish an evidence-informed faculty development program.
Survey derived from a needs-assessment tool.
Department of Academic Family Medicine at the University of Saskatchewan, which is geographically dispersed across the province.
Full-time faculty members in the Department of Academic Family Medicine at the University of Saskatchewan.
Creation of an evidence-informed faculty development program.
The response rate was 77.3% (17 of 22). The data were stratified by 2 groups: faculty members with less than 5 years of experience and those with 5 or more years of experience. Those with less than 5 years of experience rated the following as their top priorities: teaching, developing scholarly activities, and career development. Those with 5 or more years of experience rated the following as their top priorities: administration and leadership, teaching, and information technology. Although there were differences in overall priorities, the 2 groups identified 17 out of 54 skills as important to faculty development.
The results of the needs-assessment tool were used to shape a dynamic, evidence-informed faculty development program with full-time faculty in the Department of Academic Family Medicine at the University of Saskatchewan. Future programs will continue to be dynamic, faculty-centred, and evidence-informed.
Work-hour restrictions have had a profound impact on surgical training. However, little is known of how work-hour restrictions may affect the future practice patterns of current surgical residents. The purpose of this study is to compare the anticipated career practice patterns of surgical residents who are training within an environment of work-hour restrictions with the current practice of faculty surgeons.
An electronic survey was sent to all surgery residents and faculty at 2 Canadian university-affiliated medical centers. The survey consisted of questions regarding expected (residents) or current (faculty) practice patterns.
A total of 149 residents and 125 faculty members completed the survey (50.3% and 52.3% response rates, respectively). A greater proportion of males were in the faculty cohort than in the resident group (77.6% vs 62.4%, p = 0.0003). More faculty than residents believed that work-hour restrictions have a negative impact on both residency education (40.8% vs 20.8%, p = 0.008) and preparation for a surgical career (56.8% vs 19.5%, p