Being able to predict which residents will likely be unsuccessful on high-stakes exams would allow residency programs to provide early intervention.
To determine whether measures of clinical performance in clerkship (in-training evaluation reports) and first year of residency (program director ratings) predict pass-fail performance on the Medical Council of Canada Qualifying Exam Part II (MCCQE Part II).
Residency program directors assessed the performance of our medical school graduates (Classes 2004-2007) at the end of the 1st postgraduate year. We subsequently collected clerkship in-training evaluation reports for these graduates. Using a neutral third party and unique codes, an anonymous dataset containing clerkship, residency, and MCCQE Part II performance scores was created for our use. Data were analyzed using descriptive statistics, correlations, receiver operating characteristics, and the Youdin index. Regression was also performed to further study the relationship among the variables.
Complete data were available for 78.6% of the graduates. Of these participants, 94% passed the licensing exam on their first attempt. Receiver operating characteristics revealed that the area under the curve for clerkship in-training evaluation reports was 0.67 (p
The continuity of undergraduate to postgraduate training suggests that performance in medical school should predict performance later in residency.
The goal is to determine whether undergraduate performance is predictive of postgraduate performance.
Residency program directors assessed the performance of medical school graduates (Classes 2004-2006) at the end of the 1st postgraduate year. Measures of undergraduate performance were retrieved including grade point averages, clerkship in-training evaluation reports, and the total score on the Medical Council of Canada Part 1 exam.
Complete data were available for 242 (81.5%) graduates. Postgraduate performance consisted of two reliable factors (clinical acumen and human sensitivity) that explained 78% of the variance. Correlations between undergraduate and the two postgraduate measures were low (.03-.31).
Measures of undergraduate performance appear to be poor predictors of performance in residency that consisted of two primary dimensions (clinical acumen and human sensitivity).
Advocates of holistic assessment consider the ITER a more authentic way to assess performance. But this assessment format is subjective and, therefore, susceptible to rater bias. Here our objective was to study the association between rater variables and ITER ratings. In this observational study our participants were clerks at the University of Calgary and preceptors who completed online ITERs between February 2008 and July 2009. Our outcome variable was global rating on the ITER (rated 1-5), and we used a generalized estimating equation model to identify variables associated with this rating. Students were rated "above expected level" or "outstanding" on 66.4 % of 1050 online ITERs completed during the study period. Two rater variables attenuated ITER ratings: the log transformed time taken to complete the ITER [ß = -0.06, 95 % confidence interval (-0.10, -0.02), p = 0.002], and the number of ITERs that a preceptor completed over the time period of the study [ß = -0.008 (-0.02, -0.001), p = 0.02]. In this study we found evidence of leniency bias that resulted in two thirds of students being rated above expected level of performance. This leniency bias appeared to be attenuated by delay in ITER completion, and was also blunted in preceptors who rated more students. As all biases threaten the internal validity of the assessment process, further research is needed to confirm these and other sources of rater bias in ITER ratings, and to explore ways of limiting their impact.
A major goal of any evaluation is to demonstrate content validity, which considers both curricular content as well as the ability expected of learners. Whether evaluation blueprints should be published and the degree of blueprint transparency is controversial.
To examine the effect of blueprint publication on students' perceptions of the validity of the evaluation process.
This study examined students' attitudes towards the Renal Course evaluation before and after blueprint publication. There was no significant change in the course objectives, blueprint or evaluation between the two time periods. Students' attitudes were evaluated using a questionnaire containing four items related to evaluation. Also collected were the overall course ratings, minimum performance level (MPL) for evaluations and students' performance on each exam.
There were no significant differences in the MPL or evaluation scores between the two time periods. A significantly greater proportion of students perceived that the Renal Course evaluation was a fair test and was reflective of both important subject matter and the delivered curriculum. The increased satisfaction process did not appear to be a reflection of their overall satisfaction with the course as there was a trend towards reduced overall satisfaction with the course.
Publication of the evaluation blueprint appears to improve students' perceptions of the validity of the evaluation process. Further studies are required to identify the reasons for this attitude change. We propose that blueprint transparency drives both instructors teaching and student learning towards key educational elements.
When the University of Calgary implemented the clinical presentation (CP) curriculum in 1994, it was prospectively decided to administer the National Board of Medical Examiner's Comprehensive Basic Science Exam (CBSE) as a measure of students' basic science knowledge retention.
The exam performance from 2 classes (1995, 1996) of the previous system-based (SB) curriculum was compared to exam performance of 2 classes (2000, 2002) of the CP curriculum.
Data analyses employed 2 statistical models (covariate multiple linear regression and hierarchical mixed effects), and effect sizes were computed.
Differences between CBSE mean scores produced by students from the SB and CP curricula showed a curricular effect on students' retention of basic science knowledge. However, preexisting differences between groups were found to be in the small-to-medium range.
Evidence supporting the potential of schemes within a CP curriculum and their relation to basic science knowledge retention was observed. Effect size for the CP curriculum on students' retention of basic science knowledge was substantial; however, a notable part of that difference can be accounted for by extraneous and confounding factors. Further research utilizing more rigorous designs to investigate the relation between schemes and basic science retention is warranted.
Medical students are typically asked to complete course evaluations, but little is known about how students decide to rate courses. This study sought to examine the student feedback process by exploring the dimensionality of a course evaluation tool and examining the relationship between resulting factors and the overall rating of a course.
During the 2007-2008 academic year, all first- and second-year students were asked to provide feedback on various aspects of curricular content, delivery, and assessment for seven courses taught in the first two years of a clinical presentation curriculum. The authors examined the structure of the evaluation instrument using principal component factor analysis and used multiple linear regression to study the relationship between factors and overall course ratings.
Four stable and reliable factors were identified (assessment of students, small-group learning, basic science teaching, and teaching diagnostic approaches) that accounted for about 50% of the total variance in overall course ratings. Student assessment displayed the strongest association with overall course ratings, and for second-year students it was the only variable associated with overall course ratings.
Of the four factors, student assessment was by far the strongest predictor of overall course ratings, and this association strengthened over time. These results are consistent with the "peak-end rule" and "negativity dominance" for rating emotional experiences.
Despite there being considerable literature documenting learner distress and perceptions of mistreatment in medical education settings, these concerns have not been explored in-depth in Canadian family medicine residency programs. The purpose of the study was to examine intimidation, harassment and/or discrimination (IHD) as reported by Alberta family medicine graduates during their two-year residency program.
A retrospective questionnaire survey was conducted of all (n = 377) family medicine graduates from the University of Alberta and University of Calgary who completed residency training during 2001-2005. The frequency, type, source, and perceived basis of IHD were examined by gender, age, and Canadian vs international medical graduate. Descriptive data analysis (frequency, crosstabs), Chi-square, Fisher's Exact test, analysis of variance, and logistic regression were used as appropriate.
Of 377 graduates, 242 (64.2%) responded to the survey, with 44.7% reporting they had experienced IHD while a resident. The most frequent type of IHD experienced was in the form of inappropriate verbal comments (94.3%), followed by work as punishment (27.6%). The main sources of IHD were specialist physicians (77.1%), hospital nurses (54.3%), specialty residents (45.7%), and patients (35.2%). The primary basis for IHD was perceived to be gender (26.7%), followed by ethnicity (16.2%), and culture (9.5%). A significantly greater proportion of males (38.6%) than females (20.0%) experienced IHD in the form of work as punishment. While a similar proportion of Canadian (46.1%) and international medical graduates (IMGs) (41.0%) experienced IHD, a significantly greater proportion of IMGs perceived ethnicity, culture, or language to be the basis of IHD.
Perceptions of IHD are prevalent among family medicine graduates. Residency programs should explicitly recognize and robustly address all IHD concerns.
The objective of GEMS (General Emergency Medicine Skills) is to deliver an emergency skills program relevant to rural physicians. The cognitive component was disseminated by four interactive CD-ROM modules while the practice component consisted of skill practice on a human patient simulator (HPS) at a rural hospital. During the pilot year, 16 participants completed the program and filled in an evaluation survey to collect participant feedback. Modules facilitated maintenance of current knowledge, new knowledge acquisition, and confidence in dealing with emergencies. The HPS reinforced module knowledge, allowed practice of skills, and was necessary to fully benefit from the modules. The application process was easy and program administration was good. GEMS positively affected participants' delivery of emergency medicine and their willingness to continue the practice of rural emergency medicine. GEMS provides rural physicians relevant, interactive skills training at a rural setting.
To compare the academic performance of students who entered family medicine residency programs with that of students who entered other disciplines and discern whether or not family physicians are as academically talented as their colleagues in other specialties.
Retrospective quantitative study.
University of Calgary in Alberta.
Three graduating classes of students (2004 to 2006) from the University of Calgary medical school.
Student performance on various undergraduate certifying examinations in years 1, 2, and 3, along with third-year in-training evaluation reports and total score on the Medical Council of Canada Qualifying Examination Part I.
Complete data were available for 99% of graduates (N = 295). In the analysis, residency program (family medicine [n = 96] versus non-family medicine [n = 199]) served as the independent variable. Using a 1-way multivariate ANOVA (analysis of variance), no significant difference among any of the mean performance scores was observed (F(5289) = 1.73, P > .05). Students who entered family medicine were also well represented within the top 10 rankings of the various performance measures.
The academic performance of students who pursued careers in family medicine did not differ from that of students who chose other specialties. Unfounded negativity toward family medicine has important societal implications, especially at a time when the gap between the number of family physicians and patients seeking primary care services appears to be widening.
The family medicine clerkship at the University of Calgary is a 4-week mandatory rotation in the final year of a 3-year programme. Students are given the opportunity to experience rural practice by training at 1 of several rural practices.
To determine whether exposure to a rural educational experience changes students' likelihood of doing a rural locum or rural practice and whether student background and gender are related to these practice plans.
Clinical clerks from the Classes of 1996-2000, who trained at rural sites, responded to questionnaire items both before and after the rural educational experience. Responses to the questionnaire items and discipline of postgraduate training served as dependent variables. Student background and gender were independent variables.
As a result of the rural educational experience all students were more likely to do a rural locum. Compared to their urban-raised peers, students from rural backgrounds reported a significantly greater likelihood of doing a rural locum and practising in a rural community, irrespective of gender or participating in a rural educational experience. There was no relationship between background and career choice.
A rural educational experience at the undergraduate level increases the stated likelihood of students participating in rural locums and helps to solidify existing rural affiliations. Students with rural backgrounds have a more favourable attitude toward rural practice. This pre-post study provides further support for the preferential admission to medical school of students with rural backgrounds to help alleviate the rural physician shortage.