The quality of Resident Rotational Evaluations by Faculty
|Policy:||The quality of Resident Rotational Evaluations by Faculty|
|Policy Nr:||Sec 06 - 05|
|Target Review Date:||2017-12-31|
|Main Stakeholder:||Managing Director|
The Royal College of Physicians and Surgeons of Canada mandates documentation of evaluation of residents based on specific objectives at the end of each clinical rotation. Their site review processes require documentation of completion of these evaluations, however there is little or no evidence pertaining to the quality or utility of these evaluation processes.
The Department of Internal Medicine uses two CanMEDS compliant standard forms for evaluation of Core Internal Medicine residents. One has been developed by the Section of Critical Care Medicine, and the other by the Core Internal Medicine Residency Training Program. These evaluations are completed on-line through VENTIS. Most evaluations are completed by individual faculty members, but a substantial number are submitted as group evaluations of individual trainees. The group evaluation process is particularly subscribed by both the Clinical Teaching Unit and critical care rotations. Group evaluations are attributed to each individual faculty member whose name is identified on the group evaluation form.
We undertook analysis of all completed residents evaluations (3400) in the most recent six-year period to identify a simple group of indicators that could be used to improve the quality and utility of these evaluations.
For each individual attending physician, the following information was produced:
- The total number of individual or group evaluations completed.
- The distribution of checks in categories 1 through 4 on the evaluation form.
- The average number of characters or letters provided in the text section of the evaluation form.
- The number of times face-to-face discussions of the evaluations took place with the residents as indicated by check off in the evaluation form.
A review of this information identified a number of disturbing patterns:
- There were a number of individuals frequently attached to clinical services who had completed or participated in <9 resident evaluations.
- Many evaluators attributed with twenty or more evaluations had zero Category 1 or 2 check offs.
- Several evaluators checked off Category 3 for more than 90% of entries.
- A substantial number of evaluators checked off Category 4 more than 40% of the time.
- Many evaluators included an average of <40 characters of text per evaluation, and a few averaged zero.
- Many evaluators provided no evidence of face-to-face discussions with trainees concerning their evaluations.
In contrast some individuals provided frequent evaluations, occasionally identified performance weaknesses by checking off Category 1 or 2, described some residents as exceptional, but <40% consistently provided two more points of commentary, and frequently had face-to-face discussions with residents concerning their evaluations. It is reasonable to assume that evaluations provided by these individuals are more helpful than from those who check off only Category 3, never identify a performance weakness or an exceptional characteristic, provide little or no text commentary, and never discuss an evaluation with the trainee.
Conclusions and Recommendations
The overall poor quality of evaluations provided by attending physicians has significant negative consequences. It is not unusual for us to fail to identify areas of resident performance requiring substantial improvement until several rotations or even years of training have elapsed. Even when such performance problems are identified, contradictory evaluations supplied by less conscientious evaluators provide ample justification for appeal by residents who would benefit from remediation programs.
There are several possible reasons for poor performance on evaluations. Some individuals may regard them as having little value and see no harm in signing the form and drawing a line through the column of threes as evidence of completion. A number of faculty members, both junior and senior, have indicated that they are hesitant to criticize a resident's performance since it might be challenged, or the trainee may retaliate by giving them a poor evaluation. Others have indicated that they are uncomfortable discussing evaluations with trainees especially when there is an area of concern.
Some of these concerns may be reflected by the fact that evaluations done by groups are more than likely to contain the positive elements of evaluations which have been alluded to, since the process provides some level of anonymity.
Another reason for poor evaluation performance may relate to the fact that historically good evaluators are not rewarded.
Prior to the 2012-2014 cycle the Department of Internal Medicine process for evaluating faculty as teachers and role models was based exclusively on resident evaluation of their performance as clinicians and teachers.
For the 2012-2014 cycle, the process was modified so that resident evaluations of faculty constitutes 85% of the total score. The remaining fifteen percent are now based on an analysis of evaluation performance by faculty using a group of six negative indicators each valued at negative one mark, and a group of six positive indicators each valued at positive one mark. The baseline score will be nine and the total score will be out of fifteen.
For the 2014-2016 teaching evaluation cycle, we will include all evaluations submitted by faculty from 2010-2016. All individuals start with a base score of nine out of 15. The following scoring system will be applied:
(+1 each indicator)
- Participation in 13 or more individual or group evaluations.
- Evidence of at least one Category 1 or 2 checks among the accumulated evaluations.
- Greater than an average of 60 characters in the text portion of the evaluation.
- Greater than an average of 80 characters in the text portion of the evaluation.
- Indication of at least 2 face-to-face discussions with residents concerning their evaluations.
- Indication of at least 5 face-to-face discussions with residents concerning their evaluations.
(-1 each indicator)
- Less than eight completed individual or group evaluations in a two-year cycle.
- Zero Category 1 or 2 checks among the total evaluations.
- More than ninety percent Category 3 checks.
- More than forty percent Category 4 checks.
- An average of 40 or less characters of text.
- Absence of any face-to-face discussions with residents concerning their evaluations.
. The cumulative teaching evaluation performance score will consist of the cumulative score from the resident evaluations multiplied by 0.85 plus the new evaluator performance score out of fifteen.
This policy was approved unanimously at the Department of Internal Medicine Executive Committee meeting of Wednesday, January 12, 2011.
February 2017 update: positive indicator #1 and negative indicator #2 were prorated because WebEval is no longer being used as of April 7, 2016.
Policy will be reviewed in two years to take into account the use of the new program, VENTIS.