Adult Gastroenterology Training Program

Evaluation

OVERVIEW

The goal of the Adult Gastroenterology Training Program is to produce subspecialists in Gastroenterology who can work independently in any clinical setting. Trainees are also expected to sit the subspecialty exams of The Royal College of Physicians of Canada, and to fulfil all CANMEDS role competencies. Their progress through the program is monitored in order to meet these objectives.

The McMaster University Gastroenterology Residency Training Program maintains a collegial atmosphere in which feedback is frequently exchanged between residents and faculty in order to promote excellence in resident education and resident performance. We want to ensure that every resident successfully achieves or exceeds rotation objectives and that every rotation meets or exceeds resident learning objectives. The evaluation process is instrumental in meeting these goals.

EVALUATION PROCESS

Faculty are expected to follow the official policy and procedures of the Postgraduate Medical Education (PGME) Office for the evaluation of residents’ performance. Appeal processes are defined by the PGME Office. The Evaluation Policy in its entirety is available on the PGME website: //fhs.mcmaster.ca/postgrad/documents/ResidentEvaluation1.pdf.

  1. At the beginning of the Rotation, the site coordinator / supervisor and the Resident should meet to discuss objectives and how the Resident will be evaluated, and in particular, should discuss the following:
    1. Delineate the Resident’s role during the rotation;
    2. Outline the duties and responsibilities expected of the Resident;
    3. Outline the goals and objectives of the rotation;
    4. Explain the structure and inter-relationships of the health care team; and
    5. Advise the Resident on what evaluation tools will be used in the evaluation process, how the ITER is completed and the timing of evaluations (including on-going informal feedback and the ITER)
  2. During the Rotation, there should be regular informal face-to-face feedback to the Resident (on a weekly basis). The Resident is responsible for scheduling a face-to-face (formative) evaluation at the mid-point of their rotation, so the Resident has an opportunity to address any deficiencies that may have been identified.
  3. At the end of the Rotation, the Clinical Supervisor should draw on feedback of other members of the health care team and any other evaluation forms utilized (e.g. Evaluation of Written Consultation Dictation in Ambulatory Care) to complete the ITER and then discuss the evaluation and the ITER with the Resident prior to the end of the Rotation. If the supervisor is not able to meet these deadlines and has not met with the Resident within 10 working days after the Rotation, the Clinical Supervisor should submit the ITER and send any supporting document to the Program Director.
  4. It is expected that the Resident will review the evaluation within 20 working days from the end of the Rotation on the One45 WebEval System.

All evaluations are site-, year-, and rotation-specific based on the CanMEDS competencies. Within each domain and for each goal and objective on the ITER, there may be several levels of competence identified. However, the overall (summative) evaluation on the ITER should indicate one of the following designations:

Satisfactory

Resident has successfully met the goals and objectives of the rotation

Provisional Satisfactory

Resident has demonstrated significant deficiencies in one or more of the RCSC competencies identified in the rotation objectives, or any other requirement of the rotation, and that while such deficiencies require remediation, they are not so severe to necessitate the Resident repeating the entire rotation. The Clinical Supervisor believes that the Resident can satisfy the deficient rotation objective(s) or requirement(s) during other rotations. These deficiencies often relate to non-Medical Expert CanMEDS domains.

Unsatisfactory

Resident has demonstrated significant deficiencies in one or more of the RCPSC competencies identified in the rotation objectives, or any other requirement, and the Clinical Supervisor believes that the rotation objective(s) or requirement(s) can only be reasonably met by remediation and having the Resident repeat the entire rotation. These deficiencies often relate to Medical Expert CanMEDS domains.

Incomplete

“Incomplete” indicates that the Clinical Supervisor has been unable to properly and fully evaluate the Resident because the Resident’s time spent on the rotation was insufficient, for whatever reason, e.g. illness, extenuating circumstances etc. As the rotation is incomplete, time will have to be made up to fulfil the requirements of the rotation. A designation of “incomplete” may be appropriate where the Resident has not spent at least 50% of the required time on the rotation. Even where a designation of “incomplete” is indicated, the Clinical Supervisor should complete the ITER in order to document the Resident’s time spent in the rotation and the Resident’s performance during that limited time.

Rotation In-Training Evaluation Reports (ITERS)
The ITER is the main modality of assessing resident knowledge and observed performance according to CanMEDS competencies. Residents are evaluated at the end of each rotation with a rotation-specific and year-specific ITER that encompasses all of the CanMEDS domains and follows each rotation description in this syllabus. Our program uses the one-45 computer-based evaluation system. Once completed, residents can review their ITERs electronically, provided they have no outstanding rotation evaluations to complete.

Mock Examinations
In preparation for the Royal College of Physicians and Surgeons Examination in Gastroenterology, trainees undergo a mock OSCE and written exam twice per year (November and May). This exam is intended to follow the format of the Royal College Examination in Adult Gastroenterology. The OSCE consists of several stations that test history taking, physical examination, endoscopic / histopathologic / radiologic interpretation, and clinical management of a wide variety of gastroenterologic problems. The stations are designed to assess all CanMEDS roles. Individualized instruction and feedback are provided after each station as part of formative feedback that helps learners become aware of any gaps that exist between their goals and their current knowledge, understanding, or skill and guides them through actions necessary to obtain the goals. OSCE stations are generated by the OSCE Coordinating Committee and reviewed by the OSCE Coordinator and the Program Director before being implemented. Written exams are prepared by 2 faculty members (1 luminal and 1 hepatology) and submitted to the Written Exam Coordinator for review and approval prior to implementation. This experience is invaluable preparation for the Royal College of Physicians and Surgeons of Canada licensing examinations, and also facilitates organized thinking around problems and ethical dilemmas in Gastroenterology. The results of these tests are reviewed and used as formative assessment tools to guide future learning objectives.

Resident Practice Audit Gastro-Enterology
Residents are responsible for maintaining a Procedural Skill Competency Log via the Resident Practice Audit Gastro-Enterology (RPAGE) Program. The RPAGE is designed to provide trainees with a point-of-care, peer-comparator practice audit tool. The purpose of the RPAGE program is to monitor and document the trainee’s progress and development in endoscopic skills. With the help of the endoscopic trainers, all trainees enter details of each procedure they complete in real-time. Anonymized trainee, patient and practice data are collected using touchscreen smartphones or desktop computer with automated data upload for data analysis and review by participants. The program allows trainees to objectively record key endoscopic quality indicators. There is also an evaluation tool built in the program that allows trainees to have their endoscopic performance objectively evaluated by their trainers on a regular basis. The RPAGE program allows trainees to review their own performance and compare this with their peers, promoting the identification of learning needs and objectives, as well as the basis for the development of targeted education programs. The Program Director can review procedure volumes, endoscopic quality indicators and evaluation results on a regular basis.

Progress Review
Each resident meets with the Program Director semi-annually to review progress, career goals, fellowship plans and other issues. Evaluations of all rotations are reviewed with residents at the semi-annual meetings. Particular areas of weaknesses and strengths are brought to the attention of the Site Coordinator and Mentor, to generate discussion among faculty and the resident involved. In this plan, plans for remedial help and supervision can be initiated promptly.

The clinical faculty also meet as a group once each year to discuss trainees’ performance on the wards, in the emergency department, in clinics, in endoscopy, and on the mock OSCE/written examinations. For each resident, completed online evaluations are reviewed and summarized by the Program Director. A broad and open discussion is initiated for each resident to counterbalance discrepant evaluations by individual faculty members. The discussion complements the online evaluations from each rotation, and generates a formal appraisal of each trainee’s strengths and weaknesses that is used to inform the FITER. These are then submitted to the Postgraduate Medical Education Office and then forwarded to the Royal College. Feedback from this meeting is also reviewed with the individual resident during the semi-annual meeting with the Program Director.

Final In-Training Evaluation Report
At the end of training, a final evaluation form (FITER) is prepared by the Program Director, with assistance from all faculty members and the Residency Training Program Committee. The FITER is forwarded to the Royal College and is used specifically in cases of borderline Royal College Examination results.

Allied Health Profession Evaluation of Resident
Evaluations from the health care team can provide feedback useful in guiding residents’ professional growth. Feedback from allied health professionals on residents’ competencies of interpersonal and communication skills and professionalism are obtained by means of a multi-source evaluation twice each year. These evaluations are very useful for formative feedback in residents’ professional development.

Peer Evaluation and Self-Assessment
360-degree assessment is a way of providing feedback about progress by placing the persons to be evaluated at the “hub of the wheel”. To provide a full-circle view of resident skills and abilities of interpersonal and communication skills, a self-assessment and a peer 360-degree evaluation form will be obtained twice per year. These evaluations encourage reflection and promote development of a self-improvement plan.

Evaluation of Written Consultation Dictation
To assess and improve the ability of residents to communicate effectively with referring physicians, written evaluation is formally evaluated during every ambulatory rotation by means of a Written Consultation Dictation Evaluation Form. It is the expectation that the resident on clinic block will review one consultation letter with an attending staff and have the evaluation form completed for each clinic rotation.

Rotation Evaluati on

Residents are encouraged to complete confidential Rotation Evaluations at the end of each rotation using the web-based evaluation system. The Residency Program Committee (RPC) reviews individual rotations and the rotation evaluations formally twice per year, but also on an ad hoc basis at its regular meetings in order to make adjustments as necessary. To maintain anonymity, only cumulative evaluation summaries are prepared for review by the Program Director and the RPC. Residents are free to bring concerns about rotations to the Program Director directly, and to table concerns for formal discussion by the RPC.

Faculty Evaluation

Residents are encouraged to complete a confidential faculty evaluation in respect of each member of the clinical teaching faculty with responsibility for the resident via the web-based evaluation system. Evaluation and feedback is expected to reflect the four broad domains of influence which the clinical faculty have with the residents: supervision; teaching; evaluation; and professional behaviour. At 6 month intervals, the Program Director receives and reviews cumulative summary evaluations for every individual faculty member, and each faculty member receives his/her summary evaluation as feedback. The average score obtained by their colleagues is also given. The summaries of all evaluations are forwarded to the Division Chief and to the Site Coordinators and Service Chiefs. These summaries are brought to the RPC for discussion at least once per year. Any concerns are reviewed by the Program Director and the Division Chief. These are then discussed with the individual faculty and the Service Chief at the hospital site. Depending on the nature of the concern, various measures are taken. This may simply require a discussion with the individual faculty concerned with on-going monitoring. If the concerns are more serious, changes would be made to the resident rotations to remove the residents from being taught by the individual faculty member. This would involve discussions with the Postgraduate Medical Education (PGME) Office, the Division Chief and the Department Chair according to the PGME policy.

PROMOTION PROCESS

Evaluations are reviewed with the Resident at his / her semi-annual meetings with the Program Director. Particular areas of weakness are brought to the attention of the Site Coordinator and the respective Mentor, to generate discussion among faculty and the Resident involved. In this way, plans for remedial help and supervision can be initiated promptly. Career counselling is also discussed. Resident progress is reported to the Residency Program Committee.

In the second half of each academic year, a special meeting of the Residency Program Committee is convened to discuss resident performance. Resident representatives do not attend this meeting, but all faculty members (including those who are not members of the Residency Program Committee) are invited to attend. For each resident, completed online evaluations are reviewed and summarized by the Program Director. A broad and open discussion is initiated for each resident to counterbalance discrepant evaluations by individual faculty members. Promotion of a Resident to the next academic level occurs if all rotation periods during the academic year have been completed with satisfactory evaluations. The Committee discussion forms the basis for the Program Director’s completion of FITER reports. These are then submitted to the Postgraduate medical Education Office and then forwarded to the College. Feedback from this special meeting is also reviewed with the individual resident by the Program Director.

APPEAL PROCESS

Only ITERs rated as “Unsatisfactory” or “Provisional Satisfactory” can be appealed. An appeal of an ITER can be made only on the basis that the Clinical Supervisor failed to follow the process set out in the Evaluation Policy by the PGME office, or on the basis that there are extraordinary mitigating personal circumstances that ought to be considered. All appeals must be made within 15 days after the Resident being sent the ITER.

The first level of appeal is made to the Program (Level 1). At the Program level, there is an emphasis on informal resolution.

  1. The first stage is an informal stage in which the Resident must discuss the ITER with the Clinical Supervisor who completed it and identify whatever additional information the Resident believes should be considered (e.g. external factors which influenced the Resident’s performance; identification of other individuals who could add an additional perspective on the Resident’s performance).
  2. Within 15 working days of the informal discussion between the Resident and the Clinical Supervisor, the Clinical Supervisor must either a) revise the ITER in which event the revised ITER becomes the official ITER, replacing the earlier one; or b) advise the Resident in writing that the ITER will remain unchanged.
  3. If the Resident is not satisfied with the review by the Clinical Supervisor, the Resident may proceed to the formal stage of the appeal process by notifying the Program Director in writing of his or her intention to do so. The notice must be delivered no later than 5 working days following receipt of the Clinical Supervisor’s decision. The appeal must document the perceived error in process.
  4. Upon receipt of written notice from the Resident, the Program Director will forward the appeal to the Residency Program Committee (RPC). The Program Director and the RPC will give the Resident an opportunity to meet with them and provide oral submissions and any additional documentation (i.e. evaluations, correspondence) relevant to the issues under appeal no later than 10 working days following receipt of the written notice. The Resident may be accompanied by a colleague, however ordinarily any oral submissions or presentations must be made by the Resident him/herself. The RPC will review all of the relevant documentation and meet with the Clinical Supervisor and other individuals if it deems necessary before making a decision.
  5. The RPC will issue a decision in writing with reasons and a copy will be provided to the PGME office.
    1. If the RPC decides that the evaluation was inaccurate or unfair, it may require that the evaluation be corrected or it may remove the evaluation from the file and allow a further period of evaluation under such terms as the RPC may require.
    2. If the RPC decides that there are compelling extenuating or compassionate circumstances that warrant an additional period of assessment and evaluation, it will permit the Resident to undergo an additional evaluation under such terms as the RPC may require.
    3. If the RPC concludes that the ITER was accurate and fair, the ITER should remain in the file and that there will be no additional assessment or evaluation.

If the issue is not resolved at the Program Level (Level 1), it can be directed to the PGME Office, which is Level 2. The PGME office will convene a meeting of the Appeal Review Board. Level 3 is the final level of appeal and at this point the Dean, Faculty of Health Sciences will strike a Tribunal, whose decision is final. The PGME Appeal Policy in its entirety is available on the PGME website: //fhs.mcmaster.ca/postgrad/documents/ResidentEvaluation1.pdf.