Remediating a Struggling Learner: Due Process, Action Planning

---

Identifying and Assessing Underperformance

The thirteen years, at a minimum, of additional education and training is a significant amount of time that is invested by society in a physician. This physician is then expected to fulfill their fiduciary responsibility to that society for multiple years.  Undergraduate, graduate and resident education is underwritten by society in the form of endowments, grants and government funding.  The total per resident per year of fixed and variable costs have been estimated at $103,000 to 225,000, bringing the overall cost of educating a surgeon in the United States to an estimated $1.1 to 1.5 million.^{1, 2}

Rather than lose the multiple years of time investment as well as hundreds of thousands of dollars, attempts are made to identify and remediate the struggling resident as early as possible in their training.³   The residents themselves have also invested an incredible amount of time, energy and money into their education and have a stake in the educational process being successful.

Identifying the struggling resident can be difficult and doing so early can be an additional challenge.  Much of the study of the evaluation systems in place for resident physicians has been done attempting to identify what will highlight the struggling or at-risk resident at the earliest possible time in their residency program. The Surgery literature is rife with attempts to identify those residents that will struggle with medical knowledge (the ABSITE)^4-7, clinical reasoning skills (Surgical Trainee Assessment for Readiness exams)⁸ and technical skills (X-games skill tests)⁹, to name a few.  Approximately 22% of residents will have some manner of performance problem ¹⁰ and there is no one perfect way to identify the at-risk resident^{4, 10}.  Ancillary staff, peer, Junior resident and faculty evaluations in multiple environments are key 360-degree evaluation components.  Additionally, the use of rotation evaluations, self evaluations, reporting systems, ABSITE performance, conference attendance, faculty concerns, and clinical and operative evaluations^{10, 11, 12} are essential.  A robust advisor program, both at the faculty level as well as the resident level is also valuable¹³.  It is a strong evaluation system that will allow the earliest subjective impression of a struggling learner.   Once the learner is identified, remediation must be implemented.   A systematic manner in which to attempt to identify the cause and assist the rescue of the struggling learner is delineated here.¹⁴   The elements of the process of remediation also follows.

Subjective (recognition of underperformance)

There is a subjective impression that is made from direct observation of the learner and from formal, summative evaluations.  This subjective impression of a trainee who is not meeting the expectations of the program and may require closer supervision, additional technical skill or medical knowledge teaching and/or professionalism guidance is the beginning of the process.  Clinical teachers should trust their impressions and doubts should prompt further observation and documentation.¹⁴

Objective (documentation)

Gathering and documenting objective data should be thought of as establishing a diagnosis and “identifying discrepancy between the expected performance standard and the demonstrated performance, and then trying to establish the reason for underperformance.”¹⁵  The standards can be defined by the ACGME core competencies, the Milestones and Entrustable Professional Activities ^13,16.  Underperformance is a symptom not a diagnosis in itself and therefore requires further observation to better define issues and clarify its cause.  This will guide the appropriate response and follow up if needed.  It is extremely important that examples of underperformance be specific.  This will guide the clinical teachers in identifying and verifying the performance¹⁴.

Data is gathered by both indirect and direct observation of learners’ performances.  Direct observation is key for those procedural or technical skills.  Indirect observation can be collected by reviewing documentation in H&Ps, progress notes and operative notes¹⁴.

Data can also be collected having “diagnostic conversations” ¹⁷.  This allows identification of potential gaps in perceptions and can be used to gauge improvement in the remediation process.  It can also determine the attitude in which the learner approaches remediation and can improve the chance that remediation will be adhered to.  It can also help to identify any organic cause of underperformance.

Assessment (delineating the cause of underperformance)

Making a diagnosis based on assessment of the collected data is crucial.  There are two main types of difficulties: cognitive and non-cognitive (see table below)¹⁴.  The cognitive difficulties are insufficient knowledge and poor clinical reasoning, the non-cognitive are attitude, affective and health problems.  Each of these difficulties has multiple subsets and these are often interrelated and overlapping in the struggling learner.  Therefore, they must be identified correctly in order to best address the “diagnosis” or cause of underperformance.

Plan (developing an action plan)

The action plan details the necessary steps to take for improvement.  This should be mapped to the ACGME Core Competencies and very specific to the issue and to the resident.  Each step of the action plan should have activities that if successfully completed will demonstrate improvement. Each activity should reference which issue of underperformance is being addressed and what documentation will be required from supervisors (clinical evaluations, operative evaluations, module completion, ABSITE etc).  There should be a finite time frame for remediation and clear delineation of what successful remediation entails.

Remediation Process

Remediation is defined as “additional teaching above and beyond the standard curriculum”¹⁸. The ACGME has no specific definition for resident remediation and references it in regard to the Clinical Competency Committee (CCC) “recommending concerns to the Program Director regarding resident performance (which may result in remediation, probation, termination)” ¹⁹.

Planning a targeted remediation can be difficult as there is no standardized and universally accepted remediation processes that have been developed.  This is especially true in Surgery because of the need to include, potentially, technical skills.  Targeting a specific “diagnosis” in the remediation is crucial to being efficient in a remediation process and thus being successful.  The more well-defined, targeted and individualized the remediation, the more successful it is likely to be ^{14, 20}.

When a struggling learner is identified, the first step is to meet with the resident.  This is done to voice concern and can be a starting point for the “diagnostic conversations”¹⁷.  There are times that this is sufficient to alert the resident to deficiencies, potential improvement approach and remediation will not be required.  These meetings should be held between the Program Director and the resident with the Program Coordinator or other administrative assistant to document the process.  Documentation is required to show identification of and intervention to remedy difficulties and will be essential should the remediation be unsuccessful.

---

Notification

After verbal warnings, written warnings are the next step.  The first should be a formal ‘Notice of Concern’.  This would ideally set forth the issues and have a plan for intervention.  This can also set the time frame for gathering additional data.  Home Institutional policies regarding Academic Due Process should be referenced.   For the most part, GME policies at various institutions typically line up with the ACGME policies.  Therefore, should there be a grievance from the resident regarding remediation (or probation or termination), being able to present documentation and demonstrate the following the Academic Due Process ²¹ is invaluable.  Written notifications should also reference specific Program Policies not covered under the institutional policy and procedures.  These program-specific policies are usually noted in the institutional policy as “at the discretion of” the program.

If there is no improvement or ongoing concerns, then there should be to be a formal ‘Notice of Remediation’ with the deficiencies very simply spelled out and mapped to the Core Competencies.  With the Notice of Remediation, there should be a targeted remediation plan, the “Action Plan”, with clear requirements for remediation and clear definition of successful remediation.    All notifications of remediation require language that is consistent with programmatic and Institutional Policies, both for successful remediation and for remediation failures.

---

Action Plans

The exact requirements of the Targeted Action Plan are dependent upon the “diagnosis” of the underlying cause of the deficiencies.  To that end, remediation Action Plans should include those processes that may give a complete picture of the “diagnosis”.   All residents placed on remediation should be sent to the Wellness Committee to determine if time management issues, non-cognitive problems such as mental health, substance abuse or physical health issues exist and if referral to outside experts is required for thorough psychosocial evaluation.  Depending on the nature of the deficiency, a psychiatrist may be necessary for fitness for duty evaluations.  For instance, at UCSF Fresno our Wellness Committee includes a psychologist who is able to work with time management issues and identify potential learning disorders that require referral for neuropsychological evaluation.  Our department pays for neuropsychological evaluation of residents as long as it is part of the Action plan.  If there is concern on the part of the Wellness committee psychologist for fitness for duty, the resident is directed to a psychiatrist for a “forensic” evaluation of fitness for duty.

If a learning disorder or mental or physical health issue is identified, ongoing treatment of that issue becomes part of the remediation plan.

At the end of the remediation time, the decision to be made is whether there has been adequate improvement and remediation can be concluded, whether there is need for ongoing remediation because there has been some improvement or whether there is no improvement and ongoing concern requiring probation.  The possibility of probation and ultimately termination should be made clear early in the notice of remediation.

In applying the system above it is important to keep in mind the ACGME core competencies.  The remediation Action Plan should identify the domains involved in the deficiency.  There may be multiple and each part of the action plan should identify the domains that it addresses.

---

Documentation

Documentation of improvement must be done in a transparent manner.  Written evaluations from clinical teachers throughout the remediation period provide more timely feedback with which to evaluate progress of the resident, or lack thereof.  At our institution, the faculty is expected to give immediate verbal feedback as well as the written evaluation.  The progress of the resident is assessed by the faculty during monthly faculty meetings, bi-annual resident review and at the clinical competency committee.  At the end of the remediation period, it is the CCC that determines successful completion of remediation or unsuccessful remediation with recommendations for probation or termination.  Our feedback forms include the “areas needing improvement” or the indications for remediation as reference for the evaluators.  We are also currently trialing an online feedback system utilizing QR codes.  The QR code is specific to the resident and when scanned it opens the specific evaluation (clinical vs operative).  Once filled out by the faculty, it is automatically forwarded to the Program Coordinator.  This is in hopes of mitigating the lag time for written evaluations from those faculty who work at other hospitals or who have chronic issues with paperwork.

Ensuring that the resident is aware of their progress, ensuring the resident well-being, reviewing feedback and discussing additional means of improvement involves meetings between the resident and their faculty advisor as well as between the resident and Program Director.  Documentation of these meetings is imperative as well.  Faculty and resident signatures confirm the written documentation is agreed upon by both parties.

Having the resident acknowledge the feedback and evaluations is key to attempt to increase the acceptance of the resident of the process in order to increase their engagement in the process.  Resident buy-in increases the likelihood of success of remediation²⁰.

---

Scenarios

These are a few cases that our department has dealt with.  The templates and forms used in our remediation process are included as separate files.

1.Resident A (PGY1) scored at the 8^th percentile for the ABSITE.  Per our department policy, trainees who score below the 30^th percentile will be placed immediately on academic remediation.

• Objective data was gathered in the form of the ABSITE
• A diagnostic conversation was held between the PD and resident to determine the causation of poor performance on the ABSITE.
• Identification of the cause and an action plan was made.
  • Study suggestions
  • Source (textbook) suggestions
  • Reading plan
  • Advisor meetings
  • PD meetings
• Time frame of remediation delineated
  • One year pending repeat ABSITE
• Written notification of remediation and the institutional Academic Due Process Policy
• Determination of success of remediation

This was all fairly straightforward.  If, during the diagnostic conversation or identification of the cause, there is identification of a possible learning issue, referral to a psychologist or neuropsychological is made part of the action plan. This mapped to the ACGME core competency of:

IV.B.1.c) Medical Knowledge (MK)

Residents must demonstrate knowledge of established and evolving biomedical, clinical, epidemiological and social behavioral sciences, as well as the application of this knowledge to patient care.

2.Resident B is a PGY2 resident whose critical reasoning skills and communication skills with the healthcare team have been a concern.  These concerns were voiced at the bi-annual resident review.

• The subjective data has been voiced by the clinical teachers in the resident evaluation conference.
• Objective data was gathered by
  • Reviewing all prior evaluations and milestones to determine if this is an existing issue or newly developed issue
  • A diagnostic conversation that can lead to obtaining additional real-time evaluations and evaluation for possible substance abuse, mental health, health issues.
• The action plan included evaluation of critical reasoning skills and professionalism and referral to the institutional Well-being committee for further evaluation to determine if this was new onset.
  • A professionalism assessment was performed by the resident and compared to those completed by faculty.
• Time frame of remediation is delineated
  • Typically, for a new issue, I allow 3-4 weeks of evaluations to determine how pervasive the issue is.
  • If there is an issue of concern after gathering additional evaluations, I extend the remediation another 2-3 months to allow enough time to show sustained improvement.
  • This was an on-going issue with increased frequency and therefore remediation was instituted.
• Written notification of remediation and the institutional Academic Due Process Policy
• Determination of success of remediation

This remediation action plan mapped to the ACGME core competency of:

IV.B.1.a) Professionalism (PROF)

Residents must demonstrate a commitment to professionalism and an adherence to ethical principles. IV.B.1.a).(1) Residents must demonstrate competence in:

IV.B.1.a).(1).(a) compassion, integrity, and respect for others;

IV.B.1.a).(1).(b) responsiveness to patient needs that supersedes self-interest;

IV.B.1.a).(1).(c) respect for patient privacy and autonomy;

IV.B.1.a).(1).(d) accountability to patients, society, and the profession;

IV.B.1.a).(1).(e) respect and responsiveness to diverse patient populations, including but not limited to diversity in gender, age, culture, race, religion, disabilities, national origin, socioeconomic status, and sexual orientation;

IV.B.1.a).(1).(f) ability to recognize and develop a plan for one’s own personal and professional well-being; and,

IV.B.1.a).(1).(g) appropriately disclosing and addressing conflict or duality of interest.

IV.B.1.d) Practice-based Learning and Improvement (PBLI)

Residents must demonstrate the ability to investigate and evaluate their care of patients, to appraise and assimilate scientific evidence, and to continuously improve patient care based on constant self-evaluation and lifelong learning

IV.B.1.d).(1) Residents must demonstrate competence in:

IV.B.1.d).(1).(a) identifying strengths, deficiencies, and limits in one’s knowledge and expertise;

IV.B.1.d).(1).(b) setting learning and improvement goals;

IV.B.1.d).(1).(c) identifying and performing appropriate learning activities;

IV.B.1.d).(1).(d) systematically analyzing practice using quality improvement methods, and implementing changes with the goal of practice improvement;

IV.B.1.d).(1).(e) incorporating feedback and formative evaluation into daily practice;

IV.B.1.d).(1).(f) locating, appraising, and assimilating evidence from scientific studies related to their patients’ health problems; and,

IV.B.1.d).(1).(g) using information technology to optimize learning.

IV.B.1.e) Interpersonal and Communication Skills (ICS)

Residents must demonstrate interpersonal and communication skills that result in the effective exchange of information and collaboration with patients, their families, and health professionals.

IV.B.1.e).(1) Residents must demonstrate competence in:

IV.B.1.e).(1).(a) communicating effectively with patients, families, and the public, as appropriate, across a broad range of socioeconomic and cultural backgrounds;

IV.B.1.e).(1).(b) communicating effectively with physicians, other health professionals, and health-related agencies;

IV.B.1.e).(1).(c) working effectively as a member or leader of a health care team or other professional group;

IV.B.1.e).(1).(d) educating patients, families, students, residents, and other health professionals;

IV.B.1.e).(1).(e) acting in a consultative role to other physicians and health professionals; and,

IV.B.1.e).(1).(f) maintaining comprehensive, timely, and legible medical records, if applicable.

IV.B.1.e).(2) Residents must learn to communicate with patients and families to partner with them to assess their care goals, including, when appropriate, end-of-life goals.

IV.B.1.f) Systems-based Practice (SBP)

Residents must demonstrate an awareness of and responsiveness to the larger context and system of health care, including the social determinants of health, as well as the ability to call effectively on other resources to provide optimal health care.

IV.B.1.f).(1) Residents must demonstrate competence in:

IV.B.1.f).(1).(a) working effectively in various health care delivery settings and systems relevant to their clinical specialty;

IV.B.1.f).(1).(b) coordinating patient care across the health care continuum and beyond as relevant to their clinical specialty;

IV.B.1.f).(1).(c) advocating for quality patient care and optimal patient care systems;

IV.B.1.f).(1).(d) working in interprofessional teams to enhance patient safety and improve patient care quality;

IV.B.1.f).(1).(e) participating in identifying system errors and implementing potential systems solutions;

IV.B.1.f).(1).(f) incorporating considerations of value, cost awareness, delivery and payment, and risk-benefit analysis in patient and/or population-based care as appropriate; and,

IV.B.1.f).(1).(g) understanding health care finances and its impact on individual patients’ health decisions.

IV.B.1.f).(2) Residents must learn to advocate for patients within the health care system to achieve the patient’s and family’s care goals, including, when appropriate, end-of-life goals.

3.Resident C is a PGY 1 who throughout the first portion of the academic year had multiple verbal concerns brought up by faculty and residents regarding professionalism.  The examples ranged from inappropriate touching of a male CRNA, not notifying her senior resident of important information regarding patient care, disrespecting senior residents, and disregarding the orders of senior residents and substituting her own judgement in patient care.

• The subjective data has been voiced by the clinical teachers and peers to faculty and PD. The resident issue was discussed by the CCC and determination made that immediate remediation was necessary due to the gravity of the concerns.
• Objective data was gathered by
  • Reviewing all prior evaluations. For this resident, there was a paucity of formal evaluations and no milestones, therefore real-time written evaluations were deemed mandatory
  • A diagnostic conversation led to obtaining additional evaluations for possible substance abuse, mental health, health issues. This resident was sent for forensic psychiatric fitness for duty evaluation.
  • The resident was required to fill out a professionalism self-assessment tool, which was also completed by multiple clinical teachers. This allowed a more exact discussion of where her perceptions and those of her clinical teachers differed.
• The action plan included evaluation of professionalism and a required Professionalism module and referral to the institutional Well-being committee for further support.
• Time frame of remediation is delineated
  • I allowed 4 weeks of evaluations to determine how pervasive the issue was.
  • The full remediation, once an issue was fully documented was 8 weeks.
• Written notification of remediation and the institutional Academic Due Process Policy
• Determination of success of remediation
  • This resident continued to have poor evaluations with many of the same concerns
  • The resident was non-compliant with the terms of the remediation.
• The resident was placed on probation with many of the same action plans plus additional oversight
• There was no improvement and continued issues and the resident was terminated.

Remediation is not always successful and therefore consideration must be made for the next step in the academic due process.  Probation and termination must be on the table at the beginning of the remediation in order to emphasize the gravity of the situation.  The ACGME competencies that we mapped this to:

IV.B.1.a) Professionalism (PROF)

Residents must demonstrate a commitment to professionalism and an adherence to ethical principles. IV.B.1.a).(1) Residents must demonstrate competence in:

IV.B.1.a).(1).(a) compassion, integrity, and respect for others;

IV.B.1.a).(1).(b) responsiveness to patient needs that supersedes self-interest;

IV.B.1.a).(1).(c) respect for patient privacy and autonomy;

IV.B.1.a).(1).(d) accountability to patients, society, and the profession;

IV.B.1.a).(1).(e) respect and responsiveness to diverse patient populations, including but not limited to diversity in gender, age, culture, race, religion, disabilities, national origin, socioeconomic status, and sexual orientation;

IV.B.1.a).(1).(f) ability to recognize and develop a plan for one’s own personal and professional well-being; and,

IV.B.1.a).(1).(g) appropriately disclosing and addressing conflict or duality of interest.

IV.B.1.d) Practice-based Learning and Improvement (PBLI)

Residents must demonstrate the ability to investigate and evaluate their care of patients, to appraise and assimilate scientific evidence, and to continuously improve patient care based on constant self-evaluation and lifelong learning

IV.B.1.d).(1) Residents must demonstrate competence in:

IV.B.1.d).(1).(a) identifying strengths, deficiencies, and limits in one’s knowledge and expertise;

IV.B.1.d).(1).(b) setting learning and improvement goals;

IV.B.1.d).(1).(c) identifying and performing appropriate learning activities;

IV.B.1.d).(1).(d) systematically analyzing practice using quality improvement methods, and implementing changes with the goal of practice improvement;

IV.B.1.d).(1).(e) incorporating feedback and formative evaluation into daily practice;

IV.B.1.d).(1).(f) locating, appraising, and assimilating evidence from scientific studies related to their patients’ health problems; and,

IV.B.1.d).(1).(g) using information technology to optimize learning.

IV.B.1.e) Interpersonal and Communication Skills (ICS)

Residents must demonstrate interpersonal and communication skills that result in the effective exchange of information and collaboration with patients, their families, and health professionals.

IV.B.1.e).(1) Residents must demonstrate competence in:

IV.B.1.e).(1).(a) communicating effectively with patients, families, and the public, as appropriate, across a broad range of socioeconomic and cultural backgrounds;

IV.B.1.e).(1).(b) communicating effectively with physicians, other health professionals, and health-related agencies;

IV.B.1.e).(1).(c) working effectively as a member or leader of a health care team or other professional group;

IV.B.1.e).(1).(d) educating patients, families, students, residents, and other health professionals;

IV.B.1.e).(1).(e) acting in a consultative role to other physicians and health professionals; and,

IV.B.1.e).(1).(f) maintaining comprehensive, timely, and legible medical records, if applicable.

IV.B.1.e).(2) Residents must learn to communicate with patients and families to partner with them to assess their care goals, including, when appropriate, end-of-life goals.

4.Resident D was a transfer into the program in his PGY2 year.  By the end of his PGY 3 year it was noted that despite 99^th percentile on the ABSITE, his knowledge did not translate to the clinical arena and became a concern that as he became a chief resident, he would be unable to make safe decisions.  In addition, there were concerns regarding his technical skills.

• The subjective data has been voiced by the clinical teachers during resident review and CCC. The resident issue was discussed by the CCC and determination made that remediation was necessary.
• Objective data was gathered by
  • Reviewing all prior evaluations. These showed that the resident was below his peers in most evaluations.
  • A diagnostic conversation led the faculty to believe that with additional work to make the resident verbalize consideration of worst case diagnoses via pushing for a robust differential diagnosis. While this takes some dedication of a very busy faculty pool, it is imperative.
• The action plan included evaluations in real-time with faculty feedback regarding thought process in diagnosis and treatment decision-making and working with their advisor in the surgical skills lab on a bi-weekly basis.
• Time frame of remediation is delineated
  • Remediation was 3 months.
• Written notification of remediation and the institutional Academic Due Process Policy
• Determination of success of remediation
  • This resident continued to have evaluations with many of the same concerns but was noted to have made some improvement.
  • The resident was continued on remediation for an additional amount of time.
• The resident was found to have made some additional improvement but faculty felt it was insufficient to allow him to practice independently.
• The resident was kept on remediation and repeated the clinical year.
  • That additional year was delineated to the resident’s weaknesses.
• There was improvement and the resident was promoted and eventually graduate. He was able to successfully pass the QE and CE.  He also successfully performed as a safe surgeon in practice.

This remediation mapped to the ACGME core  competencies:

IV.B.1.b) Patient Care and Procedural Skills (PC)

IV.B.1.b).(1) Residents must be able to provide patient care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health.

IV.B.1.b).(2) Residents must be able to perform all medical, diagnostic, and surgical procedures considered essential for the area of practice.

IV.B.1.d) Practice-based Learning and Improvement (PBLI)

Residents must demonstrate the ability to investigate and evaluate their care of patients, to appraise and assimilate scientific evidence, and to continuously improve patient care based on constant self-evaluation and lifelong learning

IV.B.1.d).(1) Residents must demonstrate competence in:

IV.B.1.d).(1).(a) identifying strengths, deficiencies, and limits in one’s knowledge and expertise;

IV.B.1.d).(1).(b) setting learning and improvement goals;

IV.B.1.d).(1).(c) identifying and performing appropriate learning activities;

IV.B.1.d).(1).(d) systematically analyzing practice using quality improvement methods, and implementing changes with the goal of practice improvement;

IV.B.1.d).(1).(e) incorporating feedback and formative evaluation into daily practice;

IV.B.1.d).(1).(f) locating, appraising, and assimilating evidence from scientific studies related to their patients’ health problems; and,

IV.B.1.d).(1).(g) using information technology to optimize learning