Overconfidence of medical residents to perform entrustable professional activities in a General Surgery Residency Program

Abstract

Introduction

Competency-based medical education (CBME) has been implemented in postgraduate medical education, and Entrustable Professional Activities (EPAs) seem appropriate for operationalizing it. However, the association between residents´ self-perceived confidence in performing EPAs and entrustment levels assigned by a Clinical Competence Committee (CCC) must be better understood. We aimed to compare general surgery residents' self-assessment of confidence to perform EPAs and entrustment levels assigned by a CCC.

Material and methods

Cross-sectional study conducted in the General Surgery Residency Program at Santa Casa BH Hospital between March and May 2023. Residents self-assessed their confidence to perform 16 EPAs using a 5-point entrustment scale. Confidence levels were compared to entrustment levels assigned by the CCC. A comparison was performed separately for each EPA and by EPA characteristics (clinical surgical vs. procedural and simple x complex).

Results

There was a significant difference between the self-assessed confidence levels and the autonomy levels assigned by the CCC in nine of the 16 EPAs. Residents&apos; self-assessment levels were higher than those assigned by the CCC for clinical surgical EPAs (4.3 ± 0.8 vs. 3.2 ± 1.4, respectively; p < 0.001) and simple EPAs (2.8 ± 1.0 vs. 2.2 ± 0.8, respectively; p = 0.023). In procedural and complex EPAs, this difference was not statistically significant (p > 0.05).

Conclusion

Residents tended to overestimate their level of autonomy, especially for clinical-surgical EPAs and simpler EPAs.

Resumen

Introducción

La educación médica basada en competencias (EMBC) se ha implementado en la educación médica de posgrado, y las Actividades Profesionales Confiables (EPA) parecen apropiadas para su operacionalización. Sin embargo, se debe comprender mejor la asociación entre la confianza autopercibida de los residentes en la realización de EPA y los niveles de confianza designados por un Comité de Competencia Clínica (CCC). Nuestro objetivo esomparar la autoevaluación de la confianza de los residentes de cirugía general para realizar EPA y los niveles de confianza designados por un CCC.

Materiales y métodos

Estudio transversal realizado en el Programa de Residencia de Cirugía General del Hospital Santa Casa BH entre marzo y mayo de 2023. Los residentes se autoevaluaron en relación a su confianza para realizar 16 EPA utilizando una escala de confianza de 5 puntos. Los niveles de confianza se compararon con los niveles de confianza designados por la CCC. Se realizó una comparación por separado para cada EPA y por características de la EPA (clínica quirúrgica vs. de procedimiento y simple x compleja).

Resultados

Hubo una diferencia significativa entre los niveles de confianza autoevaluados y los niveles de autonomía designados por el CCC en nueve de las 16 EPA. Los niveles de autoevaluación de los residentes fueron más altos que los designados por el CCC para los EPA clínicos quirúrgicos (4,3 ± 0,8 vs. 3,2 ± 1,4, respectivamente; p < 0,001) y los EPA simples (2,8 ± 1,0 vs. 2,2 ± 0,8, respectivamente; p = 0,023). En las EPAs de procedimientos quirúrgicos y más complejas, la diferencia no fue estadísticamente significativa (p > 0,05).

Conclusión

Los residentes tendieron a sobreestimar su nivel de autonomía, especialmente para las EPA clínico-quirúrgicas y las EPA más simples.

Introduction

Entrustable Professional Activities (EPAs) are units of professional practice that correspond to the core clinical responsibilities or tasks of a particular profession or specialty. Using entrustment decisions to assess the level of autonomy of medical residents to perform a set of EPAs helps implement competency-based assessment in medical residency training.¹

In EPA-based assessment, faculty members and preceptors regularly assign more complex responsibilities to trainees as they acquire the necessary competencies, increasing their autonomy.¹ An appropriate entrustment process ensures a safe learning environment for residents and improves patient safety.²

The assignment of responsibilities to medical residents is already implicitly present in residency training programs, but it is usually made heterogeneously and non-systematically. In an EPA-based residency program, a Clinical Competence Committee (CCC) provides structured entrustment decisions to formally certify that a trainee has passed a given threshold that allows for decreased supervision.³ The CCC is comprised of a group of preceptors who analyze evaluations conducted by different preceptors throughout a rotation period and entrust residents to perform the EPAs.⁴ A frequently used entrustment scale was developed by Ten Cate and Scheele⁵ and is divided into five levels representing what a trainee is allowed to¹: be present and observe but not perform²; perform the EPA under direct and proactive supervision in the same environment³; perform the EPA without the supervisor present in the same environment but quickly accessible (indirect supervision)⁴; perform the EPA without supervision⁵; supervise novice learners. For most EPAs, entrustment level 4 is expected at the end of the training program, representing the readiness for unsupervised execution of that specific professional activity.

Self-assessment can be defined as the “process of interpreting data about our performance and comparing it to an explicit or implicit standard”.⁶ Although previous studies have shown poor self-assessment accuracy, particularly among those least skilled,⁷ it seems essential for the appropriate generation of learning goals.⁸ In residency training, self-assessment can promote residents' reflection, fostering their development of the ability to identify possible learning gaps, plan their studies, and increase their confidence in working in public and private health services.⁹

Analyzing the alignment between residents' self-perceived confidence and the entrustment levels determined by the CCC, we can estimate the calibration of self-assessment. This estimation might identify eventual gaps between the preceptors' perception of tasks that seem ready to be assigned to the residents within a given level of confidence and the residents' perceived level of preparedness. Miscalibration might signal friction that can either stimulate or inhibit learning.¹⁰ We aimed to analyze the calibration of general surgery residents' self-assessed confidence compared to the entrustment levels defined by the CCC.

Methods

Study design

This cross-sectional study, conducted between March and May 2023, compares residents´ self-assessed confidence in performing a given set of EPAs with entrustment levels defined by the CCC.

Setting

The study was conducted at Santa Casa de Belo Horizonte (SCBH), Minas Gerais, Brazil. The 3-year General Surgery Residency Program at SCBH had 27 residents enrolled at the time of the study. Residents´ level of autonomy is assessed for 16 EPAs, described in Supplementary Material, and the expected level of entrustment differs by year of training and professional activity. The preceptors use a mobile app (ResLife®) to register workplace-based assessments of residents´ performance. In each assessment, preceptors assign an ad hoc entrustment level for that given EPA and provide written feedback. After a three-month rotation period, a CCC composed of eight surgical team coordinators assigned entrustment levels for each resident, for each EPA, after carefully reviewing all ad-hoc entrustment levels and feedback provided by the preceptors during that period. All CCC members have received previous training on EPA-based assessment and at the time of the study had at least one year of experience at the CCC. The residents receive a final report with the CCC-determined entrustment levels and recommendations for improvement.

Participants

The study included residents regularly enrolled in the General Surgery Residency Program at SCBH between 2021 and 2023. All participants signed informed consent. Residents with irregular attendance during the study period due to medical leave or external rotations were excluded. Ethical approval was obtained from the Santa Casa BH Ethics Committee (CAAE#61638122.0.0000.5138).

Procedures

Residents completed a single self-assessment after three months of training, in which they provided their self-perceived entrustment level for each of the 16 EPAs of the General Surgery Medical Residency Program, described in Supplementary Material. Information on the confidence level determined by the CCC was obtained from academic registries at SCBH. The residents completed their self-assessment form before receiving the final report from the CCC.

Statistical analysis

The confidence levels in self-assessment were compared to those determined by the CCC using Student's t-test. This analysis was stratified by year of training and by EPA category (clinical surgical vs. procedural and simple vs. complex).

Analysis of variance (ANOVA) was used to analyze the interaction between factors such as year of training and EPA type (clinical surgical vs. procedural and simple vs. complex) and the estimated difference between self-assessed performance and that determined by the CCC. A p-value of 0.05 was set to determine statistical significance.

EPAs were classified as “clinical surgical” or “procedural” according to the task characteristics. The following EPAs were considered as “clinical surgical”: EPA 1 - “Admitting a Surgical Patient,” EPA 2 - “Providing Preoperative Patient Care”, and EPA 3 - “Providing Postoperative Patient Care.” The “procedural” EPAs were EPA 12 - “Performing Urinary Catheterization” and EPA 13 - “Obtaining a Central Venous Access / Performing Venous Dissection.”

The definition of the complexity of the EPAs was based on the entrustment levels expected for each resident year. EPAs for which a higher level of entrustment was expected throughout the training years were EPA 5 - Surgical Treatment of Patients with Abdominal Wall Defect and EPA 7 - Treatment of Patients with Acute Appendicitis. Activities considered more complex, for which greater autonomy was expected only in the third year, were those related to EPA 11 - Surgical Approach to Patients with Gastrointestinal Tract Cancer and EPA 15 - Surgical Approach to Patients with Morbid Obesity.

Results

Twenty-six eligible residents agreed to participate (nine first-year residents (PGY-1)), eight second-year residents (PGY-2), and nine third-year residents (PGY-3). Thirteen of the participants were female. During the study period, a total of 572 assessments were performed (mean 22 per resident), with 174 assessments for PGY-1 residents (mean 19.3 per resident), 177 for PGY-2 residents (mean 22.1 per resident), and 221 for PGY-3 residents (mean 24.6 per resident). Clinical surgical EPAs accounted for 400 (30.5%) assessments, while procedural EPAs accounted for 227 (17.3%) assessments.

When comparing the confidence levels assigned by the Clinical Competence Committee (CCC) with the self-assessment completed by PGY-1 residents, 65 (45.1%) self-assessments showed a higher confidence level than that attributed by the CCC. (Table 1) In comparison, 13 (9.0%) EPAs were self-assessed with lower confidence levels than the CCC. Sixty-four assessments showed the same level of confidence between self-assessment and the CCC.

Table 1.

Comparison between Confidence Levels determined by the CCC and Self-assessment Confidence Levels for first-year General Surgery Residents.

Note: the yellow cells represent a pair where the resident's self-assessed entrustment level is higher than the one determined by the CCC. The green cells represent a pair where the resident's entrustment level matched the CCC's. The red cells represent a pair where the resident's entrustment level was lower than the one determined by the CCC. N/A: the number of assessments was inadequate for a CCC entrustment decision.

Regarding the number of PGY-1 residents evaluated, 5 (55.6%) self-assessed themselves with a lower confidence level than that attributed by the committee in at least one EPA. Eight residents self-assessed themselves with a higher level than the CCC in at least one EPA. In total, 78 (54.9%) of the compared assessments exhibited differences between the residents' perceptions and those of the preceptors.

Analyzing the comparison of confidence levels between PGY-2 residents self-assessment and those attributed by the CCC, we observe that 21 (16.4%) EPAs were self-assessed with lower confidence levels than those attributed by the CCC, 55 (42.97%) EPAs were self-assessed with higher levels than those attributed by the CCC, and 36 (28.1%) EPAs showed the same level of confidence in both forms of assessment (Table 2). Concerning the residents, 7 (87.5%) self-assessed themselves with lower confidence levels than those attributed by the CCC in at least one EPA, and all residents self-assessed themselves with higher levels than those attributed by the CCC in at least one activity. Overall, 76 (67.9%) assessments showed differences between the self-assessed levels and those attributed by the preceptors.

Table 2.

Comparison between Confidence Levels determined by the CCC and Self-assessment Confidence Levels for second-year General Surgery Residents.

Note: the yellow cells represent a pair where the resident's self-assessed entrustment level is higher than the one determined by the CCC. The green cells represent a pair where the resident's entrustment level matched the CCC's. The red cells represent a pair where the resident's entrustment level was lower than the one determined by the CCC. N/A: the number of assessments was inadequate for a CCC entrustment decision.

Comparing the confidence levels assigned by the CCC and the PGY-3 residents´ self-assessed levels, we found that 16 (11.1%) activities were self-assessed with lower confidence levels than those attributed by the CCC, 44 (30.6%) EPAs were self-assessed with higher levels than those attributed by the CCC, and 64 (44.4%) self-assessments matched the levels defined by the committee (Table 3). Regarding the residents, 5 (55.6%) self-assessed themselves with lower confidence levels than those attributed by the CCC in at least one activity, and all 9 (100%) residents self-assessed themselves with higher levels than those defined in the Committee meeting in at least one EPA. Overall, 60 (48.4%) assessments showed differences between the residents' and preceptors' perspectives.

Table 3.

Comparison between Confidence Levels determined by the CCC and Self-assessment Confidence Levels for third-year General Surgery Residents.

Note: the yellow cells represent a pair where the resident's self-assessed entrustment level is higher than the one determined by the CCC. The green cells represent a pair where the resident's entrustment level matched the CCC's. The red cells represent a pair where the resident's entrustment level was lower than the one determined by the CCC. N/A: the number of assessments was inadequate for a CCC entrustment decision.

Table 4 compares the self-assessed entrustment levels and those determined by the CCC stratified by EPA. For half of the EPAs, there was a statistically significant difference between the student's perception and that of the CCC, with the residents' confidence consistently higher than that of the CCC. Effect sizes ranged from 0.58 to 1.05.

Table 4.

Comparisons of average confidence levels in self-assessment and Clinical Competence Committee assessment of residents/fellows from the three years of the Program for each EPA.

	Minimum	Maximum	Mean	Standard Deviation	Valor p⁎	d (Cohen)
EPA 1 (AA)	2,00	5,00	4,46	0,81	< 0,001	1,00
EPA 1 (CCC)	0,00	5,00	3,35	1,35
EPA 2 (AA)	2,00	5,00	4,38	0,85	< 0,001	1,05
EPA 2 (CCC)	0,00	5,00	3,19	1,36
EPA 3 (AA)	2,00	5,00	4,12	0,91	0,003	0,89
EPA 3 (CCC)	0,00	5,00	3,00	1,52
EPA 4 (AA)	2,00	5,00	3,31	1,01	0,099	0,47
EPA 4 (CCC)	0,00	5,00	2,81	1,13
EPA 5 (AA)	1,00	5,00	2,92	1,09	0,271	0,31
EPA 5 (CCC)	2,00	4,00	2,65	0,56
EPA 6 (AA)	2,00	5,00	3,58	1,14	0,017	0,69
EPA 6 (CCC)	2,00	5,00	2,88	0,86
EPA 7 (AA)	1,00	5,00	2,69	1,05	0,007	0,78
EPA 7 (CCC)	0,00	4,00	1,77	1,31
EPA 8 (AA)	2,00	5,00	3,23	1,24	0,088	0,48
EPA 8 (CCC)	0,00	5,00	2,69	0,97
EPA 9 (AA)	1,00	5,00	2,81	1,27	0,026	0,64
EPA 9 (CCC)	0,00	4,00	2,08	1,02
EPA 10 (AA)	1,00	3,00	2,19	0,75	0,533	0,17
EPA 10 (CCC)	0,00	4,00	2,04	1,00
EPA 11 (AA)	1,00	3,00	2,00	0,69	0,241	0,33
EPA 11(CCC)	0,00	4,00	1,69	1,12
EPA 12 (AA)	2,00	5,00	4,12	1,03	0,143	0,41
EPA 12 (CCC)	0,00	5,00	3,65	1,20
EPA 13 (AA)	2,00	5,00	3,96	1,11	0,170	0,39
EPA 13 (CCC)	0,00	5,00	3,46	1,45
EPA 14 (AA)	2,00	5,00	4,04	1,04	0,043	0,58
EPA 14 (CCC)	0,00	5,00	3,27	1,56
EPA 15 (AA)	1,00	3,00	2,08	0,80	0,050	0,56
EPA 15 (CCC)	0,00	4,00	1,54	1,10
EPA 16 (AA)	1,00	5,00	2,96	1,46	0,027	0,63
EPA 16 (CCC)	0,00	5,00	2,00	1,57

⁎

Student t-test.

In the stratified analysis by EPA type, a statistically significant difference between residents´ self-assessment and CCC entrustment levels was observed for “surgical clinical” EPAs (4.3 ± 0.8 vs. 3.2 ± 1.4, respectively; p < 0.001). For “Procedural” EPAs (EPAs 12 and 13), the difference was not statistically significant (4.0 ± 1.0 vs. 3.6 ± 1.2, respectively; p = 0.134).

Regarding the complexity of the EPA, a significant difference was observed between the residents´ self-assessed confidence and the confidence estimated by the CCC only for simple EPAs (2.8 ± 1.0 vs. 2.2 ± 0.8, respectively; p = 0.023). For complex EPAs, the difference was not statistically significant (2.0 ± 0.7 vs. 1.6 ± 1.1, respectively; p = 0.102).

Self-assessed and CCC entrustment levels increased progressively over the three years of the residency training program. Still, there was no interaction between the year of residency and the difference between entrustment levels estimated by the CCC and those self-assessed by the residents (Fig. 1).

Figure 1.

Confidence Level determined by the Clinical Competence Committee (CCC)

and in Self-assessment

for each EPA, stratified by year of training.

Discussion

We observed that residents' self-assessment of their entrustment levels for surgical EPAs consistently exceeded those determined by the CCC. A mismatch was observed in a considerable number of EPAs.

Barnsley et al.¹¹ found a similar mismatch for junior medical officers in their first post-graduate year (PGY-1). When comparing the self-assessed confidence levels of PGY-1 s with the scores obtained through the standard OSCE evaluation, their self-reported confidence levels were higher than their actual assessed competence.¹¹ Conversely, Allameddine et al.¹² have shown a mismatch in the opposite direction. They found that surgery residents underrated their laparoscopic skills in performing cholecystectomies.

Hodges et al. observed that the mismatch varied according to physicians' expertise.¹³ Comparing experts' rates with first-year family physician residents' self-assessed scores on communication skills, they found that those rated by experts in the top tertile often underestimated their performance. In contrast, those in the bottom tertile overestimated it. Violato and Lockyer¹⁴ also showed that psychiatrists, pediatricians, and internal medicine specialists ranked in the lowest and highest quartiles by their peers tended to under- or overestimate their performance, respectively. This is often referred to as the Dunning-Kruger effect, as these authors published a seminal paper showing that college students at the lowest quartile of performance consistently self-rated their performance as better than the actual measured performance.¹⁵

Residents who are unconscious of their incompetence cannot recognize when they need help, which might eventually compromise patient safety. Hays et al. introduced the concept of the “unconsciously incompetent“to describe physicians with low insight and performance and argued that the lack of insight makes this group more difficult to remediate.¹⁶

This mismatch may challenge the teaching-learning process, as assessments that do not align with the learner's perception can result in poor appreciation for the feedback received.¹⁷ Entrusting residents to perform activities whose competencies are slightly underestimated by the residents can accelerate the learning process in a process called “constructive friction”.¹⁰ However, caution is advised since transferring responsibility too early can increase the resident's insecurity and compromise patient safety, resulting in “destructive friction”.¹⁰

In contrast, when the CCC underestimates residents' performance, this might result in assigning more straightforward tasks that do not challenge the resident, which could hinder learning.² Kempenich et al.¹⁸ identified the underestimation of residents' performance as strongly associated with the delayed progression of residents' autonomy. These circumstances may result in the excessive delegation of healthcare responsibilities to supervisors and reduce trainees' chances to cultivate autonomy, failing to benefit trainees and patients.¹⁹ This mismatch might also lead to the “imposter syndrome” where the resident does not experience a sense of success despite having the appropriate competencies to perform a given task. If the CCC decision makes residents inappropriately insecure about their performance, future external critiques may create significant emotional distress.²⁰

The mismatch between preceptors' and residents' assessments has many possible explanations, such as residents' unjustified doubt about their performance or preceptors' reluctance to entrust their residents.^7,21 Several studies have shown that the skills for competence assessment can be developed through training, reflection, and feedback.^7,9,21,22

In our study, the mismatch between residents' and CCC's assessments was more pronounced for clinical surgical EPAs. Residents seemed overconfident in performing clinical surgical activities that do not require procedural skills. The fact that residents have more previous experience with such EPAs during undergraduate training and the lower perceived risk of immediate complications from possible failures might help explain such findings.

Conversely, we identified a better alignment between self-assessed and CCC-determined entrustment levels for complex procedural EPAs, such as performing bariatric or oncological surgeries. Some studies have shown that learners feel more prepared for less demanding surgical activities than what their preceptors perceive and are less confident about performing more demanding tasks.^23–25

Determining when surgical residents can perform activities with greater autonomy is complex. Entrustment decisions are directly related to factors such as the perception of learner characteristics (their abilities, experiences, integrity, reliability, and humility), the preceptor's inclination to delegate tasks (depending on the specialization practice of the task), the complexity of the EPA, the context (patient complexity, environment, structure, schedule, personal need, and supervision rules), and the nature of the teacher-student relationship.^2,26

Our results revealed that the entrustment level defined by the CCC progressively increased over the years of residency training, reaching its highest levels in the third year, as expected for residency training and corroborated by previous studies.^9,25 In parallel with such progression, students' self-assessed confidence also increased, peaking in the third year. An increase in residents' confidence throughout surgical training was also expected and has been described in previous studies.^21,25 This indicates that, from both the students' and preceptors' perspectives, there was a significant gain in the acquisition of competencies over the years.

Our study observed a gradual and progressive transfer of responsibility, as perceived by the increase in the residents' confidence and the CCC-defined entrustment levels throughout the training period. This progression demonstrates that, from both perspectives, residents have acquired the required competencies along with training. This trend was expected and observed in several other studies.^9,25,27,28

The asymmetry between self-assessment and evaluation by the CCC was identified across all three years of the postgraduate program, with no interaction observed with the resident's year of training. Despite the overall tendency for residents to overestimate their level of autonomy, we observed individual cases where residents underestimated their performance in one or more EPAs.

When supervising underperforming overconfident residents, providing direct and specific multi-source feedback is essential but often insufficient to drive performance improvement. Clearly defining competency expectations and outlining potential risks in case of failure may be necessary. For residents performing adequately but still insecure, superficial reassurance might not be enough to foster growth. Program directors can help residents overcome low self-confidence by reinforcing the learning journey and emphasizing perseverance.²⁹

The study had some limitations. The assessment period was short (a three-month rotation) and was conducted at a single institution, which may limit the extrapolation of the findings.

In conclusion, entrustment levels self-assessed by the residents on surgical EPAs were consistently higher than those determined by the CCC. The difference was more remarkable for clinical surgical and less demanding EPAs. There was no association between the years of training and the difference between self-assessed entrustment levels and those determined by the CCC. Future studies are needed to assess the impact of such mismatch on residents´ development of competencies and the effect of faculty development strategies to align residents´ and preceptors´ perceptions better.

Ethical responsibilities

The Research Ethics Committee of the Santa Casa Hospital in Belo Horizonte approved the study under protocol number 5.887.987. All study participants signed informed consent. The study's inherent risks were small and justified by its potential benefits.

Acknowledgements

The authors are grateful for the invaluable support provided by the residents and preceptors from the General Surgery Residency Program at Santa Casa BH throughout the research process.