Regístrese
Buscar en
Atención Primaria
Toda la web
Inicio Atención Primaria Evaluation of Clinical Competence in Trainers of Family and Community Medicine R...
Información de la revista
Vol. 34. Núm. 2.
Páginas 68-72 (Junio 2004)
Compartir
Compartir
Descargar PDF
Más opciones de artículo
Vol. 34. Núm. 2.
Páginas 68-72 (Junio 2004)
Acceso a texto completo
Evaluation of Clinical Competence in Trainers of Family and Community Medicine Residents
Evaluación de la competencia clínica de tutores de residentes de medicina familiar y comunitaria
Visitas
2671
M. Pedregal Gonzáleza, E. Molina Fernándeza, JA. Prados Castillejob, F. Quesada Jiménezc, P. Bonal Pitzd, C. Iglesias Arrabale
a Unidad Docente de Medicina Familiar y Comunitaria de Huelva, Spain.
b Centro de Salud Pol??gono de Guadalquivir, C??rdoba, Spain.
c Centro de Salud de Cartuja-Almanj??yar, Granada, Spain.
d Unidad Docente de Medicina Familiar y Comunitaria, Centro de Salud Universitario de Bellavista, Seville, Spain.
e Casa del Mar (Instituto Social de la Marina), El Puerto de Santa Mar??a, C??diz, Spain.
Contenido relaccionado
Este artículo ha recibido
2671
Visitas
Información del artículo
Resumen
Texto completo
Bibliografía
Descargar PDF
Estadísticas
Tablas (4)
Mostrar másMostrar menos
Figuras (3)
Mostrar másMostrar menos
Objectives. We show the first experience of the application of an objetive and structured clinical evaluation (OSCE) procedure to family medicine trainers, that has been carried out in Andalucia. The objective is to use a competence evaluation instrument that, in the short term, will be used not only for trainer accreditation but also for other public sanitary professionals. Participants. Tutors of family and commnity medicine residents. Design. Observational descriptive. Setting. Educational unity of family medicine. Principal measurements. The competencial components to be assessed are the following: anamnesis, physical exploration, communication, technical skill, management, family attention y preventive activities. The clinical situations were selected using the following priority criteria: prevalence, clinical gravity, prevention and early diagnosis importance, case complexity, doctor's capacity of evaluation and simplicity. Results. Thirteen family medicine trainers took part in the OSCE. Their average age was 42.8±3.6 years. The test had an overall reliability coefficient (Cronbach's alpha) of 0.73. The overall mean score of the participants was 73±6.2. The best results about the competencial components were family attention, communication and technical skill. Conclusions. The OSCE can be a convenient tool for family medical trainer evaluation, helping to orientate their education in the weak points and, in the near future, it can also be used as an instrument do accredit family medicine trainers.
Keywords:
Competencia clínica
Tutores de residentes de medicina de familia
Evaluación clínica
Objetivo. Describir la primera experiencia de una evaluación clínica objetiva y estructurada (ECOE) a tutores de residentes realizada en la comunidad autónoma de Andalucía. Diseño. Observacional descriptivo. Emplazamiento. Unidad Docente de Medicina Familiar y Comunitaria de Huelva. Participantes. Tutores de residentes de medicina familiar y comunitaria. Mediciones principales. Los componentes competenciales que se consensuaron y ponderaron fueron los siguientes: anamnesis, exploración física, comunicación, habilidades técnicas, manejo, atención a la familia y actividades preventivas. La selección de las 10 situaciones clínicas de las que constaba la prueba se realizó utilizando unos criterios de priorización según la prevalencia, gravedad clínica, importancia de la prevención y del diagnóstico precoz, complejidad del caso, evaluación de la capacidad resolutiva del médico y simplicidad evaluativa. Resultados. Realizaron la ECOE 13 tutores. La edad media ± desviación estándar de los participantes fue de 42,8 ± 3,64 años. La prueba tuvo un coeficiente de fiabilidad (alfa de Cronbach) de 0,73. Por lo que se refiere a los resultados por participantes, la media global fue de 73 ± 6,2. Al analizar los resultados para los diferentes componentes competenciales, los mejores resultados se obtuvieron en la atención a la familia, la comunicación y las habilidades técnicas. Conclusiones. Las pruebas de evaluación clínica objetiva y estructurada pueden ser útiles para la valoración de los tutores, con el objetivo de orientar su formación en los puntos débiles e incluso para, en un futuro no muy lejano, servir como instrumento para acreditar y reacreditar a los tutores de residentes de medicina familiar y comunitaria.
Palabras clave:
Clinical competence
Family medicine trainer
Clinical evaluation
Texto completo

Introduction

Developed over 20 years ago, objective structured clinical examination (OSCE) is an assessment method that is used by a growing number of health institutions, medical schools and scientific societies. Its use has spread throughout nearly all English-speaking countries.1

In Spain, specifically in Catalonia, the Institute of Health Studies of the Catalonian Regional Government (Generalitat) has carried out assessment exercises since 1994 to evaluate graduate and postgraduate trainees in different medical specialties.2,3 This has made it possible to detect gaps in health care skills among both students and practitioners, and to identify changes needed in medical training activities.4

The number of clinical cases that should be included in an OSCE reflects a compromise between validity, reliability and feasibility. The use of 10 or more cases ensures good reliability, but in general most OSCE include approximately 20 cases distributed in 30 stations.5

The specialty designated family and community medicine comprises a wide range of knowledge fields and skills, and family physicians have pioneered efforts to implement innovative training methods such as a specific program for this specialty, continuing education, certification for training activities, and postgraduate training and evaluation.6

Among the criteria proposed for certification and recertification for trainers of family and community medicine residents is the trainer's professional competence. The OSCE has been proposed as a reliable way to evaluate this.7

Few assessment exercises have been carried out in Spain with the OSCE instrument to evaluate trainers of family and community medicine residents, and no such assessments have been attempted in the autonomous community of Andalusia, the region comprising most of southern Spain. We report the first such experience, carried out at the Family and Community Medicine Teaching Unit in the city of Huelva, southwestern Spain. The aim of the exercise was to test the usefulness of the OSCE in evaluating clinical skills and detecting areas in need of improvement, to enable trainers to enhance their own training.

Material and Methods

The test committee was constituted in 2001. The committee consisted of 6 family physicians experienced in clinical practice at primary care centers, and in the training of residents as tutors and course leaders. All committee members had knowledge of and experience in the evaluation of competence. The functions of the test committee were to define components of competence, prepare the list of clinical situations that merited evaluation, construct the table of specific components of competence, and design some of the clinical cases to be used in the evaluation.

The components of competence and weightings, chosen by consensus, were based on recommendations of the Catalonian Society of Family and Community Medicine,8 and were weighted as follows: clinical interview, 12%; physical examination, 12%; interpersonal communication, 16%; technical skill, 10%; management (ability of the participant to reach a correct diagnosis from the information obtained from the clinical interview, physical examination, and complementary tests; knowledge of treatments; ability to devise a treatment plan), 40%; attention to family members, 4%, and preventive activities, 6%.

The choice of clinical situations was agreed by consensus on the basis of the following priority criteria: prevalence, clinical severity, importance of prevention and early diagnosis, complexity of the case, evaluation of the physician's ability to reach a decision, and feasibility of evaluation.9 The most appropriate instrument for evaluation was chosen depending on the clinical situation: standardized patient, mannequin, clinical image interpretation, set of open questions requiring brief answers, or multiple-choice questions. The table of specific components contained the competence areas to be evaluated and their corresponding clinical situation and instrument. Two clinical cases with their script, practical instructions and scoring checklist (along with additional advice) were supplied by the Institute of Health Studies. All other clinical cases were developed by the test committee.

Logistics and psychometric analyses were managed by the staff of the Family and Community Medicine Teaching Unit in Huelva.

The test consisted of 10 clinical situations, and candidates to be evaluated were allowed 6 minutes per station. This period reflects the mean duration of each patient-physician contact at primary care centers in the region of Andalusia. The stations represented a variety of situations commonly encountered in primary care: emergencies, scheduled appointments and minor surgery (Table 1). The exercise was divided into 2 groups of stations (rounds), and no breaks were allowed between stations. The total time allowed to complete the test was 104 minutes.

The OSCE was carried out at the La Orden health center in the city of Huelva on May 16, 2003. This center was chosen because it provided an appropriate physical environment for the test.

Participants were chosen from among trainers of residents at the Family and Community Medicine Teaching Unit in Huelva, and participation was voluntary. Other participants in the exercise were 5 standardized patients, 7 observers and 3 persons responsible for logistics.

The number of stations was set at 10 on the basis of available resources, and because this was the minimum number of stations that yields reliable results.

The highest score possible for each case was 100 points, distributed between the various components of competence to be evaluated. The total score was calculated as the mean of the scores for each case. At the end of the exercise all participants were given a questionnaire to solicit their opinions and feelings about the exercise.

The data were analyzed with SPSS software (version 11.0). Reliability of the test in general and of each component was calculated as Cronbach´s alpha. Correlations of the scores with age and with years as a trainer were calculated as Pearson's correlation coefficient. The results per component of competence were found for each participant, and mean scores and standard deviations were also calculated. The scores per case were also calculated, as were the results for each participant.

Results

Thirteen of the 22 certified tutors at the teaching unit (8 men, 5 women) participated in the OSCE. Mean age was 42.8 years (SD, 3.6 years), and mean duration of teaching and tutoring activities for residents at the time of the test was 8.6 years (SD, 4.6 years). Of the 9 trainers who did not take part in the OSCE (4 men, 5 women), 8 had personal reasons (the exercise was held on a Friday afternoon) and one women was on maternity leave. Mean age of the 9 nonparticipants was 41.2 years (SD, 2.4 years) and mean time as a trainer and tutor was 4.4 years (SD, 2.9 years). There were no significant differences in age between participants and nonparticipants in the OSCE (P=.27), although the difference in the number of years as a trainer and tutor was significant (P=.03).

The global reliability coefficient for the test (Cronbach's alpha) was 0.7287. By component of competence, the highest scores were obtained for preventive activities, interpersonal communication and technical skill (Table 2).

The global mean score for all participants was 73 (SD, 6.2), with a range of 64.4 to 81.9 and an interquartile range of 11.7 (Figure 1).

Figure 1. Distribution of trainers' scores by case.

 

The best scores were obtained for the components of competence attention to family members, technical skill and interpersonal communication (Table 3). The worst scores were for preventive activities and physical examination, including clinical interview and management. Table 4 shows the mean score and standard deviation for each case.

Test scores and ages of the trainer showed a normal distribution (Shapiro-Wilk's test). Correlations were sought between scores and years as a trainer or age with Pearson's correlation coefficient. Neither correlation was significant.

Most participants (10 of 13) agreed partially or completely that the test and the time allowed for each station reflected conditions encountered in actual practice. Approval was also expressed with the items on the participants' evaluation form that inquired about how the stations were designed, interpretation of the cases, and whether this type of evaluation was useful in improving training for residents. All participants agreed that this type of exercise measured professional capacity better than classical written examinations. The open item on the evaluation form provided space for participants to express emotional features of the test, and elicited comments such as "I was really nervous," "I felt really stressed and a bit nervous," "My self-esteem has hit bottom," "I really liked it," "Overall, I felt comfortable," and "I really enjoyed it."

Discussion

Reliability is the best documented psychometric parameter in evaluations with simulated patients. This indicator is defined as a measure of the reproducibility of a process across a series of replications.10 A minimum of 10 cases are needed to achieve good reliability. In the present study, Cronbach's alpha was similar to values reported in earlier research.2

The global findings for components of competence were satisfactory. However, it was surprising that the skill that yielded the lowest mean score was preventive activities (notwithstanding the large standard deviation), a result that reflected the wide variation in individual scores. This component of competence thus requires improvement. The second lowest mean score was obtained for physical examination skills. This may reflect a loss of skills with time, probably because the patient load leads to simplification of some steps in the physical examination.

On the other hand, the highest scores were obtained for attention to the patient's family (although the weighting of this element in the overall OCSE score was low), interpersonal communication, and technical skill. These results reflect the conscientiousness with which practitioners treat these elements, and possibly the efforts the teaching unit devotes to training the trainers in these areas.

Of the different situations encountered in the exercise, the lowest scores and greatest dispersion were obtained for cases 4 (basic CPR) and 5 (chronic obstructive pulmonary disease). The results for case 4 may reflect the fact that only 4 of the trainers who performed the OSCE worked in rural health centers where a physician is on duty around the clock. This requires them to keep their training in basic life support techniques up to date. In case 5 the greatest weighting was given to the preventive activities component, according to which physicians should give the patient brief counseling about the benefits of quitting smoking. Overlooking this element lowered the score for this case considerably. The stress of the evaluation exercise, and the short time allowed for each case (although the period was a reflection of the time available in real-life conditions) might explain this result. In any case we believe the global outcome and scores for the remaining cases can be considered good.

In general, participants were willing to accept the exercise although some trainers were nervous, and one participant noted that the test had affected self-esteem.

Our findings show that the teaching unit for family and community medicine should make efforts to improve the components of competence that yielded the worst results. Further training will be offered in preventive activities such as counseling and other elements of the motivational interview. Training in physical examination skills will also be improved with appropriate tools, including workshops, CDs, standardized patients, and videotapes of encounters with real patients. These measures will probably lead to improvements in residents´ training.

Objective structured clinical examinations can be useful for evaluating trainers with the aim of enhancing their training in areas where skills are weak. In the not-too-distant future OSCE exercises may serve as a certification and recertification tool for trainers of residents in family and community medicine.

Acknowledgments

We thank the test committee, the trainers who volunteered to participate in this study, Milagros Villamarín Casal for her dedication, Juan Pedro Moreno García and the Huelva-Costa Health District for their assistance, and José Luis Sánchez Ramos for his judicious advice, and the

Bibliography
[1]
Evaluaci??n cl??nica objetiva y estructurada: ECOE. Educaci??n M??dica 1998;1:13-6.
[2]
Primeras experiencias en evaluaci??n de la competencia cl??nica de los m??dicos de familia de Catalunya. Aten Primaria 2001;28:105-9.
[3]
Evaluaci??n cl??nica objetiva y estructurada en geriatr??a. Resultados del primer proyecto realizado en Espa??a. Rev Esp Geriatr Gerontol 2002;37:298-303.
[4]
Evaluando la competencia de los profesionales sanitarios: dudas fundamentos y experiencias. Educaci??n M??dica 1998;1:17-23.
[5]
Screening test length for sequential testing for a SP examination: a receiver operating characteristic (ROC) analysis. Acad Med 1992;67:592-5.
[6]
??Debemos evaluar la competencia de los m??dicos de familia? Aten Primaria 2002;30:539-40.
[7]
Acreditaci??n y reacreditaci??n de tutores de MFyC en Espa??a. Aten Primaria 2002;29:164-6.
[8]
Certificaci??-recertificaci?? de la competencia profesional dels metges d'atenci?? primaria. Butllet?? Societat Catalana de Medicina Familiar i Comunit??ria 1996;63:9-16.
[9]
Evaluaci??n de la competencia: reto y necesidad. Barcelona: semFYC, 2002.
[10]
Technical issues: test application. Acad Med 1993;68:454-60.
Opciones de artículo
Herramientas
es en pt

¿Es usted profesional sanitario apto para prescribir o dispensar medicamentos?

Are you a health professional able to prescribe or dispense drugs?

Você é um profissional de saúde habilitado a prescrever ou dispensar medicamentos

es en pt
Política de cookies Cookies policy Política de cookies
Utilizamos cookies propias y de terceros para mejorar nuestros servicios y mostrarle publicidad relacionada con sus preferencias mediante el análisis de sus hábitos de navegación. Si continua navegando, consideramos que acepta su uso. Puede cambiar la configuración u obtener más información aquí. To improve our services and products, we use "cookies" (own or third parties authorized) to show advertising related to client preferences through the analyses of navigation customer behavior. Continuing navigation will be considered as acceptance of this use. You can change the settings or obtain more information by clicking here. Utilizamos cookies próprios e de terceiros para melhorar nossos serviços e mostrar publicidade relacionada às suas preferências, analisando seus hábitos de navegação. Se continuar a navegar, consideramos que aceita o seu uso. Você pode alterar a configuração ou obter mais informações aqui.