Buscar en
Educación Médica
Toda la web
Inicio Educación Médica Measuring the educational environment in ambulatory settings
Información de la revista
Vol. 16. Núm. 2.
Páginas 131-140 (Abril - Junio 2015)
Compartir
Compartir
Descargar PDF
Más opciones de artículo
Visitas
1468
Vol. 16. Núm. 2.
Páginas 131-140 (Abril - Junio 2015)
ORIGINALES
DOI: 10.1016/j.edumed.2015.09.007
Open Access
Measuring the educational environment in ambulatory settings
Medición del ambiente educacional en contextos ambulatorios
Visitas
...
Arnoldo Riquelmea,b,
Autor para correspondencia
a.riquelme.perez@gmail.com

Corresponding author.
, Oslando Padillac, Cristian Herrerac, Trinidad Olivosd, José Antonio Romand, Alberto Sarfatise, Nancy Solísa, Margarita Pizarroa, Luis Antonio Díaze, Patricio Torresf, Sue Roffg
a Department of Gastroenterology, Pontificia Universidad Católica de Chile, Santiago, Chile
b Centre for Medical Education, Pontificia Universidad Católica de Chile, Santiago, Chile
c Department of Public Health, Pontificia Universidad Católica de Chile, Santiago, Chile
d Faculty of Psychology, Universidad Alberto Hurtado, Santiago, Chile
e Department of Internal Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile
f Department of Psychiatry, Pontificia Universidad Católica de Chile, Santiago, Chile
g Centre for Medical Education, University of Dundee, Dundee, United Kingdom
Información del artículo
Resumen
Texto completo
Bibliografía
Descargar PDF
Estadísticas
Figuras (1)
Tablas (3)
Table 1. Ambulatory Care Learning Educational Environment Measure (ACLEEM).
Table 2. Mean scores of items separated by domains of the ACLEEM questionnaire. Comparison between residents and clerks.
Table 3. D Study Results Using G Coefficients of the ACLEEM questionnaire.
Mostrar másMostrar menos
Abstract
Background and objectives

Students’ perceptions of their educational environment (EE) have been studied in undergraduate and postgraduate curricula. The Ambulatory Care Learning Educational Environment Measure (ACLEEM) is an inventory that was recently developed to measure the EE in postgraduate ambulatory settings. The aim of this study was to assess the psychometric properties of the inventory.

Methods

A mixed methodology was used to develop the ACLEEM including: Grounded theory (8 focus groups); a two-round Delphi technique to identify consensus; and a pilot study. The inventory was refined to 50-items after the pilot study and it was prospectively administered to a large cohort of clerks and residents in Chile during 2010-2011. Psychometric measurements included factor analysis followed by Varimax rotation for construct validity, Cronbach's alpha coefficients for internal consistency and Generalizability theory for test reliability.

Results

Four-hundred and eleven students responded: 151 clerks (83.9% of the target population) and 260 residents (74% of the target population) from 31 postgraduate programs. The factor analyses showed an eight factor instrument. ACLEEM was found highly reliable with a Cronbach's alpha of 0.94 and D-study revealed a reliable outcome for residency programs with at least 15 respondents with a G coefficient of 0.831. The EE perceived by residents and clerks was positive without differences between groups: 152.52 ± 23.36 (76.26%) and 150.61 ± 24.62 (75.30%), respectively (p=0.761).

Conclusions

The 50-item ACLEEM inventory is a multidimensional and valid instrument requiring only 15 respondents for reliable results. We recommend using it to measure the EE in the ambulatory postgraduate Spanish-speaking programs.

Keywords:
Educational environment
Postgraduate medical education
Ambulatory medicine
Chile
Resumen
Introducción y objetivos

Las percepciones de los estudiantes sobre el ambiente educacional (AE) han sido estudiadas en programas de pregrado y posgrado. El cuestionario Ambulatory Care Learning Educational Environment Measure (ACLEEM) es un instrumento desarrollado para medir el ambiente educacional en programas de posgrado ambulatorios. El objetivo de este estudio fue evaluar las propiedades psicométricas de este instrumento.

Métodos

Se utilizó metodología mixta para desarrollar el instrumento ACLEEM, incluyendo: teoría fundada (8 grupos focales) y panel Delfi de 2 rondas (consenso). Se realizó un estudio piloto y luego se refinó el cuestionario a 50 ítems, que fue administrado prospectivamente en una cohorte de internos y residentes chilenos durante los años 2010-2011. Las evaluaciones psicométricas incluyeron análisis factorial seguido de rotación Varimax (validez de constructo), coeficiente alfa de Cronbach (consistencia interna) y teoría de la generalizabilidad (confiabilidad).

Resultados

Se obtuvieron 411 respuestas: 151 internos y 260 residentes (tasa de respuesta: 83,9% y 74%, respectivamente) correspondientes a rotaciones de internado y 31 programas de posgrado en Medicina. El análisis factorial mostró un instrumento de 8 categorías. ACLEEM presentó una alta confiabilidad (alfa de Cronbach 0,94) y un estudio de generalizabilidad que mostró resultados confiables en programas con al menos 15 encuestados (coeficiente G de 0,831). El ambiente educacional percibido por los residentes e internos fue positivo, sin diferencias entre ambos grupos: 152,52 ± 23,36 (76,26%) y 150,61 ± 24,62 (75,30%), respectivamente (p = 0,761).

Conclusiones

El cuestionario ACLEEM de 50 ítems es un instrumento multidimensional y válido, que requiere solo 15 encuestados para contar con resultados confiables. Recomendamos su uso para medir el ambiente educacional en programas de posgrado ambulatorios de habla hispana.

Palabras clave:
Ambiente educacional
Educación médica de posgrado
Medicina ambulatoria
Chile
Texto completo
Introduction

Educational environment (EE) can be defined as “the impressions, beliefs and expectations held by members of the school community about their school as a learning environment, their associated behaviour, and the symbols and institutions that represent the patterned expression of behaviour”1. In other words, the EE is the learning climate within a school seen through the eyes of all of its members. Moreover, it is known that the EE is an important aspect that must be considered in postgraduate medical education. Particularly, its impact is well recognized and accepted because of its real influence over students’ achievement, satisfaction and success2–4.

In the last years, many new instruments have been developed and validated to evaluate the EE in healthcare professions. Specifically related with postgraduate medical education, Soemantri et al. found 9 instruments that have been used in this level, but none of them have been designed particularly for the ambulatory setting5. The World Health Organization in the document “Primary Health Care: Now more than ever” evidenced the importance of the primary health care (PHC), considering it as the most important level in the healthcare systems6. This is also supported by scientific evidence showing that health systems with stronger PHC has better health outcomes, reduce inequities and has less healthcare costs7–9. These are some of the reasons why international agencies such as the Pan American Health Organization have recommended medical schools to orientate their curricula more strongly towards components of PHC10. This setting is clearly different from the in-hospital location where the teaching characteristics are substantially different11.

Given the agreed relevance of ambulatory care and in the absence of a specific instrument to evaluate the EE in this setting, it becomes an imperative to develop a valid and reliable instrument to measure EE in postgraduate ambulatory medical education. Our team started to carry out this task and the results related to the qualitative research process and the development of the 50-items inventory that we called “Ambulatory Care Learning Educational Environment Measure” (ACLEEM) are published elsewhere12.

The aims of the present study were to prospectively administer the ACLEEM inventory to clerks and postgraduate residents at the Pontificia Universidad Católica Medical School (PUCMS) and to analyze the psychometric properties (validity and reliability) of the instrument in order to measure the EE in postgraduate ambulatory medical education.

MethodsInstrument

The ACLEEM inventory was developed in a three-stage process that included: grounded theory, a Delphi technique to identify consensus and a pilot study. Three quota samples of approximately 60 stakeholders were formed, one as Focus Groups and 2 as Delphi panels. Stage 1: Eight focus groups were carried out including 58 residents (Latin-American Spanish speakers) from 16 postgraduate programs. The results were analyzed and 173 items were offered in the first round to a National Delphi panel. Stage 2: Sixty-one residents and teachers identified 64 items that were considered important by the panel (>3 points in a 0-4 Likert scale). In the second round, the Delphi panel reduced the number of important items to 54 items. Stage 3: The 54-item inventory was then piloted with 63 residents. The refined version of the ACLEEM consists of 50 items with each one of them scored on a five-point Likert scale with 4 = Strongly agree, 3 = Agree, 2 = Unsure, 1 = Disagree and 0 = Strongly disagree. Because items 24 and 27 contained negative statements, we reverse coded the scores for these questions. Hence, higher scores indicate a more positive result. The details of these stages are published elsewhere12 (Table 1).

Table 1.

Ambulatory Care Learning Educational Environment Measure (ACLEEM).

1.  Working in the OPC enables me to develop my problem solving skills. 
  (El trabajo en el consultorio/policlínico me ayuda a desarrollar mis destrezas para resolver problemas
2.  The teaching staff in the OPC have good clinical skills. 
  (Los profesores clínicos de consultorios/policlínicos tienen buenas destrezas clínicas
3.  The teaching staff in the OPC are up to date in their knowledge and skills. 
  (Los profesores clínicos del consultorio/policlínico están actualizados en conocimientos y destrezas clínicas
4.  My teachers in the OPC use teaching methods that are appropriate for each subject matter. 
  (Mis profesores clínicos en el consultorio/policlínico utilizan metodologías de enseñanza adecuadas para cada uno de los contenidos impartidos
5.  I feel that my clinical teachers are appropriately qualified to carry out their teaching duties. 
  (Siento que mis profesores clínicos están apropiadamente capacitados para realizar sus actividades docentes) 
6.  My clinical teaching staff are interested in improving the quality of the teaching activities in the OPC. 
  (Mis profesores clínicos demuestran interés en mejorar la calidad de la docencia en el consultorio/policlínico
7.  I can develop my interpersonal skills in the OPC. 
  (Puedo desarrollar mis habilidades interpersonales en el consultorio/policlínico) 
8.  I get my evaluations in a timely manner from the teachers in the OPC. 
  (Conozco el resultado de mis evaluaciones por parte de los profesores del consultorio/policlínico de manera oportuna
9.  I feel that the assessment methods used in the OPC are compatible with the teaching methodology. 
  (Siento que los métodos de evaluación utilizados en el consultorio/policlínico son compatibles con la metodología de enseñanza
10.  I have a clear idea about the objectives and learning outcomes of my educational activities in the OPC. 
  (Tengo claros los objetivos/logros educacionales de aprendizaje de mis actividades docentes en el consultorio/policlínico) 
11.  I feel that the learning objectives and outcomes of the OPC are achieved appropriately. 
  (Siento que los objetivos/logros educacionales de aprendizaje del programa de mis rotaciones ambulatorias se cumplen a cabalidad
12.  I am allowed to participate actively in external educational events and medical meetings. 
  (En el consultorio/policlínico me dan las facilidades para participar en eventos educacionales y congresos
13.  My teachers in the OPC use teaching and learning activities effectively. 
  (Mis profesores clínicos del consultorio/policlínico utilizan las oportunidades de enseñanza y aprendizaje en forma efectiva
14.  The allocated teaching time in the OPC is respected by the clinical teachers. 
  (El horario de tiempo protegido para la docencia es respetado por mis profesores clínicos del consultorio/policlínico
15.  My clinical teachers provide me with feedback about my strengths and weaknesses. 
  (Mis profesores clínicos me proveen de retroalimentación (feedback) respecto a mis fortalezas y debilidades
16.  My clinical teachers are enthusiastic about teaching. 
  (Mis profesores clínicos son entusiastas al enseñarme
17.  Working in the OPCs gives me learning opportunities in a wide variety of diseases. 
  (La rotación por distintos consultorios/policlínicos me permite aprender una gran variedad de enfermedades
18.  In the OPC I learn to treat patients with conditions that are specifically related to ambulatory care. 
  (En el consultorio/policlínico aprendo a tratar enfermedades específicamente ambulatorias
19.  My clinical teachers in the OPC appropriately emphasise the doctor-patient relationship. 
  (Mis profesores clínicos del consultorio/policlínico ponen el énfasis apropiado en la relación médico-paciente
20.  In the OPC I learn from the experience of my clinical teachers. 
  (En el consultorio/policlínico aprendo de la experiencia de mis profesores clínicos
21.  My clinical teachers are good professional role models for me. 
  (Mis profesores clínicos son buenos modelos profesionales para mí
22.  The clinical facilities in the OPC are suitable for working with patients in my specialty. 
  (Los box de atención de los consultorios/policlínicos son adecuados para la atención de los pacientes de mi especialidad
23.  I have the opportunity to follow up my patients appropriately in the OPC. 
  (Tengo la oportunidad de realizar seguimiento a mis pacientes del consultorio/policlínico
24.  I have insufficient time with each patient in the OPC. 
  (El tiempo que tengo para la atención ambulatoria de cada paciente es insuficiente
25.  My activities in the OPC are clearly programmed. 
  (Mis actividades en el consultorio/policlínico están claramente programadas
26.  I am able to refer my patients for evaluation by multidisciplinary teams. 
  (En la atención ambulatoria puedo derivar a mis pacientes para evaluación por equipos multidisciplinarios
27.  There are insufficient clinical supervisors for the number of residents in the OPC. 
  (Los supervisores clínicos son insuficientes para el número de residentes que trabajamos en el consultorio/policlínico
28.  I can obtain clinical supervision when I need it. 
  (Puedo obtener supervisión clínica en el consultorio/policlínico cuando lo necesito
29.  I feel that I have the appropriate level of responsibility for my patients in the OPC. 
  (Siento que tengo el apropiado nivel de responsabilidad con mis pacientes del consultorio/policlínico
30.  I feel that my clinical supervisors consider my opinions in clinical decision making about my patients. 
  (Siento que mis supervisores clínicos consideran mis opiniones en la toma de decisiones respecto a mis pacientes
31.  I feel that I treat my patients in the OPC according to the treatment protocols for their conditions and illnesses. 
  (Siento que trato a mis pacientes del consultorio/policlínico de acuerdo a los protocolos clínicos existentes para el manejo de sus enfermedades) 
32.  I am able to learn the required practical procedures in the OPC. 
  (Tengo oportunidades para aprender los procedimientos prácticos requeridos en el consultorio/policlínico
33.  I feel that I am learning to become confident in my speciality in the OPC. 
  (Siento que la enseñanza que recibo en el consultorio/policlínico me ayuda a desarrollar las competencias relacionadas con mi especialidad
34.  I feel that the clinical rotations in the OPC are preparing me properly for my professional future. 
  (Siento que me preparan adecuadamente en las rotaciones ambulatorias para mi futuro ejercicio como profesional
35.  In the OPC I manage clinical problems taking into account the social and emotional aspects of my patients. 
  (En el consultorio/policlínico manejo los problemas clínicos considerando los aspectos sociales y emocionales de mis pacientes
36.  I am able to learn to adjust my work to the resources available in the OPC. 
  (Aprendo a trabajar adaptándome a los recursos disponibles en el consultorio/policlínico
37.  I am able to carry out health education activities in the OPC. 
  (Tengo la posibilidad de realizar actividades de educación en salud en el consultorio/policlínico
38.  I feel that my time in the OPC is preparing me to address the health needs of the country. 
  (Siento que en el consultorio/policlínico me preparan para atender las necesidades de salud del país
39.  The teachers in the OPC respond to my personal concerns appropriately. 
  (Cuento con el apoyo del profesor encargado de la rotación ambulatoria para resolver mis problemas personales
40.  The workload allows me to balance the clinical care of my patients with my educational activities. 
  (La carga asistencial me permite compatibilizar la atención de mis pacientes con las actividades docentes
41.  I can keep my work and personal life in balance when I am working in the OPC. 
  (Puedo compatibilizar el trabajo con mi vida personal cuando trabajo en el consultorio/policlínico
42.  My working hours in the OPC permit adequate rest and eating times. 
  (Mi jornada laboral ambulatoria considera tiempos de descanso y alimentación adecuados
43.  I feel part of the team in the OPC. 
  (En el consultorio/policlínico me siento parte del equipo de trabajo
44.  I receive support from other OPC residents when I need it. 
  (En el consultorio/policlínico recibo apoyo de otros residentes cuando lo necesito
45.  I feel that other members of the healthcare team are willing to help me when I need it. 
  (Siento que otros miembros del personal de salud tienen buena disposición a ayudar cuando los necesito
46.  I have adequate access to computers and Internet in the OPC. 
  (En el consultorio/policlínico cuento con acceso adecuado a computadores con conexión a internet cuando lo requiero
47.  The OPC provides lockers to keep my personal belongings safe. 
  (Los consultorios/policlínicos disponen de un lugar seguro para guardar mis pertenencias
48.  There are adequate bathroom facilities in the OPC. 
  (En los consultorios/policlínicos cuento con un baño adecuado para el uso de los residentes
49.  The OPCs have adequate supplies and instruments to render quality professional care. 
  (En el consultorio/policlínico cuento con los insumos e implementos necesarios para prestar una atención de calidad
50.  The clinical files and/or information systems of the OPC give me adequate access to patient information. 
  (Las fichas clínicas y/o sistema informático del consultorio/policlínico me permiten un acceso adecuado a la información del paciente
Subjects and procedure

The inventory was administered to clerkships (students in their last 2 years of medical school) and residents of 31 different programs of the PUCMS (family medicine, psychiatry, paediatrics with their sub-specialties, surgery with their sub-specialties, orthopaedics, dermatology, otorhinolaryngology, ophthalmology, neurology and internal medicine with their sub-specialties), during 2010 and 2011. The ambulatory facilities were primary and secondary care teaching clinics owned and/or managed by the PUCMS. The instrument was administered by members of the Center of Medical Education of the PUCMS with the support of the Postgraduate Department and Directors of each program. The project was approved by the Ethics Committee of our institution. The inventory was administered in both printed and online formats. Response was taken as informed consent and the results were anonymized.

Statistical analyses

To evaluate the construct validity of the 50-item questionnaire, we used an exploratory factor analysis followed by a Varimax rotation. Factors were chosen using the following 2 criteria: 1) the Kaiser-Guttman criterion, in which all factors with an Eigen value >1 were included13,14; and 2) the Cattell criterion, where the inflexion point of the scree plot curve is the cut off, and all factors above are accepted15. Data were analyzed by the SPSS statistical program.

We calculate Cronbach's alpha to test internal consistency16. In addition, we used Generalizability theory (G-theory) to test the reliability of the questionnaire as a complement of the Cronbach's alpha17. This theory allows estimation of the size of the relevant influences that affect the measurement. The subsequent estimation of the reliability of the instrument is based on a variety of reliability indices. To perform G-theory, and because the number of students by gender, year and level of study was different, we used the Program urGenova18. For the D-study (Decision Study) we used the Genova program as well. Results were expressed as mean ± standard deviation of the mean. Data between groups (clerks and residents) were analyzed by a 2-tailed t test. A p value of less than 0.05 was considered statistically significant.

Results

The inventory was responded by 151 clerks (83.9% of the target population) and 260 residents (74% of the target population) of which 46.72% were female and 3.9% were non-Chilean nationals. The respondents had a mean of 27.41 years of age. The women's mean age was 27.66 (CI 95%, 27.45-28.75) and for men was 27.22 (CI 95%, 27.01-28.31), and the difference was non-statistically significant (p=0.41). No difference was found between Chilean and non-Chilean respondents.

Construct validity and internal consistency

Construct validity of the subscales was performed using the exploratory factor analyses followed by Varimax rotation of the data and resulted in 11 factors with an Eigen value > 1. The first factor had an Eigen value of 14.479 (accounting for 28.96% of the variance) and the next 10 factors had Eigen values > 1.037. The 11 factors together explained 59.88% of the variance. The inflexion point of the curve was observed between factors 8 and 9 in the 11-factor scree plot (fig. 1). Therefore, we forced a factor analyses with 8 factors. Distribution of the 8 factors accounts for 50.83% of the variance. Eight domains or subscales of resident's perceptions were defined (Table 2).

Figure 1.

Full scree plot of the factors identified in the exploratory factor analysis of the ACLEEM.

(0,14MB).
Table 2.

Mean scores of items separated by domains of the ACLEEM questionnaire. Comparison between residents and clerks.

Domain  Items  ResidentsClerksp value1 
    RR (%)  Mean  SD  RR (%)  Mean  SD   
1. Teachers  99  3.55  0.521  100  3.35  0.675  0.002 
(11 items; maximum score 44)  99  3.52  0.612  100  3.38  0.641  0.030 
  99  3.12  0.800  100  3.13  0.822  0.878 
  99  3.31  0.800  100  3.13  0.936  0.044 
  100  3.12  0.924  100  3.15  0.905  0.778 
  13  99  3.17  0.790  100  2.96  0.832  0.010 
  16  99  3.28  0.783  100  3.27  0.702  0.933 
  19  99  3.27  0.728  100  3.19  0.905  0.322 
  20  99  3.37  0.784  100  3.36  0.668  0.893 
  21  99  3.41  0.678  99  3.12  0.819  0.0001 
  39  99  2.97  1.038  98  2.87  1.048  0.367 
  Total domain  99  36.11  6.092  98  35.01  6.460  0.087 
  (%)    (82.0)      (79.5)     
2. Clinical activities and patient care  23  99  3.34  0.868  99  2.64  1.177  0.0001 
(11 items; maximum score 44)  25  100  3.29  0.784  100  2.89  0.920  0.0001 
  29  100  3.36  0.780  100  3.25  0.902  0.184 
  30  100  3.43  0.651  100  3.05  0.985  0.0001 
  31  100  3.35  0.668  100  3.04  0.871  0.0001 
  32  100  2.84  1.008  99  2.17  1.236  0.0001 
  33  100  3.47  0.572  98  3.07  0.723  0.0001 
  34  100  3.43  0.645  99  3.25  0.741  0.009 
  43  100  3.13  0.954  99  2.79  1.137  0.002 
  44  99  3.54  0.672  99  3.15  0.822  0.0001 
  45  100  3.52  0.654  98  3.19  0.836  0.0001 
  Total domain  98  36.70  5.098  97  32.76  6.519  0.0001 
  (%)    (84.4)      (74.4)     
3. Protected time  14  99  2.78  1.120  100  2.92  1.030  0.210 
(5 items; maximum score 20)  37  100  2.71  1.155  99  2.97  1.068  0.019 
  40  100  2.64  1.115  99  2.87  0.981  0.035 
  41  100  2.70  1.089  99  3.05  1.002  0.001 
  42  100  2.37  1.307  99  3.06  0.998  0.0001 
  Total domain  99  13.20  4.084  99  14.89  3.896  0.0001 
  (%)    (66.0)      (74.4)     
4. Infrastructure  22  100  2.98  1.190  100  3.11  0.913  0.209 
(6 items; maximum score 24)  26  100  3.25  0.849  98  3.03  0.951  0.015 
  47  100  2.21  1.464  99  2.99  1.156  0.0001 
  48  100  3.17  1.076  99  3.21  0.931  0.649 
  49  100  3.08  1.026  99  3.07  0.875  0.971 
  50  100  3.19  0.959  99  3.29  0.879  0.290 
  Total domain  100  17.88  4.530  98  18.68  3.755  0.056 
  (%)    (74.5)      (77.8)     
5. Clinical skills  100  3.63  0.521  100  3.54  0.619  0.102 
(6 items; maximum score 24)  99  3.52  0.637  100  3.32  0.836  0.013 
  17  100  3.03  0.994  100  2.92  1.099  0.298 
  18  100  3.37  0.741  100  3.41  0.646  0.533 
  35  100  3.43  0.668  99  3.25  0.725  0.014 
  36  100  3.54  0.522  98  3.30  0.760  0.001 
  Total domain  99  20.51  2.654  98  19.72  3.149  0.011 
  (%)    (85.4)      (82.1)     
6. Assessment and feedback  99  2.68  1.115  99  2.55  1.282  0.331 
(5 items; maximum score 20)  99  2.85  0.963  100  2.88  0.909  0.768 
  10  100  2.83  1.084  100  2.93  0.974  0.354 
  11  100  2.73  0.988  100  2.66  0.916  0.463 
  15  99  2.61  1.138  100  3.01  0.894  0.0001 
  Total domain  99  13.73  4.110  99  14.04  3.523  0.444 
  (%)    (68.65)      (70.20)     
7. Information. communication and technology  12  100  2.67  1.145  100  2.42  1.157  0.033 
(3 items; maximum score 12)  38  100  2.62  1.079  98  2.81  1.105  0.085 
  46  100  1.96  1.550  99  2.31  1.362  0.016 
  Total domain  100  7.25  2.719  98  7.58  2.754  0.239 
  (%)    (60.41)      (63.16)     
8. Clinical supervision2  24  100  2.00  1.353  100  2.25  1.254  0.065 
(3 items; maximum score 12)  27  99  2.17  1.282  99  2.04  1.175  0.313 
  28  100  3.00  1.045  100  3.18  0.792  0.076 
  Total domain  99  7.17  2.540  99  7.46  2.182  0.242 
  (%)    (59.75)      (62.16)     
Global questionnaire  97  152.52  23.36  93  150.61  24.62  0.761   
(50 items; maximum score 200) (%)    (76.26)      (75.30)       

RR: relative risk; SD: standard deviation.

Reliability analyses

Response rates varied from 99.27% (items 23 and 39) to 100%. The internal consistency of the 50-item inventory was measured with a Cronbach¿s alpha of 0.94. G-theory was carried out and the following facets were considered for the analysis: Type of Undergraduate University (Traditional, Private or Foreign) (T), PUCMS or Other University (U), Gender (G), Year of Residency or Clerkship (Y), Nationality (Chilean or Foreigner) (N), Specialty or Clerkship Program (S) and Residency or Clerkship (R)18.

We adjusted two types of models. The first group of models considered the total of each person (p) in each domain (D). In this class we have the following models: Dx(p:(UxGxY)), Dx(p:(UxYxS)), Dx(p:(TxGxY)), Dx(p:(UxG)), Dx(p:(TxG)), Dx(p:(GxY)), Dx(p:(GxN)), Dx(p:(GxS)), Dx(p:(UxR)), Dx(p:(NxR)), Dx(p:(NxS)), Dx(p:(YxR)), Dx(p:(SxT)), Dx(p:(GxR)), Dx(p:(UxS)), Dx(p:(TxR)), Dx(p:G), Dx(p:S), Dx(p:U), Dx(p:R), Dx(p:Y) and Dxp. Our main objective at this stage was to analyze how much importance had the facets in the overall variability.

The second group of models included the results at the item level (i). The corresponding models were: (i:D)x(p:(UxGxY)), (i:D)x(p:(UxYxS)), (i:D)x(p:(TxGxY)), (i:D)x(p:(UxG)), (i:D)x(p:(TxG)), (i:D)x(p:(GxY)), (i:D)x(p:(GxN)), (i:D)x(p:(GxS)), (i:D)x(p:(UxR)), (i:D)x(p:(NxR)), (i:D)x(p:(NxS)), (i:D)x(p:(YxR)), (i:D)x(p:(SxT)), (i:D)x(p:(GxR)), (i:D)x(p:(UxS)), (i:D)x(p:(TxR)), (i:D)x(p:G), (i:D)x(p:S), (i:D)x(p:U), (i:D)x(p:R), (i:D)x(p:Y) y (i:D)xp.

In order to adjust the models at the domain level, the overall scores of each domain were expressed as percentages, because each domain included a different number of items, ranging from 3 items (maximum score of 12 points) and others with 11 items (maximum score of 44 points). For the majority of these models the facets were not important due to low variability associated to the facets. A simple model: Dx(p:S), reflects well the situation, in which 26.8% of the variance is due to the variability among persons whom respond the questionnaire, 18.96% of the variance can be explained by the domains, 9.6% to the interaction between specialties and domains, 5.9% to the variability among specialties and 38.7% the variance corresponds to the interaction among persons (inside the specialty programs) with domains confounded with the error.

If we consider the models with item, we conclude that the majority of the facets are irrelevant and we consider that the final model (i:D) x (p:S) reflects well the structure of the data. In this model we can appreciate that half of the variance corresponds to the interaction between person and item (both nested because we need to remember that the persons are inside the specialties and items inside the domains) confounded with the error. Again, the variability among persons is in the first place, with 16% followed by the interaction person-domain with 10.4% and in third place the variance explained by the domains with 8.3%.

We can conclude from these two models that the major components were explained by persons (first place) and the domains (second place), and an important percentage of the variance is confounded with error.

Based on previous analyses, we were focused on following the model selected at the Domain level. The D-study was performed by using the Genova program to investigate the reliability of the instrument for absolute and relative decisions, and the number of students whom had to respond the questionnaire to produce a reliable measurement of the EE. Due to the fact that the Genova program requires balanced designs, we randomly selected a sample of 14 students from each one of the 10 most numerous specialty programs. The most important results of the D-study are shown in Table 3.

Table 3.

D Study Results Using G Coefficients of the ACLEEM questionnaire.

Number of students  G coefficients 
0.567 
0.621 
0.696 
0.724 
10  0.766 
15  0.831 
20  0.868 
30  0.908 
40  0.929 

We have shown here that with reasonable sample size (15 respondents) it achieves high generalizability coefficients, above 0.83. However, with 10 respondents the G-coefficient is 0.766. On the other hand, we can see that to have an excellent reliability (G-coefficient>0.9), the instrument requires 30 respondents.

Finally, the mean values of the domains 2 (Clinical Activities & Patient Care) and 5 (Clinical Skills) were significantly higher among residents compared to clerks. On the other hand, the third domain (Protected Time for Non-Clinical Activities) was significantly higher among clerks compared to residents. No differences were observed in the EE perceived by residents and clerks based on the 50-item global score (Table 2).

Discussion

Quality assurance of postgraduate educational programs and residency training is increasingly important19. EE is one of the aspects to evaluate the quality of training programs providing information about several domains like atmosphere, feedback and supervision in hospital and ambulatory settings20. Several inventories have been developed to measure the EE and ACLEEM is the first one particularly developed to measure aspects related to the EE in ambulatory settings.

The ACLEEM was evaluated in this study to determine its psychometric properties to measure the learning environment in sixteen specialization programs, with a total of 411 respondents. We reached a greater proportion of clerks (151 respondents corresponding to 83.9% of the total universe) and 260 residents (74% of the total universe) from 31 postgraduate programs, which gave us a large representation of the results. According to Kane, the first stage of the research project, development of the ACLEEM questionnaire, could be considered a criterion and content based approach21. Content validity issues were analyzed in detail in the first publication which described the development of the ACLEEM12. However, it is important to take into account the limitation that even when residents from Argentina, Colombia and Ecuador participated in focus groups, giving relevant feedback about the content and meaning of the statements, some words in Spanish could be interpreted in a different way by residents from other Spanish-speaking countries12.

The construct validity was carried out by using an exploratory factor analysis followed by a Varimax rotation. Factors were chosen using the Kaiser-Guttman criterion and the Cattell criterion (inflexion point of the scree plot curve). According to the results observed, the statistical analyses suggested a multidimensional instrument with eight factors based on the criteria used in this study. Revising the preliminary 3 domains emerged from grounded theory with the 8 factors arising from factor analysis we found the following agreements: The domain “Clinical Teaching” originated 2 new factors “Teachers” and “Assessment and feedback”; “Clinical Training” was separated into 3 new factors: “Clinical activities and patient care”, “Clinical skills” and “Clinical supervision”. Finally the items of the Support¿ domain were mainly spread into “Clinical activities and patient care”, “Protected time” and “Infrastructure”.

The 51% of the variance explained by the eight factors is higher or similar to other EE evaluation instruments, such as the PHEEM (1 factor, around 30% of the variance) and the DREEM (5 factors, 52% of the variance)17,22. The high internal consistency is a remarkable aspect of the ACLEEM with Cronbach¿s alpha of 0.94.

A sound factor analysis with 5 subjects per item (250 residents) was achieved with a number of 260 residents demonstrating the construct validity14 internal consistency and reliability including Generalizability theory23. A D-study was successfully carried out and only 15 respondents are required for reliable results (G coefficient of 0.831). Based on our results, we can asseverate that the ACLEEM is suitable to measure the EE of different specialization programs (absolute decision), but not to compare them (relative decisions).

The ACLEEM can be administered at the end of any clinical rotation in an ambulatory setting by someone who was not involved in the rotation teaching activities. The results can be useful for teachers in clinical rotations to improve their teaching practices; for residence program managers to supervise the teaching activities and design improvements in the curriculum; and for students allowing them to identify topics that are shared as positive and negative areas within a rotation. If we analyse the mean scores of the domains, the EE was perceived higher among residents in the postgraduate level in the domains related to patient care and clinical skills including practical procedures compared to clerks. On the other hand, clerks perceived that they have more protected time for non-clinical activities reflecting a higher workload for residents. The results of the ACLEEM can be considered at three levels: (i) individual items, (ii) domains and (iii) overall ACLEEM. Theraw scores obtained for the items making up each of the eight domains are summed for each participant, and then the mean of this summed score is taken to give domain summary scores. To obtain the overall ACLEEM score, the domain summary scores are summed. Examination of the individual items by looking at the mean score obtained across all participants for each item enables the identification of specific strengths and weakness within the EE. This criterion was extrapolated from DREEM and PHEEM24,25 where the developers reported the achieved total and subscale scores as a percentage of the maximum score possible, but made no recommendations regarding interpretation of the DREEM. Subsequently, two of the developers provided guidance as to how to interpret scores at each of the three levels26. Aligned with this criterion we propose for the ACLEEM questionnaire to consider individual, domains and overall mean scores of ≥ 3.0 (75% of the maximum score) are regarded as especially strong areas, mean scores of ≤ 2.0 (50%) need particular attention and items, domains and overall mean scores between 2 and 3 (50-75%) are areas of the EE that could be improved. Under this criterion, the perceived EE observed among residents and clerks could be considered really positive, because the global mean score was >75% in both groups.

Regarding further research, we can mention the following challenges. First, as the ACLEEM was built in a Spanish speaking setting, it should be translated into English or other languages to be validated as well. Secondly, as this is a single-school study, external validity of the instrument can be analyzed by applying it in other Chilean schools or other countries. Thirdly, despite the ACLEEM was developed considering multiple medical specialties, its performance in different postgraduate programs could be compared to explore potential differences.

Finally, we conclude that the ACLEEM is a multidimensional, valid and highly reliable instrument to measure the EE in postgraduate ambulatory settings. It is remarkable how stable the findings are, given their applicability in different specialization programs. We recommend using it to measure the educational environment within each ambulatory postgraduate program with Spanish-speaking residents.

References
[1]
G. Homana, C. Henry, J. Torney-Purta.
School Citizenship Education Climate Assessment.
National Centre for Learning and Citizenship, Education Committee of the States, (2005),
[2]
S. Cavanaugh, P. Simmons.
Evaluation of a school climate instrument for assessing affective objectives in health professional education.
Eval Health Prof., 20 (1997), pp. 455-478
[3]
J.M. Genn.
AMEE Medical Education Guide No. 23 (Part 1): Curriculum, environment, climate, quality and change in medical education-a unifying perspective.
Med Teach., 23 (2001), pp. 337-344
[4]
P. Pimparyon, S. Roff, S. McAleer, B. Poonchai, S. Pemba.
Educational environment, student approaches to learning and academic achievement in a Thai nursing school.
Med Teach., 22 (2000), pp. 359-364
[5]
D. Soemantri, C. Herrera, A. Riquelme.
Measuring the educational environment in health professions studies: a systematic review.
Med Teach., 32 (2010), pp. 947-952
[6]
The World Health Report 2008–primary Health Care (Now More Than Ever). World Health Organization, 2008.
[7]
J. Macinko, B. Starfield, L. Shi.
The contribution of primary care systems to health outcomes within Organization for Economic Cooperation and Development (OECD) countries, 1970-1998.
Health Serv Res., 38 (2003), pp. 831-865
[8]
B. Starfield, L. Shi.
Policy relevant determinants of health: an international perspective.
Health Policy., 60 (2002), pp. 201-218
[9]
B. Starfield, L. Shi, J. Macinko.
Contribution of primary care to health systems and health.
Milbank Q., 83 (2005), pp. 457-502
[10]
Borrell RM, Godue C, García M. Serie N.°2. La Formación en Medicina Orientada hacia la Atención Primaria de Salud. Washington D.C.: OPS; 2008.
[11]
R.W. Zuberi, G. Bordage, G.R. Norman.
Validation of the SETOC instrument - Student evaluation of teaching in outpatient clinics.
Adv Health Sci Educ Theory Pract., 12 (2007), pp. 55-69
[12]
A. Riquelme, O. Padilla, C. Herrera, T. Olivos, J.A. Román, A. Sarfatis, et al.
Development of ACLEEM questionnaire, an instrument measuring residents’ educational environment in postgraduate ambulatory setting.
Med Teach., 35 (2013), pp. e861-e866
[13]
A. Field.
Discovering statistics using SPSS for Windows.
Sage Publications Limited;, (2000),
[14]
A. Field.
Discovering statistics using SPSS for Windows.
Second Edition, Sage Publications Limited, (2005),
[15]
R.B. Cattell.
The scree test for a number of factors.
Multivariate Behav Res., 1 (1966), pp. 245-276
[16]
L. Cronbach.
Coefficient alpha and the internal structure of tests.
Psychometrika, 16 (1951), pp. 297-334
[17]
K. Boor, F. Scheele, C.P. van der Vleuten, A.J. Scherpbier, P.W. Teunissen, K. Sijtsma.
Psychometric properties of an instrument to measure the clinical learning environment.
[18]
R.L. Brennan.
Generalizability Theory.
Springer, (2001),
[19]
L.B. Afrin, G.W. Arana, F.J. Medio, A.F. Ybarra, H.S. Clarke Jr..
Improving oversight of the graduate medical education enterprise: one institution's strategies and tools.
[20]
K. Boor, C. Van Der Vleuten, P. Teunissen, A. Scherpbier, F. Scheele.
Development and analysis of D-RECT, an instrument measuring residents’ learning climate.
Med Teach., 33 (2011), pp. 820-827
[21]
M.T. Kane.
Validating the Interpretations and Uses of Test Scores.
J Educ Meas., 50 (2013), pp. 1-73
[22]
G.R. de Oliveira Filho, J.E. Vieira, L. Schonhorst.
Psychometric properties of the Dundee Ready Educational Environment Measure (DREEM) applied to medical residents.
Med Teach., 27 (2005), pp. 343-347
[23]
J. Crossley, H. Davies, G. Humphris, B. Jolly.
Generalisability: a key to unlock professional assessment.
Med Educ., 36 (2002), pp. 972-978
[24]
S. Roff.
The Dundee Ready Educational Environment Measure (DREEM)--a generic instrument for measuring students’ perceptions of undergraduate health professions curricula.
Med Teach., 27 (2005), pp. 322-325
[25]
S. Roff, S. McAleer, A. Skinner.
Development and validation of an instrument to measure the postgraduate clinical learning and teaching educational environment for hospital-based junior doctors in the UK.
Med Teach., 27 (2005), pp. 326-331
[26]
S. Miles, L. Swift, S.J. Leinster.
The Dundee Ready Education Environment Measure (DREEM): a review of its adoption and use.
Med Teach., 34 (2012), pp. e620-e634
Opciones de artículo
Herramientas
es en pt

¿Es usted profesional sanitario apto para prescribir o dispensar medicamentos?

Are you a health professional able to prescribe or dispense drugs?

Você é um profissional de saúde habilitado a prescrever ou dispensar medicamentos

es en pt
Política de cookies Cookies policy Política de cookies
Utilizamos cookies propias y de terceros para mejorar nuestros servicios y mostrarle publicidad relacionada con sus preferencias mediante el análisis de sus hábitos de navegación. Si continua navegando, consideramos que acepta su uso. Puede cambiar la configuración u obtener más información aquí. To improve our services and products, we use "cookies" (own or third parties authorized) to show advertising related to client preferences through the analyses of navigation customer behavior. Continuing navigation will be considered as acceptance of this use. You can change the settings or obtain more information by clicking here. Utilizamos cookies próprios e de terceiros para melhorar nossos serviços e mostrar publicidade relacionada às suas preferências, analisando seus hábitos de navegação. Se continuar a navegar, consideramos que aceita o seu uso. Você pode alterar a configuração ou obter mais informações aqui.