Introducción: La prueba Evaluación del Desarrollo Infantil (EDI), diseñada y validada en México, se ha aplicado en las unidades de atención primaria del país. Los resultados han sido heterogéneos entre los estados en que se aplicaron las pruebas, a pesar de haber utilizado un modelo de capacitación estandarizado con la misma metodología para la aplicación. El objetivo de este trabajo fue evaluar un modelo de supervisión que permita identificar la calidad de la aplicación de la prueba EDI a nivel poblacional.
Métodos: Se realizó un estudio en la atención primaria tres estados del país para evaluar la aplicación de la prueba EDI por observación directa (estudio de sombra), y verificar la concordancia del resultado a través aplicar la prueba una vez más (estudio de consistencia).
Resultados: Se realizaron 380 estudios de sombra a 51 psicólogos. Al comparar el resultado global, se observó una concordancia del 86.1% con el supervisor (n = 327): el 94.5% por resultado verde, el 73.2% por amarillo y el 80.0% por rojo. Se aplicó la prueba nuevamente en 302 casos, con una concordancia del 88.1% (n = 266): el 96.8% por resultado verde, el 71.7% por amarillo y el 81.8% por rojo. No se encontraron diferencias significativas por grupo.
Conclusiones: Tanto el estudio de sombra como el de consistencia fueron adecuados para evaluar la calidad de la aplicación de la prueba, y pueden ser de utilidad para supervisar la aplicación de la prueba EDI en atención primaria. La decisión de uno u otro sistema depende de la disponibilidad de supervisores.
Background: The Child Development Evaluation (CDE) test designed and validated in Mexico has been used as a screening tool for developmental problems in primary care facilities across Mexico. Heterogeneous results were found among those states where these were applied, despite using the same standardized training model for application. The objective was to evaluate a supervision model for quality of application of the CDE test at primary care facilities.
Methods: A study was carried out in primary care facilities from three Mexican states to evaluate concordance of the results between supervisor and primary care personnel who administered the test using two different methods: direct observation (shadow study) or reapplication of the CDE test (consistency study).
Results: There were 380 shadow studies applied to 51 psychologists. General concordance of the shadow study was 86.1% according to the supervisor: green 94.5%, yellow 73.2% and red 80.0%. There were 302 consistency evaluations with a concordance of 88.1% (n=266): green 96.8%, yellow 71.7% and red 81.8%. There were no differences between CDE test subgroups by age.
Conclusions: Both shadow and consistency study were adequate for evaluation of the quality of the administration of the CDE test and may be useful as a model of supervision in primary care facilities. The decision of which test to use relies on the availability of supervisors.
1. Introduction
The Model of Promotion and Care for Child Development (PRADI) from the Child Development Strategy was designed to align and strengthen actions that would contribute to the monitoring, care and promotion of optimal development of children <5 years of age who are beneficiaries of the PROSPERA program (formerly Oportunidades) and live under conditions of high marginalization and poverty1. One of its components is Timely Detection and Care2 whose objective is to identify and treat in a timely manner boys and girls with developmental problems3. In order to comply with the above, the Child Development Evaluation (CDE)4 test (or Prueba EDI in Spanish) was designed and validated as a screening tool for the timely detection of developmental problems in children from 1–59 months of age5. After analyzing the available evidence6-8, a panel of national experts concluded that the CDE test was the most appropriate screening tool for the detection of developmental problems for the population <5 years of age in Mexico9.
The CDE test is organized into 14 instruments divided by age group, based on the changes in development with increasing age. Each of these instruments has 26–35 items grouped into five axes: a) biological risk factors; b) alert signs; c) developmental areas (fine motor, gross motor, language, social and knowledge); d) alarm signs; and e) neurological examination. Each item could be evaluated by direct observation of the child or by questions directed to the parents or caregivers10. Each item is classified individually with only two possible responses: yes or no. From the responses, the results for each of the five axes are obtained and on this basis the overall result of the test is established: a) normal development (green); b) developmental lag (yellow) and c) risk of delay (red). Table 1 describes the age groups and the criteria of classification for each group.
A manual was designed for the correct application of the CDE test4 and the manner of application of each of the 14 instruments is described. In brief, when applying the test the following steps should be followed:
1. Selection of the location and appropriate conditions for application of the test11.
2. The evaluator should ask the child’s date of birth to calculate the age and select one of the 14 instruments or formats from the corresponding age group so as to select the appropriate material to administer the test.
3. When the child is ≤2 years of age and if born <37 weeks of gestation (WG) it is necessary to correct the age, calculating the number of weeks missing for the child to be term (using as a standard 40 WG)4. Not doing this calculation leads to the evaluation of items of an age older than actual, so the child will not be classified correctly given that the instrument used does not correspond to his/her age.
4. When the CDE test is given it is required that the evaluator carry out two principal activities: to observe the child and to question the parents or caretakers according to what is specified for each item. It is not necessary to follow the order that they appear in the instrument, if and when all are applied and in an appropriate manner.
5. The evaluator should score each item and axis.
6. At the end of the application, a global score (green, yellow or red) should be given for each child by the evaluator, taking into account what is obtained in each axis.
A 2-day training and standardization model was designed for the implementation of the CDE in primary care units nationally in order that the healthcare professional can acquire the necessary knowledge and skills, whether they are psychologists, physicians, nurses or nutritionists, so that the results will be accurate4,10,11. After the different processes of training, application of the CDE was initiated in 26 states of the Mexical Republic. For the year 2013 there were 137,224 boys and girls evaluated. However, despite training12, heterogeneous results were detected with regard to the frequency of problems in development: the variation of green cases ranged from 41–100%; yellow cases from 3–32%; and red cases from 1–37%13. Whereas this situation could be explained by different aspects, the main concern was that related with the correct application of the CDE. It was considered necessary to assure that the healthcare personnel were appropriately carrying out the evaluations. For this reason, the objective of this study was to evaluate the quality of the CDE test in primary care units. A secondary objective was to determine if this form of evaluation could be implemented as a supervision model to ensure reliability of the results of the CDE test.
2. Methods
A study to evaluate the quality of the application of the CDE test by primary care personnel in three states of the Mexican Republic was done: Coahuila, Guanajuato and Tabasco. The quality of the evaluations was determined when the results obtained with the CDE test were compared between the working health personnel and a group of experts in the administration of the CDE test who were called supervisors.
2.1. Study population
The operative personnel evaluated were psychologists hired for the application of the CDE test as part of the Child Development Strategy of the PROSPERA program and would have received training in the CDE test11. The psychologists should have applied the CDE test routinely to children <5 years of age and beneficiaries of the PROSPERA as part of the healthy child control in primary care units from November to December 2013.
2.2. Standardization of the supervisors
The performance evaluation of the operative personnel in the application of the CDE was done by a group of six supervisors (five general physicians and one psychologist) who lived in any of the three states where the study was done. It was considered that each supervisor could evaluate between six and eight psychologists. Once in the health units, the supervisors conducted activities exclusively related to the evaluation of the performance of the psychologists.
To standardize the supervisors, a 3-day educational intervention was carried out at the Hospital Infantil de Mexico Federico Gomez (HIMFG) during which time the following activities were performed:
a. Training for the application of the CDE test12
b. Competency of the supervisors for appropriately carrying out the evaluation in the 14 age groups included for the CDE was verified
c. Before beginning the study, consistency between the supervisors was determined. In the analysis of the results, inter-observer concordance was >90%
d. Instructions necessary were provided for using the instruments in the health units where the information was collected, verifying their correct completion
2.3. Evaluation of the performance of the operative personnel
The performance of the operative personnel was assessed using the concordance of the results of the CDE test applied by them with those obtained by the supervisor (considered as the standard or real value). Concordance was established through the following two processes:
• Shadow study
This was implemented through the direct observation of the operating personnel, in real time, during the administration of the CDE test. In accordance with the activities that the operating personnel performed during the evaluation of the child, the supervisor completed a custom designed check list. At the same time, the supervisor also used the CDE test to evaluate each child, scoring the items and filling out the test independently, making sure not to distract the attention of the child being evaluated. The supervisor did not intervene at the time of the application of the test to correct or to voice an opinion to the operative personnel. When the evaluation of each child was completed, the supervisor compared his/her scores assigned to each item with those assigned by the operative personnel. In the case where differences were found, the items where disagreement was detected were recorded in a file.
• Consistency study
CDE evaluations previously carried out by the operative personnel were taken into account for this phase. Children who had been evaluated 2 weeks prior to supervisor’s visit to the health department were randomly selected. The CDE test was administered once again to these children by the supervisor. At the time of the new application of the CDE, the operative personnel were not present and did not inform the supervisor of the results obtained by the child. The supervisor either went to the homes or scheduled the child in the health department for the new application of the test. After the test was given, the supervisor compared his/her score assigned for each item with those recorded by the operative personnel that were found in the clinical record of each child. In the case where differences were found in one or more items, the disagreement was recorded in one verification card.
2.4. Reliability of the evaluations
To determine that the supervisors were carrying out the recommended activities in an appropriate manner during the visits to the health departments, one group from the general coordination of the study proceeded to verify the supervisor’s compliance. This activity was carried out directly in each of the three states of the Mexican Republic where the study was performed three times: a) during the initiation of the field work in order to present the personnel to the authorities and those responsible in the state; b) midway during collecting the information so as to validate the information received; and c) at the end of the period of collecting the information.
2.5. Study variables
The state in which the tests were carried out was recorded and a unique identification number was assigned for each operator and another for the supervisor. The overall score of the CDE test was recorded for each child (green, yellow or red), both that given by the operative personnel as well as by the supervisor. In addition, the items for which there was disagreement were recorded.
2.6. Statistical analysis
Qualitative variables were presented as frequencies and percentages. To evaluate the differences between these variables, χ2 test was used and the 95% confidence interval (95% CI) was calculated. To evaluate the correlation of the results for categorical variables, Spearman correlation and Kendall’s tau-b was used; two-tailed p < 0.05 was considered statistically significant. SPSS v.20.0 (IBM) package was used for statistical analysis.
2.7. Ethical considerations
The study was approved by the Ethics, Biosafety and Research Committee of the HIMFG as part of the project HIM/2013/063. Each evaluator was notified that they would be evaluated on the quality of the application of the CDE test and that from those results and in cases of areas of discrepancy in the results they would be given recommendations for improving the quality of the application of the CDE test. The information was coded with a unique participant number by state so as to not include personal data. Application of the CDE test is one of the components of the consultation in healthy child control in the health departments14. No patient identification data were recorded for the present study, only the age group and the result of the test items. Each family member or primary caregiver who accompanied the minor was requested to provide verbal informed consent for the supervisor to be present during application of the test or for another application of the CDE test.
3. Results
3.1. Shadow study
There were 51 psychologists as operative personnel included: 25 in Guanajuato (49.0%), 18 in Tabasco (35.3%) and eight in Coahuila (16.0%). In total, there were 380 shadow studies done: 52.4% in Guanajuato (n=199), 28.2% in Coahuila (n=107) and 19.4% in Tabasco (n=74). The supervisors were present along with the psychologists on 1-33 occasions while they administered the CDE test. Taking into account the 14 age groups for the CDE test, 12–51 children per group were included.
When comparing the overall result between the psychologist and the supervisor (Figure 1), a concordance of 86.1% (n=327) was observed. Distribution of the overall result in the CDE test given by supervisors was as follows: green 57.1% (n=217), yellow 32.4% (n=123) and red 10.5% (n=40). However, concordance for these same classifications with the evaluator were 94.5, 73.2 and 80.0%, respectively (Table 2).
Figure 1 Concordance between supervisor and operative personnel in overall classification in the shadow study.
Upon analyzing by age group it was noted that there was discordance in each of the 14 age groups (Figure 2A). From the percentage point of view (Figure 2B), the groups with greater frequency of discordance (or error on the application by the operative personnel) were the following: group 3 (33.3%), group 11 (20.7%), group 13 (19.5%), group 9 (18.8%), group 8 (17.4%), group 7 (17.2%) and group 14 (15.4%). There were statistically significant differences found in the frequency of discordance between groups.
Figure 2 Concordance in the outcome of the application of the CDE test between supervisory and operative personnel in the shadow study according to age group. (A) Distribution of the agreement taking the total CDE test applications used in carrying out the shadow study (absolute frequency). (B) Distribution considering the percentage of concordance in the shadow studies (relative frequency).
When concordance is considered individually for each of the test items in the 380 shadow evaluations, there was a per item discordance from 1.6% (n=6) in two items, 1.3% (n=5) in one item; 1.1% (n=4) in 5 items; 0.8% (n=3) in 5 items; 0.5% (n=2) in 15 items and 0.3% in 33 items.
A wide variability was noted in the analysis of the performance of the application of the test on an individual basis by the operative staff (n=51). Of the total, 54.9% (n=28) obtained 100% concordance with supervisors; only one (2.0%) obtained a concordance of 0%. This result was associated with the fact that this staff member only administered a single test (Figure 3A). In the remaining 43.1%, concordance was variable (Figure 3B).
Figure 3 Agreement on the outcome of the implementation of the CDE test in the shadow study individually considering the operative staff. (A) Concordance in the tests applied by each of the operative personnel evaluated by the supervisor through the shadow study (absolute frequency). (B) Percentage of agreement for the total tests applied by the supervisory operative staff of the shadow study (relative frequency).
3.2. Evaluation of the consistency in the outcome of the test (re-test)
Supervisors applied the test to 302 children previously classified by the operative personnel. Of the total, 148 were in Coahuila (49.0%), 81 in Guanajuato (26.8%) and 73 in Tabasco (24.2%). According to the 14 age groups for the CDE test, there were from 13–42 children evaluated for each group. These 302 children were evaluated by 49 different psychologists (operative personnel); 24 from the state of Guanajuato, 18 from Tabasco and eight from the state of Coahuila.
On comparing the overall result between the supervisor and operative personnel (Figure 4), a concordance of 88.1% (n=266) was observed. From 302 children evaluated, the overall result of the CDE test by the supervisors was 62.3% in green (n=188), 30.5% in yellow (n=92) and 7.2% in red (n=22); concordance for these classifications with the evaluator was 96.8, 71.7 and 81.8%, respectively (Table 3). Concordance of the green and red results was slightly higher than in the shadow study, but decreased 1.5% in the case of yellow results.
Figure 4 Concordance between supervisors and operative personnel in overall scoring in the consistency study.
Upon analyzing by age group, there was discordance in each of the 14 age groups (Figure 5A). From the percentage point of view (Figure 5B) the groups with the highest frequency of discordance were those from group 7 (35.0%), group 13 (19.0%), group 11 (16.7%), group 9 (15.8%), group 8 (17.4%), group 7 (17.2%) and group 12 (14.8%). No statistically significant differences were found in the frequency of the discordant results between groups. Upon considering the concordance individually for each of the items of the test in the 320 evaluations of the consistency study, a discordance per item from 1.3% (n=4) in one item; 1.0% (n=3) in two items; 0.6% (n=2) in seven items and 0.3% in 48 items was found.
Figure 5 Concordance in the results of the application of the CDE test between supervisory and operative personnel in the consistency study by age group. (A) Distribution of the concordance taking the total number of applications of the CDE test in carrying out the consistency study (absolute frequency). (B) Distribution considering the percentage of concordance in the consistency studies (relative frequency).
In the comparative analysis between rating assigned by the supervisors and that assigned individually by the operative staff (n=49), as well as in the shadow study, a wide variability was found. In some a perfect concordance was found in the scoring of all the tests. However, in others no test with the same result as the supervisor was detected. Of the 49 psychologists, 53.1% (n=26) obtained a concordance of 100% and 8.2% (n=4) a concordance of 0% associated with administering less than four tests each one (Figure 6). In the remaining 38.7%, concordance was variable and probably associated with the number of tests given.
Figure 6 (A) Concordance in the tests applied by each of the operative personnel evaluated by the supervisor of the consistency study (absolute frequency). (B) Percentage of concordance for the total tests applied by each of the operative personnel under the supervisor evaluating the consistency studies (relative frequency).
4. Discussion
Using standardized supervision, the present study allowed the evaluation and reliability of the results after application of the CDE test by operative personnel in the states where the program for the detection of developmental problems in children <5 years of age has been placed into practice. Overall concordance of the results in 86.1% in the shadow study and 88.1% in the consistency study allows establishing in general terms that the training provided to the operative personnel appears to be adequate. However, the results showed that the training does not appear to be adequate for the correct application of the CDE test. This is most obvious upon observation that slightly more than 50% of the operative personnel were in concordance with the supervisor in 100% of the tests.
Results of this study indicate the necessity of establishing a permanent monitoring program for those who apply the CDE test in the various health departments of the country. This supervision will allow individually identifying staff members who require additional training in order that their evaluation be valid. Correct classification of the children into green, yellow or red results after application of the CDE test assumes that the operative personnel should have assigned differentiated indications for each child evaluated. The fact that there may not be a good classification places at risk those children who need actions geared towards improving their development.
In this study, two types of supervision were presented: the shadow study and the consistency or re-test (at different times) of the CDE test. On analyzing the results it was found that both methods allow for variations to be identified so that either method appears to be appropriate for supervising the operative staff so as to guarantee that the children evaluated have reliable results. The choice of one or another method will depend on the availability of those who supervise the operative personnel in health units at the time that the CDE tests are conducted. Thus, the shadow study appears to be more convenient because the supervisor can detect variations at the time that the CDE test is given and at the end of the evaluation give indications or feedback so that the operational staff can improve their performance. In those cases in which the supervisor arrives at the health department and no child is present to be evaluated by the operative staff, the supervisor can travel to the homes to administer the CDE test to children who previously had been evaluated.
Differences in the concordance between the age groups of the CDE test may be associated with variations in the number of tests applied in each group by the operative personnel. To reduce these variations would require homogenization of the number of tests given for each age group as well as the number of tests given and supervised. As shown in Figure 3A, assessment of the application by the shadow study on five occasions could identify areas of opportunity and, in the case of the consistency study, from three evaluations (Figure 6B).
Individual analysis of the items showed a low percentage of discordance in the total number of studies done, both for the shadow study (0.3–1.6%) as well as for the consistency
(0.3–1.3%). This corroborates the clarity of the explanation of the items in the training and application materials so it would not require an individual adjustment to them. Supervision will identify operative staff errors and provide appropriate recommendations to improve the quality of the application.
In conclusion, the present study allowed determining that implementation of a model of supervision for operative personnel who apply the CDE is necessary in order to ensure that the children evaluated using this test have reliable results. This supervision will provide differentiated training in particular aspects in the CDE test.
Ethical disclosure
Protection of human and animal subjects. The authors declare that the procedures followed were in accordance with the regulations of the relevant clinical research ethics committee and with those of the Code of Ethics of the World Medical Association (Declaration of Helsinki).
Confidentiality of data. The authors declare that no patient data appear in this article.
Right to privacy and informed consent. The authors declare that no patient data appear in this article.
Funding
The study was financed through funds provided by the Convenio CPSS/ART.1°/023/2013 between Hospital Infantil de México Federico Gómez and the Comisión Nacional de Protección Social en Salud (CNPSS).
Conflict of interest
The authors declare that they have no conflicts of interest.
Acknowledgments
To Mónica Espinoza Aranda and Ángela María González García (Coahuila); Norberto Murillo Álvarez, Mario Abraham Ruíz Ruíz and Rosa Estela Martínez Rodríguez (Guanajuato); Mara Guadalupe Torres Ruíz and Luis Ricardo Palmer Serrano (Tabasco); Ana Alicia Jiménez Burgos, Marta Lia Pirola, Jessica Guadarrama Orozco, Guillermo Buenrostro Márquez, Elías Hernández Ramírez, Rocío del Carmen Córdoba García, Socorro De la Torre García, Miriam Oran Bautista (Hospital Infantil de México Federico Gómez); Esther Valadez Correa (Comisión Nacional de Protección Social en Salud); Paula Angélica Hernández Olmos and Hugo Erick Zertuche Guerrero (Coordinación Nacional de PROSPERA Programa de Inclusión Social).
Received 1 November 2015;
accepted 2 November 2015
* Corresponding author.
E-mail:antoniorizzoli@gmail.com (A. Rizzoli-Córdoba).