Multi-method assessment of ADHD characteristics in preschool children: Relations between measures

https://doi.org/10.1016/j.ecresq.2011.08.004Get rights and content

Abstract

Several forms of assessment tools, including behavioral rating scales and objective tests such as the Continuous Performance Test (CPT), can be used to measure inattentive and hyperactive/impulsive behaviors associated with Attention-Deficit/Hyperactivity Disorder (ADHD). However, research with school-age children has shown that the correlations between parent ratings, teacher ratings, and scores on objective measures of ADHD-characteristic behaviors are modest at best. In this study, we examined the relations between parent and teacher ratings of ADHD and CPT scores in a sample of 65 preschoolers ranging from 50 to 72 months of age. No significant associations between teacher and parent ratings of ADHD were found. Parent-ratings of both inattention and hyperactivity/impulsivity accounted for variance in CPT omission errors but not CPT commission errors. Teacher ratings showed evidence of convergent and discriminant validity when entered simultaneously in a hierarchical regression. These tools may be measuring different aspects of inattention and hyperactivity/impulsivity.

Highlights

► Correlations between different ADHD assessment tools were small. ► Parent-ratings were uncorrelated with teacher-ratings of ADHD-related behaviors. ► Teacher-ratings of ADHD-related behaviors were uncorrelated with errors on the CPT. ► Parent-ratings of ADHD-related behaviors were correlated with CPT omission errors.

Introduction

Despite substantial research on the neurological, genetic, and environmental causes of Attention Deficit/Hyperactivity Disorder (ADHD), no one factor has been identified as the primary cause of the disorder. Furthermore, researchers have posited theories of multiple pathways to ADHD, which may preclude the discovery of a single definitive test for identifying ADHD. As such, it is likely that high-quality ADHD diagnostic procedures will always benefit from a combination of informant ratings, background information (e.g., school records), and neuropsychological tests.

ADHD is characterized by three distinct forms of problem behaviors: inattention, hyperactivity, and impulsivity (American Psychological Association, 2000). Inattention refers to difficulties focusing and ignoring distractions. Hyperactivity is represented by difficulties with behaviors such as remaining still and engaging in a single activity for extended periods of time. Impulsivity is characterized by behaviors such as often interrupting and intruding during social interactions and academic situations. To warrant a diagnosis of ADHD, these symptoms must be causing the child impairment in two or more settings, which may include school or preschool. However, research has shown that behaviors associated with ADHD relate to the development of academic skills (Lonigan et al., 1999) and social development (Rydell, Diamantopoulou, Thorell, & Bohlin, 2009) even in samples that include children without clinically elevated levels of these behaviors.

Some common tools that have been used in practice and research to measure inattentive and hyperactive/impulsive behaviors include behavioral rating scales that are completed by individuals who interact often with the child and objective tasks such as the Continuous Performance Test (CPT). However, each assessment tool has limitations and drawbacks; no single tool has been identified as a “gold standard” for determining the presence of ADHD-characteristic behaviors. Although informant rating scales and objective tests both have been used to measure ADHD-characteristic behaviors, research has demonstrated generally low to moderate correlations between scores on each of these measures in school-age children (e.g., Achenbach, McConaughy, & Howell, 1987). Even less is known regarding the relations between these measures in preschool children. Given the recognized link between behavior problems in preschool and later elementary school (e.g., Willcutt & Pennington, 2000), efforts to measure effectively ADHD-characteristic behaviors in preschool and use these measures to predict the course of symptomatology have recently increased (Lahey et al., 2005, Re and Cornoldi, 2009). A greater understanding of the tools used to assess ADHD in school-age children is needed for preschool populations. The purpose of this study was to examine the overlap between parent-ratings of ADHD, teacher-ratings of ADHD, and CPT scores in a preschool sample and determine the convergent and discriminant validity across these measures.

Parents often have the most contact with clinicians and, thus, serve as a valuable resource for providing information. However, several characteristics of parent-ratings raise concerns regarding possible biases. Given the heritability of ADHD (e.g., Epstein et al., 2000), many parents of children with the disorder may also be afflicted. Parental symptomatology may impact ratings, which could compromise the validity of parent-ratings (Sayal & Taylor, 2005). Compared to other informants, parents also tend to report more symptomatology in children (Re & Cornoldi, 2009). This raises concerns of potential over-reporting of symptoms by parents.

Given the need to verify the presence of symptoms in multiple settings to diagnosis ADHD, parents may be asked to report on their child's school behavior if contacting the teacher is inconvenient. However, Sayal and Taylor (2005) found that when comparing parent-ratings of school and home behavior and teacher-ratings of school behavior, parent-ratings of school behavior correlated more strongly with parent-ratings of home behavior than with teacher-ratings of school behavior. This suggests that parents do not accurately infer how a child behaves in school and most likely base their ratings on the child's behavior at home. Parent reports of school behavior also would be secondhand through information gained from teachers and, therefore, may vary in part as a function of the degree and quality of contact between parent and teacher. These factors may become problematic when attempting to assess the pervasiveness of a disorder using only parent report. This suggests that an understanding of children's manifestation of the ADHD characteristics across domains requires additional informants.

Problems in school are often the primary reason behind referrals for ADHD treatment (Loe & Feldman, 2007), making teachers important informants of children's behavior. Teachers are sometimes considered ideal informants for behavioral reports because they interact with children over long periods of time in a wide range of settings that vary in structure (e.g., Evans, Allen, Moore, & Strauss, 2005). Furthermore, teachers interact with many children of the same age, giving them a better reference point for making decisions regarding what behaviors should be deemed atypical. Correlations between teacher-ratings at different times and concurrent agreement between individuals with teacher roles (e.g., teachers and teacher's assistants) have been shown to be high (Loughran, 2003).

Although teacher-ratings are widely used in the assessment of ADHD-characteristic behaviors, this method of evaluation is not without criticism. Compared to assessments conducted by examiners from outside of the classroom, assessments of educational outcomes conducted by teachers contain a large proportion of assessor-level variance, raising concerns that teacher-level factors influence assessment scores (Waterman, McDermott, Fantuzzo, & Gadsden, in press). Concerns related to extraneous factors that may impact teacher assessments are magnified when considering behavioral problems, for which there is no “gold standard” for determining accuracy. Several potential biases have been noted in the literature. Just as parents tend to report higher levels of behavior problems than do teachers, teachers have been shown to report higher levels of behavior problems than do trained observers (Phillips & Lonigan, 2010), suggesting that teachers also may engage in over-reporting. There is evidence that teachers may be more likely to identify minority students, compared to non-minority students, as exhibiting hyperactive/inattentive behavior (Nolan, Gadow, & Sprafkin, 2001). These findings indicate that factors other than observed child behavior may affect teachers’ ratings. Furthermore, the stability of cross-grade teacher-ratings of clinically significant inattentive behaviors has also been shown to be low (Rabiner et al., 2010), making the utility of teacher-ratings for predicting future behavior questionable.

Teachers’ abilities to differentiate ADHD behaviors from symptoms of other disorders also have been challenged. Findings in several studies (e.g., Abikoff, Courtney, Pelham, & Koplewicz, 1993) have demonstrated that teachers often rate children with Oppositional Defiant Disorder (ODD) as having elevated levels of inattention and hyperactivity even when no behaviors directly indicative of these symptoms have been displayed. This bias, referred to as the halo effect, appears to be bi-directional such that teachers also rate children with ADHD characteristics with elevated oppositional scores (Hartung et al., 2010). These findings suggest a bias that may inflate teacher-ratings of ADHD in children with other comorbid behavior problems.

Although multiple informants are used often for diagnostic purposes, there is a growing body of literature suggesting only modest associations between parent and teacher ratings of ADHD (Collett, Ohan, & Myers, 2003). Low inter-rater agreement is found not only for clinical elevations of ADHD-characteristic behaviors, but also for parent and teacher ratings of more general behavior problems and social skills (Winsler & Wallace, 2002). Observed statistical relations may vary somewhat depending on the unit and method of analysis. For example, analysis of data conducted dimensionally, rather than categorically, produces results with higher estimates of parent-teacher agreement (Mitsis, McKay, Schulz, Newcorn, & Halperin, 2000). Furthermore, parent- and teacher-ratings of ADHD-characteristic behaviors are similar in that both typically rate boys as exhibiting more problematic behaviors than girls (Newcorn et al., 2001). Still, correlations between raters are generally low in school-age samples. Research with preschoolers has also shown low correlations between teacher and parent ratings of each symptom domain (Murray et al., 2007). Taken together, these findings suggest that there are inconsistencies across informants.

Discrepancies between informant ratings have created controversy regarding which informant is better at predicting diagnosis. In a study by Power et al. (1998), teacher-ratings of inattention and hyperactivity demonstrated better predictive validity than parent ratings for determining the presence or absence of an ADHD diagnosis as determined by a multi-method assessment battery including a diagnostic interview. However, the associations for both parent and teacher ratings ranged from only low to moderate. This study also found that teacher ratings were generally more useful than parent ratings for distinguishing between subtypes (e.g., primarily inattentive, primarily hyperactive/impulsive, combined inattentive and hyperactive) of children with ADHD. Other studies have found similar results (e.g., Owens & Hoza, 2003), although at least one study has found conflicting results showing that parent-ratings yielded more powerful subtype differentiation (DuPaul et al., 1998). The high sensitivity of parent-ratings supports their use as screening measures. However, the low specificity suggests that further inquiry is needed to minimize false positives (Tripp, Schaughency, & Clarke, 2006).

Although teacher ratings are often seen as a valid source of information for determining the presence of ADHD-characteristic behaviors because of teachers’ extensive experiences with children, some evidence has suggested that combining ratings from both teachers and parents is optimal for predicting diagnosis compared to relying on either rater alone (Power et al., 1998). This indicates that each informant may provide useful, but distinct, information regarding symptomatology. However, the discrepancies between raters brings to light the question of how each of these informant ratings relates to objective measures that are presumed to measure inattentive and hyperactive/impulsive behaviors without the problem of subjective bias.

Given concerns regarding bias and the common inter-rater disagreement found between informant ratings, measures that objectively assess behaviors associated with ADHD are of interest to circumvent subjective bias. Objective measures provide an opportunity to support or disconfirm the subjective ratings made by informants. The CPT is a computer-administered tool that has been reported to assess inattentive and hyperactive/impulsive behaviors. Originally developed to test for severe brain damage (Rosvold, Mirsky, Sarason, Bransome, & Beck, 1956), the CPT requires individuals to view a stimulus sequence on a screen while responding to target stimuli and withholding responses to non-target stimuli.

Several versions of CPTs exist that vary in stimulus presentation and complexity, making them more or less suitable for young children. Variations include the stimulus’ time interval on the screen and the time interim between displayed stimuli, types of stimuli, and task complexity. Simpler CPT versions require a response if a certain stimulus is presented, whereas complex versions require a response to a target stimulus contingent on whether it follows another particular stimulus. These versions of the task, such as the original A-X task (Rosvold et al., 1956), contain a working memory component that may exceed the cognitive abilities of preschoolers. Generally, both preschoolers and school-age children commit more errors on complex tasks than on simple tasks (Oades, 2000). The original task also uses letters as stimuli, creating a concern that facility with letters may affect performance and confound results (McGee, Clark, & Symons, 2000). These concerns precipitated the creation of CPTs, such as the Test of Variables of Attention (T.O.V.A.; Greenberg, Kindschi, Dupuy, & Hughes, 1993), that are designed to have minimal language demands. Across different CPT versions, performance tends to vary as a function of age such that younger children generally produce more errors (Lin, Hsiao, & Chen, 1999). Gender differences have also been found, but may vary as a function of the CPT versions. Boys make more errors than do girls on typical versions of the CPT (Greenberg & Waldmant, 1993), but girls make more errors than do boys when processing demands are added to the task (e.g., CPTs with visually degraded stimuli; Lin et al., 1999).

A large body of research demonstrates the usefulness of the CPT for differentiating individuals with ADHD from controls in school-age samples (e.g., Oades, 2000). The CPT has also been found to have adequate test–retest reliability across a range of time spans (time elapsed = 4.8 months, rs = .65 to .74; time elapsed = one week, rs = .74 to .87; time elapsed = 90 min, rs = .70 to .87) and adequate split-half reliability (rs = .67 to .92) comparing performance on the 1st and 4th quarter blocks of the task to performance on the 2nd and 3rd blocks of the task (Halperin et al., 1991, Leark et al., 2004). However, despite the adequate psychometric properties of the test and almost ubiquitous findings of abnormal performance by children with ADHD on CPTs, its utility as a primary diagnostic tool is weak (Nigg, Hinshaw, & Halperin, 1996) and its negative predictive power is low (Grodzinsky & Barkley, 1999). Although abnormal performance on the CPT indicates a high likelihood of an ADHD diagnosis, many children who are diagnosed with ADHD are able to obtain non-elevated scores on the test, making it insufficient for diagnosing ADHD.

Researchers’ attempts to ascribe a particular performance characteristic to specific subtypes also are inconsistent and sometimes contradictory. An omission error, which occurs when the child fails to respond to a target stimulus, is presumed to indicate inattention. A commission error, which occurs when the child responds to a non-target stimulus, is presumed to measure hyperactivity/impulsivity. Many studies have assumed this view; however, many studies also report that that these error types are not related to symptom dimensions in hypothesized ways (Chhabildas, Pennington, & Willcutt, 2001) or relate indiscriminately to both types of symptomatology (Epstein et al., 2003).

Research with older children often demonstrates inconsistent relations between the CPT and informant ratings of ADHD-characteristic behavior. Egeland, Johansen, and Ueland (2009) found that commission errors on the Conners’ Continuous Performance Test (C-CPT; Conners, 2002) showed no significant relation to parent- or teacher-rated ADHD behaviors in a sample of school-age children. Omission scores were significantly correlated with both parent and teacher ratings of inattention and only with parent ratings of hyperactivity. A study by McGee et al. (2000) failed to demonstrate any significant relations between overall scores on the C-CPT and parent and teacher ratings of ADHD. Omission errors were modestly related to teacher-ratings of hyperactivity – a result inconsistent with the hypothesis that omission errors specifically relate to poor attention. Epstein et al. (2003) examined the relations with item-level data of parent ratings and found similar inconsistencies. Stronger relations between CPT parameters and informant ratings are found when parent and teacher ratings are combined as composite variables (Avila, Cuenca, Felix, Parcet, & Miranda, 2004). However, the use of a composite score precludes the ability to inspect the unique associations for each observer. Taken together, these results indicate that in school-age samples, relations between objective measures and informant ratings of ADHD-characteristic behaviors are weak and inconsistent.

In analyses that examine ADHD-characteristic behaviors, several factors also associated with these constructs warrant consideration. For example, cognitive ability has been shown to impact performance on CPT-type tasks (Weyandt, Mitzlaff, & Thomas, 2002). Younger children typically produce more errors on the CPT (Lin et al., 1999) and are generally rated as having higher levels of inattentive and hyperactive/impulsive behaviors (Lahey et al., 2005) compared to older children, making age a relevant factor. Studies examining gender differences often find boys to be rated as exhibiting higher levels of ADHD-characteristic behaviors (e.g., Gershon & Gershon, 2002). Finally, socio-economic status (SES) may also influence the associations between measures of inattentive and hyperactive/impulsive behavior given that ADHD has been shown to relate to SES (Offord et al., 1987). Consequently, these potentially confounding factors should be controlled for when analyzing the relations between measures of ADHD-characteristic behaviors.

Several forms of assessment are used to assess ADHD-characteristic behaviors, each with its own strengths and weaknesses. Although these assessment tools all are presumed to measure inattentive and hyperactive/impulsive behaviors, research has demonstrated low correlations between these measures. Few studies examine all three of these tools within the same sample, making it possible to examine convergent and discriminant validity across the measures, and none have done so in preschoolers, a population increasingly targeted for ADHD assessment and intervention. The purpose of this study was to examine the convergence and divergence between teacher ratings of ADHD, parent ratings of ADHD, and CPT performance in preschool children. Although evidence regarding the convergent and discriminant validity of each of these measures is mixed, we based our hypotheses on the conceptualizations of each measure as presented in the literature. It was expected that teacher-ratings and parent-ratings of inattention would be associated significantly with each other and with omission errors on the CPT. Teacher ratings of hyperactivity/impulsivity and parent-ratings of hyperactivity/impulsivity were expected to be associated with each other and with commission errors on the CPT.

Section snippets

Participants

Participants included 65 preschoolers from a larger study designed to identify characteristics of children who do and do not go on to develop serious problems in learning to read. Children were recruited from 10 preschools in Northeast Florida. Informed consent was obtained by parents prior to their children's participation in the project. Participants included the children from this larger study who completed the CPT, whose teachers completed the Conners’ Teacher Rating Scale (CTRS-44), and

Results

Hyperactivity/Impulsivity and Inattention subscale scores were created based on the IRT analysis reported by Purpura and Lonigan (2009). Descriptive statistics for raw scores are presented in Table 1. Given that age is related to CPT performance and overall levels of ADHD-characteristic behavior, age-standardized scores for each subscale were computed by regressing raw scores on age. Outliers were identified as those values outside two inter-quartile ranges from the median. Outliers were

Discussion

The purpose of this study was to examine how teacher ratings, parent ratings, and objective measures of inattentive and hyperactive/impulsive behaviors relate in a preschool sample, both accounting for and not accounting for the high within-informant overlap in ratings of each behavior. The results of this study highlight the inconsistencies between assessments used to measure inattentive and hyperactive/impulsive behaviors. Despite using the same items to identify children as exhibiting

References (55)

  • N. Chhabildas et al.

    A comparison of the neuropsychological profiles of the DSM-IV subtypes of ADHD

    Journal of Abnormal Child Psychology

    (2001)
  • J. Cohen

    A power primer

    Psychological Bulletin

    (1992)
  • K.C. Conners

    Conners’ Rating Scales, revised: User's manual

    (1997)
  • K.C. Conners

    Conners’ Continuous Performance test (CPTII). Technical guide and software manual

    (2002)
  • G.J. DuPaul et al.

    Parent-ratings of ADHD symptoms: Factor structure and normative data

    Journal of Psychopathology and Behavioral Assessment

    (1998)
  • G.J. DuPaul et al.

    Multi-method assessment of Attention Deficit Hyperactivity Disorder: The diagnostic utility of clinic-based tests

    Journal of Clinical Child Psychology

    (1992)
  • J. Egeland et al.

    Differentiating between ADHD sub-types on CCPT measures of sustained attention and vigilance

    Scandinavian Journal of Psychology

    (2009)
  • J.N. Epstein et al.

    Familial aggregation of ADHD characteristics

    Journal of Abnormal Child Psychology

    (2000)
  • J.N. Epstein et al.

    Relations between continuous performance test performance measures and ADHD behaviors

    Journal of Abnormal Child Psychology

    (2003)
  • S.W. Evans et al.

    Measuring symptoms and functioning of youth with ADHD in middle school

    Journal of Abnormal Child Psychology

    (2005)
  • M. Fischer et al.

    Who are the false negatives on continuous performance tests?

    Journal of Clinical Child Psychology

    (1995)
  • R.R. Gerhardstein et al.

    Factor structure of the Conners’ Teacher Rating Scale—short form in a low-income preschool sample

    Journal of Psychoeducational Assessment

    (2003)
  • J. Gershon et al.

    A meta-analytic review of gender differences in ADHD

    Journal of Attention Disorders

    (2002)
  • L.M. Greenberg et al.

    Test of Variables of Attention continuous performance test

    (1993)
  • L.M. Greenberg et al.

    Developmental normative data on the Test of Variables of Inattention (T.O.V.A.)

    The Journal of Child Psychology and Psychiatry

    (1993)
  • G.M. Grodzinsky et al.

    Predictive power of frontal lobe tests in the diagnosis of attention deficit hyperactivity disorder

    The Clinical Neuropsychologist

    (1999)
  • J.M. Halperin et al.

    Assessment of the Continuous Performance Test: Reliability and validity in a nonreferred sample

    Psychological Assessment

    (1991)
  • Cited by (0)

    This work was supported by grants from the National Institute of Child Health and Human Development (HD052120) and the Institute of Education Sciences (R305B04074, R305B090021). Views expressed herein are solely those of the authors and have neither been reviewed nor cleared by the grantors.

    View full text