Psychometric properties of the Cognitive Reserve Questionnaire for Adolescents (CoRe-A)

Abstract

Introduction

The objective is to design and analyze the psychometric properties of a new instrument, The Cognitive Reserve Questionnaire for Adolescents (CoRe-A), for assessing cognitive reserve in adolescents.

Methods

A systematic review was conducted to identify cognitive reserve components, before items concerning them were discussed in terms of their suitability, relevance, and clarity. The final 12-item CoRe-A was subsequently validated among 48 adolescents diagnosed with severe mental disorders and 37 healthy controls matched by age and sex.

Results

The CoRe-A scale showed a four-factor structure (i.e., education/occupation, sociability, school performance/development, and leisure activities) that explained 65.30% of the variance. It had adequate internal consistency (Cronbach's alpha coefficient = 0.767) and test–retest reliability (r = 0.979; p < 0.001). The patients with severe mental disorders obtained significantly lower scores than the healthy controls in both the total CoRe-A score and in the four factor scores. Moreover, the CoRe-A correctly classified 85.9% of the sample as patient with severe mental disorders or healthy control. The CoRe-A total score showed a large positive correlation with the Wechsler vocabulary subtest, demonstrating good convergent validity (r = 0.514; p < 0.001). Nevertheless, when vocabulary subtest and CoRe-A total score were compared, the CoRe-A reached a better discriminative capacity.

Conclusions

The CoRe-A is a valid and reliable tool for assessing cognitive reserve in adolescents, and it may help to characterize adolescents diagnosed with a severe mental disorder.

Introduction

Cognitive reserve (CR) has been defined as the ability of the brain to cope with physiological or pathological brain damage. The concept emerged from the repeated observation that, across individuals, there is no direct relationship between the severity of brain damage and the degree of disruption in clinical and cognitive performance.^1–3 CR is considered a factor that allows people to tolerate higher levels of neural damage before displaying clinical symptoms,⁴ and it has been hypothesized that people with higher CR have more flexible brain network utilization.

In the field of psychiatry, there are very few studies about CR, and most of these have been conducted with adult samples. It has been shown that higher CR in adults with a first episode of psychosis (FEP) is associated with lower levels of psychotic symptoms, higher psychosocial functioning, and higher neurocognitive performance, suggesting that CR may be a protective factor.^5–9 In addition, longitudinal studies in patients with FEP have shown the predictive capacity of CR over some clinical, psychosocial, and cognitive outcomes over time,^5,10 indicating that CR may help to determine prognosis. To our knowledge, there is only one study conducted with children and adolescents with first episode of schizophrenia spectrum disorder. This study, conducted by our group, de la Serna et al.,¹¹ showed the predictive capacity of CR on working memory and attention in adolescents diagnosed with a first episode of schizophrenia spectrum disorder at two-year follow-up.

CR has been assessed differently in different studies. In non-psychiatric samples, Staff et al.¹² used intracranial volume to measure CR and found it was not as representative as occupational and educational level. Other authors have estimated CR by using intelligence,^13–15 educational and occupational level,^4,16 or leisure activities,¹⁷ either separately or in combination.^7,18,19 Several tools have been developed for the assessment of CR in adult healthy populations and patients with dementia, including “Lifetime of Experiences Questionnaire (LEQ)”,²⁰ “Cognitive Reserve Questionnaire (CRQ)”,²¹ “Cognitive Reserve Index Questionnaire (CRI-q)”,²² and “Cognitive Reserve Scale (CRS)”.²³

Among patients diagnosed with psychiatric disorders, the most common proxies for assessing CR are the premorbid intellectual quotient (IQ), occupational attainment, educational level, and leisure activities.^5,6,8,9,11 Previous studies conducted with a psychiatric sample of children and adolescents considered parental educational and occupational level, school performance and developmental difficulties as components of CR.^11,24 We could only identify one questionnaire that was developed to assess CR in psychiatric samples, but the “Cognitive Reserve Assessment Scale in Health (CRASH)” created by Amoretti et al.,²⁵ was designed to be used with adults with severe mental illness.

Although the CRASH is undoubtedly of use, there are relevant differences between adults and adolescents to be considered when assessing CR. For adolescents, we must not only consider the length of education but also the education and occupation of their parents. Another aspect that should be considered is that childhood must be considered (e.g., developmental skills in terms of language, motor functions, and reading and writing ability). Leisure activities and hobbies also differ dramatically between adolescents and adults, while information must also be collected about academic performance during childhood and adolescence. Thus, there is a need for a specific instrument to quantify CR in adolescents.

In this study, we designed and validated a new instrument to assess CR in adolescents, entitled the Cognitive Reserve Questionnaire for Adolescents (CoRe-A). We analyzed its psychometric properties and the differences between healthy controls (HC) and patients with severe mental disorders (SMD) during childhood or adolescence.

Methods

Subjects

The sample included 48 patients diagnosed with SMD during childhood and adolescence. Of these, 28 were diagnosed with a schizophrenia spectrum disorder (21 with schizophrenia, 6 with schizoaffective disorder, and 1 with schizophreniform disorder), 8 with a bipolar disorder with psychotic symptoms, 5 with depression with psychotic symptoms, and 7 with psychosis not otherwise specified. The mean duration of the SMD was 18.25 ± 17.55 months. All the SMD group had had only one episode of psychosis in the moment of the evaluation except for 12 patients with a mean number of episodes of 3.00 ± 1.47. We also recruited and assessed a group of 37 HCs through advertisements posted in primary health care centers and other community locations within the same geographical area as patients matched by age and sex.

This study was conducted in accordance with the ethical principles of the Declaration of Helsinki, Good Clinical Practice, and the Hospital Clinic Ethics and Research Board. The Clinical Research Ethics Committee of the Hospital Clinic of Barcelona approved the research project. Written informed consent was obtained from all legal guardians and from participants older than 12 years before inclusion in the study.

Inclusion and exclusion criteria

The inclusion criteria for patients were being aged 12–18 years and having a diagnosis of SMD (e.g., schizophrenia, schizoaffective disorders, bipolar I or II, depression with psychotic symptoms, or psychosis not otherwise specified) according to DSM-IV-TR²⁶ criteria. The exclusion criteria were as follows: (a) presence of concomitant axis I disorder at assessment that could account for the psychotic symptoms, including substance-induced psychotic disorder; (b) intelligence quotient (IQ) below 70 with impaired functioning; (c) autism spectrum disorder; and (d) neurological disorders, including history of head trauma with loss of consciousness. Occasional and regular substance use was not an exclusion criterion if positive symptoms persisted for more than 2 weeks after a negative urine toxicology test and if a substance-induced psychotic disorder was not diagnosed.

The inclusion criteria for the HC were as follows: (a) similar age and sex to patients; (b) from the same geographical areas as patients; (c) no psychiatric disorder, as measured by the K-SADS-PL²⁷; (d) and no neurological disorders, head trauma, pregnancy, or mental retardation (per DSM-IV-TR²⁶ criteria). The exclusion criteria included: (a) IQ below 70 with impaired functioning; (b) autism spectrum disorder, and (c) neurological disorders, including a history of head trauma with loss of consciousness.

Clinical and sociodemographic assessment

Sociodemographic data were systematically obtained for all participants. Socioeconomic status was estimated using the Hollingshead Redlich scale,²⁸ which was administered to parents by a clinician. This scale created a score that was the sum of multiplying the education score by 3 and the occupation score by 5. The education score ranges from 1 (less than seventh grade) to 7 (graduate professional training). The occupation score ranges from 1 (farm laborers or menial service workers) to 9 (higher executives or proprietors of large businesses). The final score resulted is classified in a five-level scale that indicates socioeconomic status (high, medium-high, medium, medium low, low).

Diagnosis was made according to the DSM-IV-TR criteria²⁶ using the K-SADS-PL,²⁷ which was administered to parents and children in separate interviews. Those older than 18 years were assessed using the Structured Interview for DSM-IV-TR²⁶ disorders (SCID-I).²⁹ To assess psychotic symptoms, the validated Spanish version of the Positive and Negative Symptom Scale (PANSS)³⁰ was used.³¹

Intelligence assessment

An estimation of the IQ was made by assessing the vocabulary subtest of the Wechsler Intelligence Scale for Children (WISC-IV)³² or Wechsler Adult Intelligence Scale (WAIS-IV).³³ Direct scores were translated into standard scores with a mean of 10 and a standard deviation of 3.

Cognitive Reserve Questionnaire

CR was assessed by the CoRe-A, as developed by the Child and Adolescent Psychiatry and Psychology Service at Hospital Clinic in Barcelona. We conducted an in-depth review of the existing literature about CR and the way it has been measured to date (ES, SA, and OP). This review showed that IQ, leisure activities, social activities, educational attainment, and occupational attainment were the most representative concepts for assessing CR in adult samples. Each specialist then created questions to devise potential items for each concept that took into account the age (adolescence) of the reference population. All generated items were discussed in terms of their suitability and relevance, and minor modifications were included when necessary.

The final version of the CoRe-A is a 12-item questionnaire covering both childhood and adolescence. It consistently takes fewer than 10 min to complete and gives a total score (maximum, 62) by adding the scores for each item, with higher score indicating greater CR.

Statistical analysis

All analyses were performed using IBM SPSS Version 23 (IBM Corp., Armonk, NY, USA) except for the factor analysis, which was calculated using R. Statistical significance was set at the p < 0.05 level. We tested the normality of the sample distribution using the Kolmogorov–Smirnov test, with the Levene test used to assess the equality of variances. Categorical sociodemographic variables were analyzed using the chi-square test, whereas continuous variables were compared between groups using an ANOVA test.

A confirmatory factorial analysis (CFA) was conducted to test if the CoRe-A questionnaire followed the hypothesized four-factor structure. The following indices were used to assess goodness-of-fit of the model tested: χ², df, the comparative fit index (CFI), the Tucker-Lewis index (TLI), the goodness-of-fit index (GFI), the root mean square error of approximation (RMSEA) and the RMSEA confidence interval. The internal consistency and reliability of the CoRe-A were assessed using Cronbach's alpha. Convergent validity for the CoRe-A (correlations between the CoRe-A and vocabulary subtest of the WISC-IV or WAIS-IV), and the test–retest reliability (CoRe-A administered 15 days from the first assessment in sub-samples of the SMD (N = 34) and HC (N = 29).

To determine if there were significant differences between groups in CR, an ANOVA test was conducted to compare the CoRe-A total score and factor scores between the SMD and HC groups. We then studied the discriminative capacity of the CoRe-A (to distinguish subjects with SMD from HC) by receiver operating characteristic (ROC) curve and logistic regression analysis. Additional ROC curve analysis was executed with the vocabulary subtest score to compare which variable (CoRe-A total score or CR assessed with vocabulary) better discriminated between the SMD and HC groups. The DeLong test was also conducted for this objective.

Results

Sociodemographic characteristic of the sample

No significant differences were found between the SMD and HC groups in sex and age, but parental socioeconomic status was significantly lower in the SMD group than in HC group. Table 1 summarizes the sociodemographic and clinical variables for each group.

Table 1.

Sociodemographic and clinical variables of the samples.

	HCN = 37Mean ± SD	SMDN = 48Mean ± SD	t/χ^2	p-Value
Sex: Female (%)	17 (45.94%)	19 (39.58%)	0.346	0.556
Age (years)	15.32 ± 1.93	14.98 ± 1.73	0.751	0.389
Socioeconomic status (%)			11.167	0.048
High	18 (48.64%)	11 (22.91%)
Medium–High	7 (18.91%)	6 (12.50%)
Medium	5 (13.51%)	13 (27.08%)
Medium–Low	6 (16.21%)	11 (22.91%)
Low	1 (2.70%)	7 (14.58%)
Clinical assessment
PANSS positive		20.33 ± 6.87
PANSS negative	NA	19.04 ± 6.46
PANSS general		38.21 ± 9.84
PANSS total		77.42 ± 17.50

Data are shown as mean ± standard deviation, unless otherwise stated.

Abbreviations: CoRe-A, Cognitive Reserve Questionnaire for Adolescents; HC, Healthy Controls; NA, Not Applicable; PANSS, Positive and Negative Syndrome Scale; SMD, Severe Mental Disorder.

Factor analysis

CFA was carried out with the entire sample. We hypothesized the existence of a four-factor structure for the CoRe-A that included education and occupation (factor 1), sociability (factor 2), school performance and development (factor 3) and leisure activities (factor 4). Tables 2 and 3 summarizes the CFA results. All the indices indicated a good fit between the model and the data. Therefore, the CFA confirmed a four-facto structure of the CoRe-A.

Table 2.

Factor loading of CoRe-A items.

Variable	Factor IEducation and occupation	Factor IISociability	Factor IIISchool performance and development	Factor IVLeisure activities
4. Adolescent non-regulated studies	0.474**
5. Parental educational level	0.839
6. Parental occupational level	0.867
9. Leisure activities with family		0.546**
10. Sociability, shyness level		0.649*
11. Number of friends		0.691*
12. Social leisure activities		0.575**
1. Actual situation of studies			0.731*
2. School performance			0.411*
3. Development skills			0.791**
7. Intellectual hobbies				0.165**
8. Physical exercise				1.000

Abbreviations: CoRe-A, Cognitive Reserve Questionnaire for Adolescents.

*

Significance level p < 0.05.

**

Significance level p < 0.001.

Table 3.

Confirmatory factor analysis model fit summary for CoRe-A.

	χ^2	df	CFI	TLI	GFI	RMSEA (CI 95%)
CoRe-A	50.38	48.00	0.99	0.99	0.96	0.02 (0.00–0.07)

Abbreviations: CoRe-A = Cognitive Reserve Questionnaire for Adolescents; CFI = comparative fit index; TLI = Tucker–Lewis index; GFI = goodness of fit index; RMSEA = root mean square error of approximation; RMSEA CI = root mean square error of approximation confidence interval.

The education and occupation factor included items measuring parental education and occupation as well as the subject's non-regulated studies. Sociability was measured by the quantity and quality of social relationships, the quantity and frequency of family social leisure activities, and the quantity and frequency of other social leisure activities. School performance and development included items concerning the current study situation, their school performance, and developmental skills. Leisure activities were measured with items regarding the quantity and frequency of each subject's intellectual hobbies and physical exercise.

Internal consistency and convergent validity

Cronbach's alpha of the full CoRe-A scale demonstrated adequate internal consistency (α = 0.767). As for the convergent validity, the CoRe-A was significantly correlated with vocabulary subtest in the whole sample (r = 0.514, p < 0.001). Test–retest reliability was studied in a sub-sample of 29 HC and 34 SMD patients. The analysis revealed high reliable scores on total CoRe-A (r = 0.979, p < 0.001).

Differences in CR between the HC and SMD groups

Significant differences in the total CoRe-A score existed between groups (F = 94.578, p < 0.001), being significantly higher in the HC group than in SMD group. Moreover, significant differences were observed in each factor between the SMD and HC groups, as follows: education and occupation (F = 40.086; p < 0.001), sociability (F = 38.761; p < 0.001), school performance and development (F = 105.565; p < 0.001), and leisure activities (F = 10.816; p = 0.001). These differences are shown in Table 4.

Table 4.

Comparison of CoRe-A results between SMD and HC groups.

CoRe-A	HCN = 37(Mean ± SD)	SMDN = 48(Mean ± SD)	F	p
Education and occupation	0.18 ± 0.03	−0.14 ± 0.03	40.086	<0.001
Sociability	0.40 ± 0.08	−0.30 ± 0.07	38.761	<0.001
School performance and development	0.87 ± 0.11	−0.67 ± 0.09	105.565	<0.001
Leisure activities	0.11 ± 0.04	−0.08 ± 0.04	10.816	0.001
Total score	45.94 ± 1.14	31.13 ± 1.00	94.578	<0.001

Abbreviations: CoRe-A, Cognitive Reserve Questionnaire for Adolescents; HC, Healthy Controls; SMD, Severe Mental Disorder.

Discriminative capacity

Logistic regression was performed to assess the discriminative capacity of the CoRe-A total score for clinical groups (SMD or HC). The Hosmer–Lemeshow goodness-of-fit test result was 8.364 with a significance level of 0.302, which supports the model. This showed that the CoRe-A explained between 52.3% (Cox and Snell R square) and 70.1% (R Nagelkerke square) of the variance in the dependent variable and correctly classified 85.9% of the sample as SMD or HC (B = −0.327; p < 0.001).

ROC curve analysis was conducted to analyze the accuracy of the CoRe-A for discriminating SMD from HC. The area under the curve (AUC) was 0.937 for a CoRe-A cutoff value of 39.5, which showed high sensitivity (81.1%) and specificity (89.6%). ROC curves analyses for each factor separately were also conducted. The area under the curve (AUC) for the factor 1 (Education and occupation) was 0.834 for a cutoff value of 0.006, which showed high sensitivity (78.4%) and specificity (75.0%). The AUC for the factor 2 (Sociability) was 0.822 for a cutoff value of −0.049, which showed high sensitivity (91.9%) and specificity (66.7%). The AUC for the factor 3 (School performance and development) was 0.942 for a cutoff value of 0.014, which showed high sensitivity (97.3%) and specificity (85.4%). The AUC for the factor 4 (Leisure activities) was 0.596 for a cutoff value of −0.083, which showed high sensitivity (89.2%) and specificity (58.3%).

An additional ROC curve analysis was performed using the premorbid IQ (assessed with the vocabulary subtest of the WISC-IV or WAIS-IV, depending on age). The AUC was 0.750, and the vocabulary subtest showed a higher sensitivity (97.2%) but a lower specificity (46.8%) at a cutoff value of 7.5. The DeLong test, performed to compare both ROC curves (Fig. 1), revealed that the CoRe-A score discriminated the SMD group significantly better than the vocabulary subtest (p < 0.001).

Fig. 1.

ROC curves of the CoRe-A and vocabulary subtest* for discriminating between the SMD and HC groups. * The vocabulary subtest of the Wechsler Intelligence Scale for Children (WISC-IV) or Wechsler Adult Intelligence Scale (WAIS-IV). Abbreviations: CoRe-A, Cognitive Reserve Questionnaire for Adolescents; HC, Healthy Controls; ROC, receiver operating characteristic; SMD, Severe Mental Disorder.

Discussion

We developed a specific instrument to assess CR in adolescents a range of SMDs, confirming the reliability and validity of the scale. To the best of our knowledge, no other instruments have been developed to evaluate CR in this population.

The CoRe-A combines various proxies of the CR in a single measure with a four-factor structure (i.e., education/occupation, sociability, school performance/development, and leisure activities). Factor analysis revealed that this explained 65.30% of the variance.

No existing questionnaire has sought to assess CR specifically during childhood and adolescence using items adapted to age, such as the assessment of developmental skills or parental socioeconomic status. Only two questionnaires considered the assessment of parental educational level as part of CR. Those factors were probably not considered in these earlier questionnaires because most of existing studies about CR have focused on adult populations where the importance of these variables is not considered as relevant. In adolescent samples, there is strong evidence that parental socioeconomic status can affect CR elements, such as physical and mental health, cognitive function, psychosocial well-being,^34,35 academic achievement and education, physical abilities, and even brain development.³⁵ Regarding language, writing/reading, and motor milestones, we know that intellectual hobbies such as reading/writing and physical activity affect CR directly.^36–38 In addition, due to the fact that schooling is regulated throughout childhood and adolescence and considering adolescents have not completed their educational stage yet, the school performance variable becomes very important because is directly related to some CR components as physical activity^39,40 and intelligence.⁴¹

Given that the target population of this questionnaire are adolescents, it could be of critical importance to include variables that may interfere in the development of these activities.

Assessment of the psychometric properties of the CoRe-A revealed good internal consistency and convergent validity when comparing the total CoRe-A and IQ scores between the SMD and HC groups. Research has already related the concept of CR with IQ. In patients with psychiatric disorders, IQ has been used as a proxy with other components, such as educational level, occupational attainment, or leisure activities.^5,6,8,9,11 Nonetheless, whereas CR is based on the acquisition of many abilities throughout life, IQ focuses only on intellectual performance.²² Consistent with this, although the correlation of the CoRe-A score and the IQ was statistically significant, confirming that a relationship exists between the two concepts, the association was not high enough to indicate that they can be consider interchangeable. Moreover, test–retest reliability was high (r = 0.979), suggesting that the CoRe-A score is stable over time.

The CR level was different between patients and controls, with the HC group having a higher CoRe-A score than the SMD group. This is consistent with the only study that has compared CR in adolescents with psychiatric disorders¹¹ and with other studies conducted in adults with SMD.^5,25 Moreover, the CoRe-A classified 85.9% of the sample correctly as either SMD or HC, doings so with high sensitivity (81.1%) and specificity (80.30%). AUC analysis also showed that IQ was inferior to the CoRe-A, while the DeLong test suggested that CoRe-A was better able to discriminate patients. Thus, although IQ is closely related to CR, it cannot be used in isolation because adding other factors, as in the CoRe-A, improves the predictive power. Comparable results were previously observed by our group in a sample of adolescents diagnosed with a schizophrenia spectrum disorder.¹¹ When we compared the predictive value of IQ and CR on neuropsychological functioning, CR was better able to predict cognitive functions than IQ.

The possible association between psychopathology and CR warrants highlighting. Previous research into CR in adolescent samples has been scarce, but we have previously found CR to be lower in adolescents diagnosed with a FEP than in HCs.²⁴ Moreover, the CR proxy, which has basically the same components as the CoRe-A, was able to predict psychotic symptoms, depressive symptoms, and psychosocial functioning at 5-year follow-up. Specifically, a high CR level was associated with fewer psychotic and depressive symptoms and higher psychosocial functioning.²⁴ Similar results were found in other studies conducted with adult patients diagnosed with FEP.^5,6 Further research is needed to study the relationship between CR and pathology in adolescents.

The main limitation of this study is the relatively small sample size that meant we could not divide the sample by diagnostic group. Dividing the group of patients into affective or non-affective psychosis, as done by Amoretti et al.,²⁵ may have revealed information about how the clinical characteristics inherent to each diagnosis influence the CR. Had we split the sample into diagnostic groups, the size of our sample means that we would have lost statistical power. In addition, although the classical test theory was suitable because of the modest sample size in this study, its use makes the statistics of our research more dependent on the nature of the sample. Nevertheless, our research also has significant strengths. The CoRe-A is the first instrument to be validated for assessing CR in adolescents and might help not only with the assessment of CR in young samples but also with predicting the clinical and cognitive outcomes in adolescents with SMD. The CoRe-A could also be a useful tool to select and design tailored therapeutic interventions and to predict treatment response.

In conclusion, we have developed the CoRe-A, a new instrument to assess CR in adolescents that has good internal consistency, good construct and convergent validity, and excellent reliability. It is expected that using the CoRe-A will contribute to a more consistent and homogeneous assessment of CR in adolescents with different disorders, and as a result, help to generate more replicable data. Longer studies should be conducted in psychiatric adolescent samples with the CoRe-A in order to confirm the structure of CR in these cohorts and to assess their association with psychopathology.

Funding

This study was supported by the Spanish Ministry of Health, Instituto de Salud Carlos III (PI070066; PI1100683; PI1500467; PI17/00741; PI1800696; PI18/00976), Ajut a la Recerca Pons Bartran (Fundació Clínic recerca Biomèdica), Fundació Marato TV3 (091630), and the Catalonia Government (2017SGR881), PERIS (SLT006/17/00346). It was co-financed by ERDF Funds from the European Commission, “A way of making Europe” and from CIBERSAM. These institutions had no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the paper for publication.

Conflicts of interest

Dr. Ilzarbe has received funding from the Spanish Ministry of Science, Innovation and Universities, Instituto de Salud Carlos III, ‘Rio Hortega’ contract CM17/00019, with the support of European Social Fund. Dr. Sugranyes received research support from a NARSAD Young Investigator Award 2017 (Brain and Behavior Research Foundation, US). The other authors do not report any conflicts of interest.

Acknowledgements

The authors of this report would like to thank the following organizations for their kind support: the Spanish Ministry of Health, Instituto de Salud Carlos III (PI070066; PI1100683; PI1500467; PI17/00741; PI1800696; PI18/00976), Ajut a la Recerca Pons Bartran (Fundació Clínic recerca Biomèdica), Fundació Marato TV3 (091630), the Catalonia Government (2017SGR881), PERIS (SLT006/17/00346), co-financed by ERDF Funds from the European Commission, “A way of making Europe” and CIBERSAM, The authors would also like to thank the families who participated in this study.