Clock drawing test with convolutional neural networks to discriminate mild cognitive impairment

Abstract

Background and objectives

The Clock Drawing Test (CDT) is a tool to assess cognitive function. Despite its usefulness, its interpretation remains challenging, leading to a low reliability. The main objective of this study was to determine the feasibility of using the CDT with convolutional neural networks (CNNs) as a screening tool for amnestic type of mild cognitive impairment (a-MCI).

Methods

A total of 177 CDT images were obtained from 103 healthy controls (HCs) and 74 patients with a-MCI. CNNs were trained to classify MCI based on the CDT images. To evaluate the performance of the CDT with CNNs, accuracy, sensitivity, specificity, precision, and f1-score were calculated. To compare discriminant power, the area under the curve of the CDT with CNNs and the Korean version of the Montreal Cognitive Assessment (MoCA-K) was calculated by the receiving operating characteristic curve analysis.

Results

The CDT with CNNs was more accurate in discriminating a-MCI (CDT with CNNs = 88.7%, MoCA-K = 81.8%). Furthermore, the CDT with CNNs could better discriminate a-MCI than the MoCA-K (AUC: CDT with CNNs = 0.886, MoCA-K = 0.848).

Conclusion

These results demonstrate the superiority of the CDT with CNNs to the MoCA-K for distinguishing a-MCI from HCs. The CDT with CNNs could be a surrogate for a conventional screening tool for a-MCI.

Introduction

Mild cognitive impairment (MCI) leads to cognitive decline that extends beyond typical age-related changes.^1,2 The interest in MCI has been intensified due to the potential for cognitive resilience through early-stage cognitive intervention in Alzheimer's disease.³ Consequently, a variety of screening tools targeting MCI have been developed, and their clinical value in clinics has been substantiated.^2,4

Clinics commonly employ screening tools to quickly assess cognitive function.⁵ While such tools are cost-effective compared to brain imaging techniques, the Mini-Mental State Examination (MMSE) and the Montreal Cognitive Assessment (MoCA), representative screening tools, typically require 5 to 10 min to administer, making it cumbersome for a large number of subjects.⁵ Furthermore, as both the MMSE and MoCA rely on language skills, they possess constraints that could be notably influenced by the subject's educational background.⁵

Conversely, while caution still needs to be required in using the Clock Drawing Test (CDT) in a multicultural context,⁶ it nonetheless offers advantages due to its simplicity and minimal influence on education and cultural background.⁵ This makes it an effective screening tool for MCI, supported by a high correlation with the MMSE,⁷ serving as a complementary screening tool.^8-10 The CDT evaluates visuospatial, executive, and memory by drawing a clock depicting 11:10.¹¹ Despite over 15 scoring systems, interpreting results remains challenging due to various factors like circle quality, number arrangement, and hand positioning.⁵

On the other hand, recent advances in deep learning, particularly a convolutional neural network (CNN), have transformed various industries, including medicine. A CNN demonstrates high accuracy in medical data classification and outperforms other techniques in image classification, excelling in identifying features imperceptible to humans.^12,13 Recent studies have employed CNNs to analyze clock drawing test outcomes to identify cognitive impairment.^14,15 However, limitations in prior studies stem from not including clinically diagnosed MCI according to the criteria within subjects with cognitive impairment,^14,15 thereby challenging the clinical utility of CNN-based CDT analysis. Furthermore, the clinical applicability of the CNN-based CDT remains unclear due to the absence of comparison with traditional screening tools.^14,15

Therefore, this study administered the CDT to subjects diagnosed with MCI. Subsequently, the CDT outcomes were analyzed using CNNs to confirm the feasibility of the CNN-based CDT for screening MCI. In addition, by comparing these results with the MoCA performance, this study aimed to substantiate the clinical efficacy of the CNN-based CDT.

Methods

Design

This study employed an observational study design using data from the author's previous study examining the feasibility of the newly developed screening system for MCI in South Korea. This study was approved by the Institutional Review Board of Yonsei University (1,041,849–201,611-BM-060–01). All subjects provided informed consent before participating in the present study according to the Declaration of Helsinki (2004).

Dataset

The original dataset utilized in the author's previous study encompassed 103 healthy controls (HCs) and 74 patients with MCI.¹ The study cohort comprised individuals aged 65 years and above, recruited from community settings and welfare centers within South Korea. The HC group consisted of 103 subjects who exhibited no memory-related complaints and demonstrated performance within the normal range on the standardized neuropsychological battery known as the Seoul Neuropsychological Screening Battery.¹⁶ In contrast, the MCI group comprised 74 individuals with amnestic MCI (a-MCI). The criteria for a-MCI followed the guidelines established in a prior study.¹⁷ Inclusion criteria were as follows: (a) presence of subjective memory complaints, (b) objective memory impairment compared to age- and education-matched HCs, confirmed by scores on the Seoul Verbal Learning Test falling below 1.5 standard deviations, (c) maintenance of intact global cognitive function, validated by the Korean version of the MMSE and (d) preservation of activities of daily living, assessed using the Seoul Instrumental Activities of Daily Living score of 7 or above.

Measurement

In the author's previous study, the CDT was implemented by trained occupational therapists. The subjects were instructed to sit comfortably in a noise-free, solitary setting without access to a clock. A standard A4-sized paper was placed on the desk in the center of each subject's body. Equipped with a pencil, each subject was directed to sketch a clock displaying the time as 11:10 onto the provided A4 sheet. To ensure comprehension of the instruction, a clarifying question was posed to the subjects. Following confirmation of their understanding, the CDT was administered. A maximum time allowance of 10 min was given for completing the test; however, if the task remained unfinished within this timeframe, the test was halted. Subsequent to the completion of the CDT, all subjects underwent the Korean version of the MoCA (MoCA-K).

Data preprocessing

A total of 177 CDT images were collected, encompassing 103 HCs and 74 patients with MCI. Given that the original CDT samples exhibited variation in figure size and orientation among subjects, they were not directly amenable to input a CNN model. Therefore, a preliminary preprocessing step was executed. The raw samples were automatically cropped into a square shape (Fig. 1) and resized uniformly to dimensions 224 × 224, irrespective of the original sample's size. Throughout this process, all samples were situated within a 600-dpi template and saved in the “.png” file format.

Fig. 1.

Sample image preprocess.

Subsequent to the preprocessing, the preprocessed samples for the training dataset underwent augmentation to augment the sample count for input into the CNN model. Augmentation involved applying a series of modifications based on a prior study (CDT-CNN). These manipulations included rotation of horizontal flip, and vertical flip (Fig. 1), resulting in a total of 531 CDT images. Conversely, for the test dataset, exclusively the original samples were utilized.

CNN architecture

Preprocessed CDT images were used as input data

A CNN architecture featuring four convolutional layers, each followed by a max pooling layer, and two fully connected layers, was employed. To mitigate overfitting, a dropout layer was introduced between the fully connected layers. Each layer incorporated convolutions employing 3 × 3 kernels with a stride of 1, and zero padding was applied to maintain spatial dimensions. The output layer consisted of two units given that dataset (HCs vs. MCI). The two fully connected layers were configured with 256 and 128 neurons, respectively. Activation functions utilized the rectified linear unit. The dropout rate in the dropout layer was set to 0.5 (Fig. 2). This CNN model configuration aligned with that of a prior study.¹⁵

Fig. 2.

The proposed convolutional neural network for classifying cognitive status.

Model training and validating

For training the CNN model, the experiment was conducted in Python using the Keras package with Tensorflow. Model training was iterated for up to 20 epochs to optimize accuracy, while validation was carried out concurrently. The batch size was set at 32. To enhance the validation process, early stopping was arbitrarily implemented based on the validation accuracy curve. The binary cross-entropy loss function was employed. For loss function optimization, the Adam optimizer was utilized with its default settings in Keras. A 5-fold cross-validation approach was employed. Subsequently, the trained CNN model was evaluated using the test subgroup to assess its performance. Standard metrics such as accuracy, precision, and F1-score were computed.

Extracted feature analysis

Gradient-weighted class activation mapping (Grad-CAM) was utilized to visualize the features on which the CNN model focuses within CDT images to classify. The output from the last convolutional layer was used for Grad-CAM. The obtain map was overdrawn on the original CDT image, to assess the extent to which the CNN model could be visually valid.

Results

Subject characteristics

Demographic and clinical characteristics encompassed sex, age, educational duration, and scores on the MoCA-K. No statistically significant differences were observed in sex ratio, age, and educational duration between HC and MCI groups (p’s > 0.05) (Table 1). However, a statistically significant difference was observed in the MoCA-K score between both groups, suggesting that patients with MCI exhibited lower cognitive function compared to HCs (p < 0.05) (Table 1).

Table 1.

General characteristics of participants (n = 177).

Characteristics		MCI (n = 74)	NC (n = 103)	χ^2 / t
Age, years (SD)		74.45 (6.51)	74.93 (6.96)	.471
Sex. N (%)	Male	33 (44.6)	45 (43.7)	.905
	Female	41 (55.4)	58 (56.3)
Education, years (SD)		6.14 (4.53)	5.83 (4.52)	−0.450
MoCA-K, scores (SD)		22.89 (2.17)	25.74 (2.10)	9.894^&#¿;&#¿;

MoCA-K: Korean version of the Montreal Cognitive Assessment.

&#¿;&#¿;

p < 0.001.

Classification performance

During the CNN analysis, an overfitting was not found. Overall classification performance was summarized in Table 2. The CDT with CNNs was more accurate in discriminating MCI from HCs than the MoCA-K (CDT with CNNs: 0.887 vs. MoCA-K: 0.818). Specifically, the CDT with CNNs achieved specificity of 0.892 whereas, the MoCA-K showed the highest sensitivity of 0.951. Furthermore, the CDT with CNNs achieved a higher AUC value than MoCA-K (CDT with CNNs: 0.886 vs. MoCA-K: 0.848) (Fig. 3). These findings suggested that the CDT with CNNs can better discriminate MCI than the MoCA-K.

Table 2.

Convolutional neural network classification model performance and the MoCA-K for detection of MCI.

Features	Accuracy	Sensitivity	Specificity	Precision	F1-score
CDT with CNNs	0.887	0.884	0.892	0.919	0.901
MoCA-K (scores)	0.818	0.951	0.630	0.784	0.859

CDT: Clock drawing test, CNNs: Convolutional Neural Networks, MoCA-K: Korean version of the Montreal Cognitive Assessment.

Fig. 3.

ROC curves of the CDT with CNNs and the MoCA-K.

Feature analysis

Grad-CAM showed that red or yellow color corresponds to the higher weighted region of interest (Fig. 4). As a result, clock letters and hands were considered crucial features to classify MCI from the CDT images.

Fig. 4.

Gradient-weighted class activation mapping examples. Yellow color corresponds to the higher-weighted regions of interest. Clock letters and hands were weighted to classify mild cognitive impairment from a clock drawing test image.

Discussion

This study aimed to investigate the potential of utilizing the CDT with CNNs as a screening tool for MCI. A total of 177 CDT images were collected, involving 103 HCs and 74 patients with MCI. Through augmentation techniques, 177 images were expanded to a dataset of 354, which was then employed to train CNNs. As a result, the CDT with CNNs achieved an accuracy of 88.7%. Moreover, the sensitivity and specificity were notable, reaching 88.4% and 89.2%, respectively.

The CDT is a tool involving the task of drawing a clock face, an item commonly encountered in daily life. The CDT holds the advantage of simplicity while offering insight into potential cognitive impairment.¹⁸ While the CDT is not a completely culturally unbiased tool, its lack of linguistic components makes it more cross-culturally applicable than other test tools,⁶ because it has no linguistic components, and is widely used to distinguish MCI.^5,19 The ability to arrange the components of a clock, as required by the CDT, necessitates cognitive processes associated with executive function,¹⁹ which is frequently impaired in patients with MCI. Specifically, regardless of the sub-types of MCI, patients with MCI, including those with pure a-MCI, exhibit notably greater impairments in comparison to HCs in terms of neuropsychological assessment of executive function.³ These findings support that the CDT with CNNs can sufficiently differentiate MCI, which is in line with previous studies.^14,15

It is noteworthy that the CDT with CNNs exhibited greater discriminatory capacity compared to the MoCA-K specifically for a-MCI. While the MoCA-K evaluates overall cognitive function across diverse cognitive domains, the CDT emphasizes visuospatial abilities and executive function. The CDT's ability to outperform the MoCA-K, despite focusing on fewer cognitive domains, can be attributed to the inherent structure of the MoCA-K. Notably, test items for executive function in the MoCA-K contribute just 5 out of 30 total points,^1,2 implying its potential inadequacy in capturing cognitive decline patterns seen in patients with MCI. This contributes to the superiority of the CDT with CNNs over the MoCA-K.

In the realm of medical advancements, machine learning is progressively employed for the monitoring and early diagnosis of cognitive impairment, aided by the advancement of medical imaging techniques.²⁰ CNNs hold a distinct advantage over traditional machine learning methods, attributed to their capacity for high accuracy facilitated by extensive parallel processing in image analysis.^21,22 This is reflected in the findings of the CDT with CNNs in the present study, demonstrating an impressive sensitivity of 88.4% and specificity of 89.2%. Notably, compared to prior studies utilizing the conventional CDT for screening MCI, which yielded sensitivity and specificity ranging from 56.7% to 90.0%, respectively,²³ the findings of this study hold considerable clinical significance. In conjunction with the robust performance of CNNs, earlier studies have extended CNN application to various neuropsychological assessments. For instance, a previous study leveraged CNNs to analyze images from the Rey-Osterrieth Complex Fig. Test, achieving an accuracy of 83% in classifying cognitive impairment.¹⁵ This underlines that CNNs offer potential beyond the simple interpretation of CDT results, extending their utility to the analysis of more intricate visual assessments.

Meanwhile, the high specificity of the CDT with CNNs signifies its proficiency in accurately discriminating individuals unaffected by a particular condition. This characteristic translates to a diminished occurrence of false positives, which, in turn, holds the potential to curtail unnecessary supplementary testing. By mitigating false positives, the efficiency of screening tools could be significantly enhanced, resulting in a more streamlined utilization of these tools.²⁴ In contrast, the MoCA-K demonstrated the highest sensitivity. As a consequence, the potent specificity of the CDT with CNNs and the robust sensitivity of the MoCA-K could offer an ideal pairing to distinguish MCI from HCs. This pairing aims to reduce instances of false positives and false negatives, respectively, enhancing the precision of differentiation.

On the other hand, this study visually confirmed the validity of feature extraction of the CDT with CNNs.¹⁵ As a result, it was found that the CDT with CNNs also considers clock letter arrangement and hand position as important features.⁵ These two features are commonly included in traditional scoring factors, which shows the validity of the CNN-based CDT analysis. These two features are also consistent with those suggested by a previous study analyzing deep neural networks-based CDT. While some previous studies focused on the discriminatory accuracy of CNN-based neuropsychological tests, this study identified these features to reveal the clinical utility of CNN-based CDT analysis.^10,15

Numerous studies have attempted to digitize conventional paper-based cognitive tests.^10,14,15,25 Computerized testing presents a notable advantage over traditional paper-based methods, as it can efficiently gather substantial datasets due to enhanced accessibility.^10,14,15,25 In this vein, the computerized CDT could be popularized and facilitate the collection of extensive data upon distribution within communities. However, effectively analyzing such voluminous data necessitates a deep learning approach, capable of delivering both high processing speed and accuracy.^21,22 From this perspective, the findings of this study, elucidating the clinical viability of deep learning-driven CNNs, hold paramount importance. The integration of deep learning technology into the computerized CDT enables periodic remote monitoring of the cognitive well-being of community residents, contributing to a reduction in the need for on-site assessments, and its high specificity can reduce false alarms of cognitive declines.²⁶ Furthermore, given the CDT with CNNs’ applicability beyond just MCI, its role as a screening tool extends to other clinical groups, including neurological conditions such as Parkinson's disease.²⁷

This study has some limitations. Firstly, the generalizability of the current findings is constrained due to the notably small sample size compared to prior studies. However, it is worth noting that this study concentrated on validating the applicability of the CDT with CNNs specifically for MCI. This differentiation was achieved by adhering to the criteria for MCI when selecting subjects, rather than simply distinguishing between the presence or absence of cognitive impairment. This distinction holds significant value. Secondly, given that the study's subjects were confined to individuals with a-MCI, the findings may not be universally applicable to other types of MCI. Nevertheless, considering that earlier studies also predominantly included a-MCI cases, as these individuals have a higher propensity towards developing Alzheimer's disease and display minimal cognitive bias of a-MCI,^1,2,28 the current findings do carry clinical relevance. Thirdly, it is important to acknowledge that the study did not endeavor to optimize the CDT with CNNs to enhance its accuracy. Therefore, there's a possibility that the accuracy could be further improved. As a result, a cautious interpretation of the study's outcomes is advised. However, it is important to emphasize that the primary goal of this study was to assess the feasibility of the CDT with CNNs for the purpose of MCI screening. Further studies involving larger datasets will be imperative to carry out optimization endeavors.

Conclusion

In the current study, the CDT with CNNs achieved similar performance as of previous studies in terms of accuracy but with subjects based on MCI criteria, suggesting its feasibility for detecting MCI. Moreover, the superiority of the CDT with CNNs to the MoCA-K was found. Since the CDT with CNNs possesses some advantages, such as the availability of considerable accumulated past data or greater accessibility than the original CDT, it could be a surrogate for a clinical-based screening tool.

Ethical consideration

The study was approved by the Institutional Review Board of Yonsei University, and all subjects signed an informed consent before the test.

Funding

This work was supported by the Soonchunhyang University Research Fund

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by Ministry of Education (no. 2021R1I1A3041487)