A psychiatric differential diagnosis task for medical students using collaborative learning and e-learning principles

Abstract

Objective

This study sought to apply collaborative learning and e-learning principles to differential diagnosis during clerkships.

Method

The educational intervention was conducted on sixth-year medical students undertaking a psychiatry clerkship in two university hospitals. It consisted of a collaborative differential diagnosis task based on psychiatric cases. It was carried out in groups of 9–10 students, either face-to-face or online, and was evaluated with a three-level rubric assessing group and individual performance. Student and supervisor satisfaction was collected through a survey and open-ended questions, respectively.

Results

A total of 114 sixth-year medical students completed the task: 57 face-to-face and 57 online. There were no significant differences between formats regarding the final score, the citation of external references, and the group performance items (all ps > 0.05). There were differences depending on the format for two individual performance items. Concretely, the quality of the interventions (p < .001) and the accuracy of the final diagnosis were higher in the face-to-face format (p = .011). A total of 104 students answered the satisfaction survey; students from the face-to-face format considered that the evaluation obtained during the clerkship had been more formative. The four supervisors praised its collaborative nature, considered the rubric useful and convenient, and warned about the risk of saturation during the activity.

Conclusions

A collaborative differential diagnosis task can enrich undergraduate education. The double rubric has the potential to assess the educational context of the group and provide formative evaluation for each student. Epistemological and practical challenges highlight the relevance of the supervisor in this learning task.

Resumen

Objetivo

El objetivo de este estudio fue aplicar los principios del aprendizaje colaborativo y el aprendizaje digital al diagnóstico diferencial durante las rotaciones.

Método

La intervención educacional se realizó en estudiantes médicos de sexto año involucrados en una rotación de psiquiatría en dos hospitales universitarios. Consistió en una tarea de diagnóstico diferencial colaborativo basado en casos psiquiátricos. Fue llevada a cabo en grupos de 9–10 estudiantes, bien presencialmente u online, y fue evaluada con una rúbrica de evaluación grupal de tres niveles y de desempeño individual. Se recopiló la satisfacción del estudiante y el supervisor mediante una encuesta y preguntas abiertas, respectivamente.

Resultados

114 estudiantes médicos de sexto año completaron la tarea: 57 por vía presencial, y 57 online. No existieron diferencias significativas entre los formatos en cuanto a la puntuación final, la cita de referencias externas y los ítems de desempeño grupal (todas las ps > 0,05). Sí existieron diferencias dependiendo del formato para dos ítems de desempeño individual. Concretamente, la cualidad de las intervenciones (p < 0,001) y la precisión del diagnóstico fueron más altas en el formato presencial (p = < 0.01). Un total de 104 estudiantes respondió la encuesta de satisfacción; los estudiantes del formato presencial consideraron que la evaluación obtenida durante la rotación había sido más formativa. Los cuatro supervisores evaluaron positivamente su naturaleza colaborativa, consideraron útil y conveniente la rúbrica, y advirtieron sobre el riesgo de saturación durante la actividad.

Conclusiones

La tarea de diagnóstico diferencial colaborativo puede enriquecer la educación universitaria. La doble rúbrica tiene el potencial de evaluar el contexto educacional del grupo, y aportar evaluación formativa para cada estudiante. Los retos epistemológicos y prácticos subrayan la relevancia del supervisor en esta tarea formativa.

Introduction

The differential diagnosis process is a fundamental aspect of medical practice,¹ as it requiressystematically considering and evaluating various potential diagnoses based on clinical findings. This process involves the integration of, among others, knowledge and clinical reasoning skills, which are essential for accurate diagnosis and effective patient management.²

Collaborative learning during undergraduate medical education aims students to engage in discussions, share their perspectives, challenge each other's assumptions, consider alternative possibilities, and arrive at a more accurate diagnosis. Besides specific knowledge, this cooperative setting also demands other skills (communication, teamwork, literature search, clinical reasoning, etc.) that relate to core, cross-cutting competencies that will be much needed during postgraduate training regardless of medical specialty.³ It also requires a series of preconditions that involve the learning situation (framework, grouping, etc.), the interaction (facilitation, peer communication, etc.), and the clinical context (familiarity, realism, etc.).⁴

Electronic learning or e-learning may provide access to medical knowledge, facilitate virtual simulations, promote interactive learning experiences, and enable time flexibility, among others⁵; however, critically evaluating and selecting reliable sources is essential.⁶ Educators and institutions need to leverage the potential of e-learning while ensuring effective utilization and equitable access.^7,8

This paper reports on an educational intervention for medical students during their psychiatry clerkship, consisting of a differential diagnosis task that incorporated collaborative learning and e-learning principles.

Methods

Population included

The participants were sixth-year students enrolled at the medical school of the University of Valencia (UV), Spain, in the 2023–2024 academic year, who had chosen to complete their sixth-year medical clerkships at Hospital Clínic Universitari de València or Hospital Universitari i Politècnic La Fe de València. To enhance anonymization, these institutions will henceforth be referred to as Hospital A and Hospital B, with these identifiers having been randomly assigned. The differential diagnosis task was part of a broader consolidated educational innovation project.⁹

A total of 156 students participated in the eligible medical clerkships (80 in Hospital A and 76 in Hospital B). Students were distributed into groups of about ten people (eight per hospital), which consecutively carried out their psychiatry clerkship. Each group was assigned a clinical vignette and a format (either face-to-face or online) for its completion. The first online and face-to-face groups at both hospitals served as pilot groups (n = 39); the other six groups at each hospital were considered fully eligible for the purpose of this study (117 students: 60 at Hospital A and 57 at Hospital B).

Development of the task in face-to-face and online formats

In the face-to-face format, students met physically to complete the task. The supervisor gave each student written instructions, the assigned clinical vignette, and the assessment rubric. The group members discussed the differential diagnosis for 30 min, being able to use electronic devices. Finally, each student had to individually and privately handwrite their own definitive diagnosis to the supervisor.

In the online format, students could connect at any time during two weeks of their psychiatry clerkship using Microsoft Teams.⁹ The platform provided them with the instructions, the assigned clinical vignette, and the rubric. Every student could make their contributions to the vignette through a whiteboard tool, thus having an asynchronous discussion. Finally, each student had to write individually and privately their own final diagnosis.

Design of clinical vignettes

A total of 16 clinical vignettes were designed by eight psychiatrists; a coordinating psychiatrist was in charge of reviewing each of the vignettes to ensure standards of quality and difficulty. The clinical cases included 384.42 words on average (median: 388 words; SD: 40.52); supervisors were provided with a list, for each vignette, that included the mental disorders that were considered relevant for their inclusion in the differential diagnosis and the one that was considered the correct definitive diagnosis.

Design of a mixed assessment rubric

We developed a rubric as an assessment tool to evaluate the task. Three psychiatrists and a senior lecturer of education participated on the design. The rubric included five items, with three levels of performance score for each item (0–2 points). The complete rubric is shown in Table 1.

Table 1.

Rubric for the collaborative task.

Group score (50%), equal for all members of the collaborative group
Item	0 points	1 point	2 points
G.1 List of possible diagnoses	Do not give a list of possible diagnoses or give a list of possible diagnoses that does not include the correct option	Give a partial list of possible diagnosespossible, including the correct option but omitting relevant alternatives	Give an exhaustive list of diagnoses possible, including the correct optionand multiple viable alternatives
G.2 Reasoning	Do not give arguments, or give mostly incorrect arguments, for or against options from the list of possible diagnoses	Give arguments for or against some but not all options from the list of possible diagnoses	Give arguments for or against each of the options on the list of possible diagnoses.
Individual score (50%), different for each member of the collaborative group
Item	0 points	1 point	2 points
I.1 Implication	Does not participate in the activity or does so in minimally	Participates in the activity, but to a lesser extent than the classmates or without integrating their own contributions with those of the group	Actively participates in the activity and integrates own contributions with those of his/her group mates.
I.2 Quality	Does not make contributions or makes incorrect contributions	Makes correct but poorly founded or argued contributions	Makes correct and well-founded or argued contributions
I.3 Final result	Gives a clearly incorrect diagnosis	Gives an incorrect but viable diagnosis	Gives a correct diagnosis

Two items assessed group performance as a whole: the list of potential diagnoses considered and the reasoning during the discussion; the scores for these items were common for all participants of a given group and made for 50% of the grade. Three items assessed the individual performance of each student: participation, quality of interventions, and final diagnosis; the scores for these items represented the other 50% of each student's grade.

The rubric was applied by four supervisors: two from Hospital A (one for each format) and two from Hospital B (one for each format).

Accessibility and quality of the sources used by the students

The supervisors recorded a series of variables in both the face-to-face and online formats in order to measure and compare the accessibility to sources by the students during the collaborative differential diagnosis process (equity principle) and the quality of the sources actually used by the students (reliability principle). The Diagnostic and Statistical Manual of Mental Disorders (DSM) and the International Classification of Diseases (ICD) are among the most relevant diagnostic manuals of psychiatry; thus, the supervisors recorded the following proxies: (a) number of students who cited DSM and ICD, books, scientific websites or articles; (b) number of times DSM or ICD were cited; and (c) number of times other books, scientific websites or articles were cited.

Satisfaction of students and supervisors

As part of a broader educational innovation project approved by the UV to improve psychiatry clerkships for sixth-year students, every student received a satisfaction survey via email once their clerkship had ended. Designed as a five-point Likert-type scale, students were asked their opinion on the evaluation of the knowledge and skills acquired during the clerkship, and whether they considered this assessment to have been formative.¹⁰

The four supervisors of the collaborative case task received a questionnaire with four open-ended questions plus a free comment space to collect their opinion on the task, the usefulness and usability of the rubric, the format that corresponded to them, and other aspects of their experience that they would like to expose.

Data analysis

IBM SPSS Statistics 28.0 for Windows (IBM Corp, Armonk, New York) was used for quantitative analysis. Significant differences between the groups in task performance results and student satisfaction were analyzed. All quantitative data showed a non-normal distribution, as determined through a Shapiro–Wilk test (all ps < 0.05), so non-parametric tests were employed. Specifically, the Mann–Whitney U test was performed to analyze differences between groups in the quantitative data.

Furthermore, we conducted a qualitative analysis of all the received responses to the supervisors' questionnaire. Analysis of each question started with data-driven codes, which were then grouped into more conceptual ones.

Results

A total of 114 sixth-year medical students completed the task (58 at Hospital A and 56 at Hospital B), which accounted for 97.4% of those eligible (96.7% at Hospital A and 98.2% at Hospital B). As for the formats, 57 students (50.0%) participated in the online version and 57 students (50.0%) in the face-to-face version. The full results are disclosed in Table 2.

Table 2.

Scores for each item on the online and the face-to-face formats.

Format	Hospital	Group items and scores				Individual items and scores
		Items	Scores			Items	Scores
			0	1	2		0	1	2
Online	A	G.1	0	0	3	I.1	0	0	29
		G.2	0	0	3	I.2	0	6	23
						I.3	0	12	17
	B	G.1	0	0	3	I.1	1	6	21
		G.2	0	0	3	I.2	0	14	14
						I.3	0	13	15
Face-to-face	A	G.1	0	0	3	I.1	0	1	28
		G.2	0	0	3	I.2	0	1	28
						I.3	0	0	29
	B	G.1	0	2	1	I.1	2	5	21
		G.2	0	2	1	I.2	0	3	25
						I.3	0	2	26

Overall and item-by-item results of the rubric

The final result for each student aggregated the group grades in the two group items (weighting them for a score in a range of 0–5) and the individual grades in the three individual items (weighting them for a score in a range of 0–5). The students who performed the task online had a median score of 9.17 (interquartile range or IQR: 0.84), while the students who performed the task face-to-face had a median score of 10 (IQR: 2.5). No significant differences were found between these two formats (U = 1572.00, Z = − 0.32, p = .752).

The two group items, which involved the list of possible diagnoses and collective reasoning by students, showed the same results: 10 groups fulfilled the educational objectives completely (2 points), and two groups reached the objectives partially (1 point). No differences were found between formats (p = .394) or hospitals (p = .394) in either item.

Regarding the first individual item (involvement), 99 students completely achieved the objective (2 points), 12 students partially achieved the objective (1 point) and 3 students had insufficient/absent performance (0 points). No differences were found depending on the format (p = .761), whereas significant differences were found between the hospitals (p < .001).

For the second individual item (quality of the interventions), 90 students reached the objective completely (2 points) and 24 students reached the objective partially (1 point). Significant differences were found depending on the format (p < .001) and the hospital (p < .001), with a better performance on the face-to-face format.

For the third individual item (final diagnosis), 87 students reached the objective completely (2 points) and 27 students reached the objective partially (1 point). Among students who had completed the task in a face-to-face format, 96.5% gave a completely correct diagnosis and 3.5% gave a partially correct diagnosis; among students who had completed the task in an online format, 56.1% gave a completely correct diagnosis and 43.9% gave a partially correct diagnosis. Differences were found depending on the format (p < .001), with a better performance on the face-to-face format; no differences were found depending on the hospital (p = .446).

Sources cited by the students

No significant differences were found between the face-to-face and online formats concerning the number of students citing external resources (online: 3.5 (4.5); face-to-face: 5 (2.5); U = 10.50, Z = − 1.22, p = .24), the number of times DSM or ICD were cited resources (online: 5 (12.2); face-to-face: 4 (3.3); U = 15.00, Z = − 0.49, p = .699), the number of times other resources were cited (online: 1 (2.5); face-to-face: 2 (3.3); U = 13.00, Z = − 0.846, p = .485), and the cites/students ratio (online: 0.9 (1.3); face-to-face: .6 (0.5); U = 16.00, Z = − 0.32, p = .818).

Student satisfaction survey

Of the 117 participants, 104 students answered the satisfaction survey about the psychiatry clerkship (58 at Hospital A and 46 at Hospital B), which accounted for a 88.9% response rate: 53 assigned to the online format (51%) and 51 assigned to the face-to-face format (49%) participated in the satisfaction survey.

There were no significant differences in students' beliefs about whether the task correctly assessed their acquired knowledge and abilities (online: 4 (1); face-to-face: 5 (1); U = 1268.50, Z = − 0.584, p = .559). Students participating in the face-to-face format agreed more strongly that the task was formative (online: 4 (1) Mean Rank (MR) = 42.25; face-to-face: 4(1), MR = 60.03; U = 986.00, Z = − 2.56, p = .011).

Qualitative comments by the supervisors

All four supervisors made an overall positive assessment of the task; they especially praised its collaborative nature, which enhanced teamwork or the establishment of a constructive discussion. Supervisors in the online format highlighted the potential of an asynchronous task, while supervisors in the face-to-face format pointed out the enriching effect of face-to-face interaction; none expressed difficulties regarding the viability of the activity.

Supervisors agreed that the rubric was useful, clear, convenient, and easy to use. The three scoring levels to evaluate performance in each of the items generated some discrepancies between those who would have preferred to have more scoring levels and those who considered that the three levels were sufficient for a task of these characteristics.

All supervisors highlighted the challenge that the risk of saturation poses for the activity in both formats: after an initial series of interventions, there were usually no new contributions (neither in quantity nor quality) and the discussion seemed to enter a dead end until it was extinguished.

Discussion

Regarding the task and the two formats

We present a collaborative case task for differential diagnosis purposes, in tune with current calls to incorporate innovative teaching strategies and promote clinical reasoning during undergraduate education.^11,12 These case-based, small-group learning initiatives are particularly suited for senior medical students who are moving toward clinical immersion.¹³

In the face-to-face format, interaction seems more natural and fluid,^14,15 closer to how a joint process of differential diagnosis occurs in a real service. When compared to their online counterparts, the students involved in the face-to-face format gave a correct final diagnosis at a significantly higher rate and considered the evaluation more formative, which benefits student learning.¹⁶ In the online format, it was not necessary to bring all participants together, affecting less day-to-day clinical practice. The Microsoft Teams ecosystem was easily accessible, in accordance with previous literature.¹⁷ In the online format, every intervention was on the whiteboard, which benefited summative assessment, susceptible to claims or contestation, more than formative assessment.¹⁸

Regarding the rubric as an assessment tool

As assessment in medical education is purpose-driven, we think that this three-tier rubric approach makes sense for the objectives of its evaluation purpose. It also fulfilled other criteria for good assessment, such as feasibility and acceptability.¹⁹ A more exhaustive ranking among those who did comply would have been well beyond the reach of the task and rubric, since it seems difficult and perhaps even unfair to assume that in the selected activity such subtle distinctions could be made.^20,21 The rubric poses a double potential: (a) group items could serve for students as a stimulus to get involved and for supervisors as proof that each group has functioned as an educational context; and (b) individual items could contribute as a source of formative and a summative evaluation. Furthermore, the collaborative case and its rubric could serve as a learning environment to detect those students with greater learning difficulties.²²

Potential areas for improvement

One potential risk of the task is that it may present the diagnostic process as a mere listing and checking of operational criteria. In the case of psychiatry, this reductionist approach has been widely questioned by prominent psychopathologists.²³ Based on international recommendations for psychiatry residents,²⁴ this problem could be partially fixed during undergraduate education during didactic lectures; during clinical clerkships, and during the collaborative case itself, with a more dynamic role on the part of the supervisor.

The risk of premature saturation may lead to different solutions. The requirement of a blinded intervention before having access to peers' contributions might boost early individual participation, but the eventual view of peers' initial contributions could either engage or disengage students;²⁵ thus, it may be insufficient on its own to enlarge and enrich the group discussion.²⁶ Providing incomplete, information-missing vignettes engages students in exploration of resources and self-directed information seeking.²⁷ This approach would better reflect real clinical practice: the diagnostic process involves an active search, and the clinician's work is closer to that of a detective than to that of a labeller.²⁸

Both epistemological issues and the risk of premature saturation seem to invite the supervisor to adopt a more active role as facilitator.²⁹ This context-sensitive role of the supervisor could increase the variability of the development of the discussion between groups, but in a fundamentally formative activity, the goal is to maximize in each group the opportunity to enrich the students' learning process.³⁰ This type of role on the part of the supervisor may be easier to adopt in the face-to-face format, due to its synchronous nature.

In terms of the strengths of this study, the use of an innovative methodology that can be easily integrated into clerkships is particularly noteworthy, addressing competencies that are often overlooked in undergraduate curricula. Furthermore, the comparison between an online and an in-person format, considering not only the scores obtained through the rubric but also the perceptions of both students and faculty, provides a deeper understanding of the strengths and limitations of each approach. Finally, the fact that the intervention was carried out in a real-world setting, with almost all the eligible students completing the task, enhances its applicability.

A series of limitations should be acknowledged regarding this study. First, the number of students included was rather modest, although this was in part due to the use of two pilot groups in order to optimize implementation. Second, the assessments were conducted by a single supervisor in each session, so it was not possible to calculate inter-rater reliability; however, the previous use of pilot groups was intended for supervisors to test themselves and improve their expertise with the rubric. Third, the responses to the satisfaction survey did not discriminate whether the respondents to the survey had actually carried out the collaborative case; however, as 97.4% of students completed the task, this does not seem to have had a significant impact. Fourth, a satisfaction survey poses the risk of social desirability and recall bias, although its anonymous nature and the fact that it was filled immediately after the clerkship may have mitigated them.

In conclusion, this collaborative task in differential diagnosis was well received by students and supervisors; both the face-to-face and online formats were feasible, but each has its own advantages and disadvantages that should be considered when planning a medical education activity. The double rubric approach has the potential to assess the educational context of the group and provide formative evaluation for each student; a more active role on the part of the supervisor could further enrich the learning impact of the activity.

Statement of ethics

The research was conducted ethically in accordance with the World Medical Association Declaration of Helsinki. The study was conducted as a part of a broader Consolidated Educational Innovation Project, approved by the Vice-Rectorate for Lifelong Learning, Educational Transformation and Employability of the University of Valencia in its call of 2023 (Project Code 2732977).

Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work, the authors used ChatGPT (OpenAI) for grammatical revision and language editing. After using this tool/service, the authors reviewed and edited the content as necessary and assume full responsibility for the content of the publication.

Funding sources

The authors have not received any funding for the present manuscript.

Acknowledgment

The authors would like to thank Professor Marta Pla i Castells for her training in the use of Microsoft Teams and her orientation to develop the online collaborative task. VB-M acknowledges support from the Instituto de Salud Carlos III (ISCIII), Ministry of Science, Innovation and Universities.