Scientometric analysis on the use of ChatGPT, artificial intelligence, or intelligent conversational agent in the role of medical training

Abstract

Introduction

In recent times, there has been a noticeable surge in the usage of artificial intelligence, including ChatGPT and other types, in the field of health sciences education. In this regard, an exploratory bibliometric study was carried out to examine the utilization of smart conversational agents, ChatGPT, and artificial intelligence bots in medical education.

Methods

A retrospective, observational, cross-sectional bibliometric analysis was employed to assess the scientific publications listed in Scopus. This study was conducted on March 11, 2023 in search for information in Scopus. A total of 220 relevant documents were identified that were available in the Scopus database during the period between 2017 and 2022. Elsevier's SciVal software was used. Subsequently, statistical tables and graphs were prepared for presentation in Bibliometrix software.

Results

Among the authors, Timothy W. Bickmore, from the United States, has the highest number of publications (10) and citations received (172), and an h-index of 45, suggesting a significant influence in the field of study. The subcategory with the highest academic output is Health Informatics with 133 publications, while Geriatrics and Gerontology has the least with only 3. Most of the analyzed publications (44.2%) originated from collaborations within the same country. Notably, the Swiss Federal Institute of Technology Zurich and Imperial College London stood out with 12 publications each that received over 200 citations indicating their significant impact on their respective fields. Despite having the highest number of academic publications (15), Brazil had a relatively low field-weighted citation impact (0.64) and received the lowest number of citations (81). A clustering analysis was performed on a sample of 10 concepts using 2 dimensions. The results indicated that all terms were part of the same cluster. Notably, the terms 'conversational agents', 'chatbots', 'conversational agent', and 'chatbot' were closely related.

Conclusions

It was found that the American Bickmore, Timothy W., led the top-10 researchers, and that the Health Informatics subject area was the most predominant. However, Brazil and Germany were the leading countries in terms of research output that was mainly published in high impact journals (Q1).

Resumen

Introducción

En los últimos tiempos, ha habido un notable aumento en el uso de la inteligencia artificial, incluyendo ChatGPT y otros tipos, en el campo de la educación en ciencias de la salud. En este sentido, se llevó a cabo un estudio bibliométrico exploratorio para examinar la utilización de agentes conversacionales inteligentes, ChatGPT y bots de inteligencia artificial en la educación médica.

Métodos

Se empleó un análisis bibliométrico retrospectivo, observacional y transversal para evaluar las publicaciones científicas listadas en Scopus. Este estudio se llevó a cabo, el 11 de marzo de 2023, se realizó una búsqueda de información en Scopus. Se identificaron un total de 220 documentos relevantes, disponibles en la base de datos Scopus durante el periodo comprendido entre 2017 y 2022. Se utilizó el software SciVal de Elsevier. Posteriormente, se elaboraron tablas y gráficos estadísticos para su presentación en el software Bibliometrix.

Resultados

Entre los autores, Timothy W. Bickmore, de Estados Unidos, tiene el mayor número de publicaciones (10) y citas recibidas (172), y un índice h de 45, lo que sugiere una influencia significativa en el campo de estudio. La subcategoría con mayor producción académica es Informática de la Salud, con 133 publicaciones, mientras que Geriatría y Gerontología es la que menos tiene, con sólo 3. La mayoría de las publicaciones analizadas (44,2%) proceden de colaboraciones dentro del mismo país. Destacan el Swiss Federal Institute of Technology Zurich y el Imperial College London, con 12 publicaciones cada uno que recibieron más de 200 citas, lo que indica su importante impacto en sus respectivos campos. A pesar de tener el mayor número de publicaciones académicas (15), Brasil tuvo un impacto de citas ponderado por campo relativamente bajo (0,64) y recibió el menor número de citas (81). Se realizó un análisis de agrupación en una muestra de 10 conceptos utilizando dos dimensiones. Los resultados indicaron que todos los términos formaban parte de este clúster. En particular, los términos “agentes conversacionales”, “chatbots”, “agente conversacional” y “chatbot” estaban estrechamente relacionados.

Conclusiones

Se comprobó que el estadounidense Bickmore, Timothy W. lideraba el Top-10 de investigadores, y que el área temática Informática Sanitaria era la más predominante. Sin embargo, Brasil y Alemania fueron los países que lideraron la producción investigadora publicada principalmente en revistas de alto impacto (Q1).

Introduction

Artificial intelligence (AI) is a field of research focused on the development and application of procedures that enable machines to reason and perform various functions. Problem-solving, object and word recognition, conclusions, and decisions about the state of the world.¹ AI is defined as the ability of machines to learn and show intelligence, which contrasts with human intelligence. In recent years, there has been a rapid development of AI that has significantly affected our personal and social lives. Advances in computing power, memory, storage, and large amounts of data have enabled computers to successfully perform increasingly complex learning tasks.²

ChatGPT is an AI program that simulates conversations using natural language algorithms to understand and respond appropriately. It undergoes constant updates that incorporate natural language processing and machine learning techniques to enhance its understanding and responsiveness. Apart from simulating conversations, it can perform various tasks, including writing short texts, conducting information searches, and solving problems. Although it can prove to be useful in academic settings, particularly for streamlining writing tasks, it is essential to regulate its usage due to the ethical concerns arising from its deployment in scientific writing. GPT models are a type of machine learning model used for natural language processing tasks. These models are pre-trained on large amounts of data to generate contextually relevant and semantically coherent language. GPT models are deep neural network architectures based on transformers.^3,4

Artificial intelligence bots and smart conversational agent have garnered significant public interest due to its impressive ability to compose stories and essays, solve programming problems, and provide concise answers to questions spanning from politics to medicine to technology.^5,6 However, there are ethical considerations that limit the use of chatbots in scientific writing.⁷ It can aid clinicians in swiftly comprehending the status of knowledge on a topic and generate an initial draft of a scientific article, along with suggested titles. Although the results are not always satisfactory, it can help save time.⁸

It has also been found that AI has undergone significant advances that have made it possible for machines to present and explain complex data more effectively and efficiently.⁹ Deep learning is rapidly emerging as a very promising tool, resulting in improved performance.¹⁰ The implementation of precision medicine through artificial intelligence poses significant challenges, such as information ownership rights, privacy, control of its dissemination, as well as potential misuse or abuse by users.¹¹

It has been argued that these problems can be solved by approaches that allow human experts to take on new roles as information specialists and generalists.^12,13

The aim of this study was to conduct an exploratory bibliometric analysis to gain insight into the use and development of artificial intelligence bots, ChatGPT, and smart conversational agents in medical education, and to identify trends, patterns, and characteristics of the relevant scientific literature.

Materials and methods

Study design

A cross-sectional, retrospective, observational, retrospective bibliometric analysis was conducted to evaluate scientific publications indexed in Scopus during the period from 2017 to 2022. This study used a bibliometric approach with the aim of analyzing and understanding the trends and characteristics of scientific production related to the research topic.

Search strategy

This study was conducted, on March 11, 2023, a search for information was conducted in Scopus, a reference source that hosts a wide variety of specialized publications in the field of health. To carry out the research, the MESH thesaurus was used, and a search strategy was defined using logical operators "AND" and "OR". The key aspects of the selected search strategy are described in detail in the following sections: TITLE-ABS (“ChatGPT” OR “ChatGPT’s” OR “bot ChatGPT” OR “Chat GPT” OR “automated conversational agent” OR “conversational agent” OR “embodied conversational agent” OR “online assistant” OR “smart conversational agent”) AND SUBJAREA(MEDI). A total of 220 relevant documents were identified, which were available in the Scopus database during the period between 2017 and 2022. Subsequently, we proceeded to export these documents to SciVal software for analysis through various bibliometric indicators with the aim of better understanding the evolution and trends of scientific production in the research topic.

Bibliometric indicators

Various metrics were used to analyze scientific production in the research topic, such as number of citations, frequency of publication, country of origin, institution and collaboration, journal quartile, authorship, h-index, CiteScore 2020, SCImago Journal Rank (SJR), Source Normalized Impact per Paper (SNIP), and Field Weighted Citation Impact (FWCI). The application of these measures provided a detailed and comprehensive picture of the scientific production in this field of study.

Data analysis

To carry out this study, Elsevier's SciVal software was used, which made it possible to extract data from scientific publications stored in .xls files (Microsoft Excel). Once the data were obtained, the categorical variables were analyzed using percentages and frequencies. Subsequently, statistical tables and graphs were prepared for presentation in Bibliometrix.

Results

The table shows the bibliometric information of 10 researchers from different countries: the United States, Switzerland, Singapore, the Netherlands, and France. Among the authors, Timothy W. Bickmore, from the United States, has the highest number of publications (10) and citations received (172), and an h-index of 45, suggesting a significant influence in the field of study. However, the impact of Bickmore's publications, as measured by the field-weighted impact index, is relatively moderate (1.72). On the other hand, Tobias Kowatsch from Switzerland has a high number of citations per publication (20) and an h-index of 24, despite having fewer publications (9) than Bickmore. In general, a wide variation in productivity and impact is observed among researchers from different countries (Table 1).

Table 1.

Top 10 authors by Scholarly output.

Name	Country	Scholarly output	Citations	Citations per publication	Field-weighted citation impact	h-index
Bickmore, Timothy W.	The United States	10	172	17.2	1.72	45
Kowatsch, Tobias	Switzerland	9	180	20	1.76	24
von Wangenheim, Florian	Switzerland	6	93	15.5	1.48	25
Denecke, Kerstin	Switzerland	5	37	7.4	4.98	17
Tudor Car, Lorainne	Singapore	5	117	23.4	1.04	35
Amith, Muhammad Tuan	The United States	5	37	7.4	0.63	9
Schachner, Theresa	Switzerland	5	91	18.2	1.68	3
Tao, Cui	The United States	5	37	7.4	0.63	26
van Velsen, Lex S.	Netherlands	4	54	13.5	0.89	20
Bibault, Jean Emmanuel	France	4	85	21.3	1.25	21

Another table presents data on academic output, citations, authors, citations per publication, and field-weighted citation impact for 10 institutions worldwide. The Swiss Federal Institute of Technology Zurich and Imperial College London stand out with 12 publications each having more than 200 citations, indicating high impact in their fields. In contrast, the University of Twente and Nanyang Technological University have lower citation rates of 121 and 132, respectively. Harvard University has a significantly higher field-weighted citation impact than the other institutions, suggesting its research is more influential (Table 2).

Table 2.

Top 10 best institutions.

Institution	Country	Scholarly output	Citations	Authors	Citations per publication	Field-weighted citation impact
Swiss Federal Institute of Technology Zurich	Switzerland	12	227	26	18.9	1.84
Imperial College London	United Kingdom	12	218	19	18.2	0.91
Northeastern University	The United States	11	172	8	15.6	1.57
CNRS	France	10	163	29	16.3	1.23
University of St. Gallen	Switzerland	9	180	6	20	1.76
University of Twente	Netherlands	9	121	16	13.4	0.87
Harvard University	The United States	9	417	11	46.3	3.05
Nanyang Technological University	Singapore	7	132	15	18.9	0.82
University of Zurich	Switzerland	7	47	20	6.7	1.11
Sorbonne Université	France	7	90	15	12.9	0.89

A third table ranks sources in the field of health technology and informatics according to Scopus metrics. Journal of Medical Internet Research and Frontiers in Public Health rank in the top quartile (Q1). Journal of Medical Internet Research has the highest number of publications (31), citations per publication (30.3), SNIP (2.318), and CiteScore (8.2). Frontiers in Digital Health has no quartile assigned and has the lowest performance metrics, while Pervasive Health has no SNIP, CiteScore, or SJR assigned (Table 3).

Table 3.

Publications by Scopus source.

Scopus source	Quartile	Publications	Citations per publication	(SNIP)	CiteScore 2021	(SJR)
Journal of Medical Internet Research	Q1	31	30.3	2.318	8.2	1.736
Studies in Health Technology and Informatics	Q4	18	7.8	0.333	1.4	0.277
JMIR Research Protocols	Q3	8	4.1	0.705	2.5	0.441
PervasiveHealth: Pervasive Computing Technologies for Healthcare	^a	7	6.6	^a	^a	^a
Frontiers in Digital Health	^a	6	1.8	^a	^a	^a
Frontiers in Public Health	Q1	5	2	1.949	4	1.298
JMIR mHealth and uHealth	Q1	5	5.6	1.675	8.2	1.362
JMIR Human Factors	Q2	5	2.4	1.053	3.6	0.651
JMIR Formative Research	Q3	5	5.8	0.844	1.8	0.49
BMC Medical Informatics and Decision Making	Q2	4	4.3	1.387	4.6	0.833

Source-Normalized Impact per Paper (SNIP).

SCImago Journal Rank (SJR).

a

Data not available.

The figure shows the CiteScore quartile of a publication in the years 2017–2022, which is divided into 4 categories: Q1, Q2, Q3, and Q4. In 2020, the publication was in the Q1 quartile with 26 publications, and in 2021 and 2022, it remained in the same quartile with 18 and 13 publications, respectively. In total, the publication has been in the Q1 quartile in 4 years, in the Q2 quartile in 2 years, in the Q3 quartile in 1 year, and in the Q4 quartile in 1 year. Overall, the publication has obtained a total score of 159 in the 6 years evaluated (Fig. 1).

Fig. 1.

Publications by CiteScore quartile.

Fig. 2 presents the research diagram of the scientific output on the use of AI or ChatGPT or conversational agent in medicine, showing the relationship between author keyword (middle), author (left), and country (right). The analysis revealed that there are some main keywords, such as “conversational agent”, “chatbot”, “artificial intelligence”, and “conversational agents”, which were mainly selected by authors Kowatsch T. and Tudor C. These authors come from the USA, Switzerland, and Australia.

Fig. 2.

Three field plot index-keyword (middle), author (left), and source (right).

The tree map analysis revealed that the words “conversational agent” and “chatbot” are the most used terms and accounted for 11% and 9%, respectively, of all terms used by the authors. On the other hand, the terms “artificial intelligence” and “conversational agents” with 7% and 6% mentions, respectively. In summary, the findings suggest that terms related to artificial intelligence, chatbots, and conversational agents were the most common (Fig. 3).

Fig. 3.

Tree Map.

Discussion

The use of artificial intelligence (AI) by medical professionals to diagnose diseases and conditions in patients can significantly reduce diagnostic time and improve the efficiency and effectiveness of the diagnostic process.¹⁴ In the medical field, the demand for physicians is overwhelming, which generates enormous pressure and possible misdiagnosis. Faced with this situation, it is important to seek alternatives to address this urgent situation. The development of AI-based healthcare applications has increased considerably in recent years.¹⁵

In the last decade, AI has gained great popularity. The success of AI has been made possible by increased computational power and data availability. This focus on machine learning is achieving unprecedented progress. The medical community is capitalizing on these advances by developing AI applications that utilize medical images, automate clinical procedures, and aid in clinical decision-making. These applications have enhanced the precision of diagnoses and treatments for various illnesses, ultimately improving patient quality of life. The expansion of computational power and data accessibility has significantly propelled AI research and implementation within the medical field.^16–20

AI, especially machine and deep learning, has demonstrated its potential to refine and automate medical practice. However, multidisciplinary collaboration is needed to integrate safely and effectively. This involves the participation of computer scientists, information technology, and medical experts to ensure that AI methods are robust and interpretable. It is critical to develop safe and effective AI-based solutions so that its benefits can be fully exploited in healthcare. Scientific and collaborative approaches are needed to drive the next generation of AI methods in medical practice.²¹

In addition, AI can improve early disease detection by analyzing data faster and more accurately than humans. This allows potential diseases to be identified at earlier stages, increasing the chances of effective treatment and faster recovery. AI is transforming the field of medicine and healthcare by providing tools and solutions that were not possible before. Its ability to analyze large amounts of data and learn from it can improve the diagnosis and treatment of diseases and improve the quality of life of patients and the clinical work of physicians.²²

Scientific production on artificial intelligence has grown exponentially in recent years, and a bibliometric analysis is necessary to better understand the trends and advances in this field, it has proven to be a valuable tool for natural language generation and text processing. However, it is important to keep in mind that its use poses ethical and social challenges, such as the possible generation of discriminatory or misleading content. Therefore, it is essential to continue researching and developing AI tools in a responsible manner that is aware of their implications.

The study has some limitations that deserve to be highlighted.²³ First, the data used come only from Scopus, which prevents an exhaustive exploration of all scientific research related to the topic. To obtain a more complete picture, it would be necessary to analyze other databases such as Embase, PubMed, or Web of Science. Finally, it should be noted that bibliometric software has some weaknesses in terms of accessibility, which may lead to under-representation of the available content. Finally, the years 2017–2022 were selected for this study as they mark a period of considerable progress and expansion in the realm of artificial intelligence and its implementation in medical education. This timeframe has seen a multitude of advances in the utilization of AI bots, ChatGPT, and intelligent conversational agents, rendering it a pertinent and enlightening interval to examine.

Within the limitations of this bibliometric study concluded that Timothy W. Bickmore, from the United States, has proven to be an influential author with 10 publications and 172 citations, reflecting his h-index of 45. In addition, collaborations within the same country have predominated, representing 44.2% of the publications analyzed. The Swiss Federal Institute of Technology Zurich and Imperial College London have demonstrated a significant impact in their respective fields. Finally, clustering analysis revealed that all terms were part of the same cluster, with “conversational agents”, “chatbots”, “conversational agent”, and “chatbot” closely related.

Aspectos éticos

No es necesario debido a que el estudio utilizó datos secundarios de acceso libre en Scopus.

Financiamiento

El estudio fue autofinanciado por los investigadores.

Conflictos de interés

Los autores declaran no presentar conflictos de interés.

Consentimiento informado

No aplica.