The role of AI and virtual environments in the Future Breast Unit

Abstract

Artificial intelligence (AI) has offered significant advances in medicine from its beginnings, with expert systems that complemented decision-making, to its current use, present in many applications where we are not fully aware. In this article, we are going to review and analyse different real uses of AI, which will enable us to correctly frame the vectors of present and future applications. From those already known in management, assistance in decision-making, radiomics, or pathomics to more recent ones such as virtual and augmented reality, remote monitoring, or the prediction of morphological results after surgery. Over time, the development of machine learning algorithms and the exponential increase of available health data will allow AI to provide more accurate and personalised diagnoses, development of new therapies and drugs, significantly reducing research and testing times. For this to be possible, greater communication and cooperation between healthcare professionals and the world of engineering will be necessary, starting in the educational field where there is a need to train professionals in engineering and biomedicine capable of tackling challenges with a complete vision, thus reducing the wall of technical understanding and deepening the development of useful applications.

Resumen

La Inteligencia Artificial (IA) ha ofrecido avances significativos en medicina, desde sus inicios, con sistemas expertos que complementaban la toma de decisión, hasta su uso actual, presente en muchas aplicaciones donde no somos del todo conscientes. En este artículo vamos a realizar un revisión y análisis de diferentes usos reales de la IA, que posibiliten enmarcar correctamente los vectores de aplicación presente y futuro. Desde los ya conocidos en gestión, asistencia en la toma de decisión, radiomics o pathomics a otros más recientes como la realidad virtual y realidad aumentada, la supervisión a distancia o la predicción de resultados morfológicos tras intervenciones. Con el tiempo, el desarrollo de algoritmos de aprendizaje automático y el aumento exponencial de datos de salud disponibles permitirá que la IA proporcione diagnósticos más precisos y personalizados, desarrollo de nuevas terapias y medicamentos, reduciendo significativamente los tiempos de investigación y ensayo. Para que esto sea posible será necesaria una mayor comunicación y cooperación entre los profesionales sanitarios y el mundo de la ingeniería, partiendo del campo educativo donde se necesita formar profesionales en ingeniería y biomedicina capaces de abordar los retos con una visión completa, para así reducir el muro de la comprensión técnica y ahondar en el desarrollo de aplicaciones útiles.

Introduction

Although in the last year, the avalanche of news about advances and controversies in the use of artificial intelligence (AI) has popularised and generated a great public debate about this technology, AI has been present in medicine since the dawn of this technology in the 70s of the last century with the first proposals such as MYCIN, an expert system developed at Stanford University to detect infectious blood diseases and advise on personalised treatments. Thus, this large amount of continuous and contradictory news has led in part to a certain disorientation about the current state of the art, which we have tried to clarify in this study through a review of real projects, the involvement of AI and its possible impact on the oncology discipline.

When we speak of AI, we think of supercomputers and computer programs that use an infinite amount of data, execute complex mathematical operations, and interpret algorithms at breakneck speed. These operations are unreachable for a human being but, paradoxically, their result must be supervised by the human being when the decision is incorrect.

Today's current AI models interact with us to the point of not distinguishing whether we are dealing with a person or a machine, a statement that is both exciting and shocking. These programming models use deep learning based on neural networks, connections similar to those of a child's brain, which means that today's computers can learn “the rules of the game” by watching others play, without needing to know the rules beforehand.

Among the many examples where we can apply AI in medicine, we find radiomics or pathomics, programmes that recognise images for the diagnosis of some pathologies, computer programmes that do not get tired, that do not need to sleep or eat to perform that function with the same precision from the first case to the last.

Decision-making assistance

Now let's imagine that instead of identifying patterns from a mammogram, a breast MRI or an anatomical pathological slide, we have an App capable of identifying a large number of variables within a clinical report to determine a therapeutic strategy for the treatment of a patient with breast carcinoma. The Fundação Champalimaud together with the Spanish company Quantum Babylon is developing a study called BreastAItegy. This is a pilot study that aims to use AI to help the medical team make decisions. The programme identifies variables from a text that may be written in different languages or contain abbreviations or acronyms that it recognises itself.

This comparative analysis between the decision taken by the multidisciplinary team and the computer programme will allow the algorithm to be improved by taking advantage of the progressive learning capacity of the AI model.

BreastAItegy not only promises to speed up the decision-making process but also has the potential to become a dynamic repository of oncological knowledge. Thanks to continuous contributions of clinicians, researchers, and breast reference units, the application constantly enriches its knowledge base with the latest scientific evidence, updated clinical guidelines, and diverse clinical experiences. This collaboration allows BreastAItegy to pool and standardise information, protocols and procedures from multiple sources, providing a more comprehensive and consensual view of breast cancer treatment.

In this way, just as today, we search for a flight for our holidays on our smartphone, in the immediate future, we will introduce a clinical history together with images, analyses, or any other medical information and we will instantly know different therapeutic strategies, the pros and cons of each one, and if we wish, we will know the prognosis and probabilities of having a complication or recurrence.¹⁸ It will be a personalised decision where family, geographic, or economic issues will be taken into account and that decision will be made as in the best reference centre in the world.

Moreover, their ability to process and analyse this large volume of clinical data could reveal patterns and associations that might otherwise go unnoticed by the human eye, potentially leading to new insights into breast cancer treatment.

We cannot forget that a characteristic of human beings is common sense, which is difficult to apply to a machine. It is true that some AI programmes are trained to interact with people with a certain empathy, but when a medical decision is to be made, a good dose of another intelligence, emotional intelligence, is also necessary, so that in some cases, there is a discrepancy between the sensible and prudent decision taken by the multidisciplinary medical team and the decision dictated by the rigour of an algorithm.

Does this mean then that medicine, as we understand it today, will undergo a radical change? In medicine, there are still tasks that are difficult to carry out on a computer, for example, the anamnesis or clinical examination. Nevertheless, devices are already being developed, such as the SenseGlove® from the Portuguese company Glooma, which uses a glove equipped with sensors to detect changes in breast tissue and alert the patient during breast self-examination.

Communication between doctor and patient and putting into practice what the Anglo-Saxons called ‘bedside manners’ is a role that can hardly be interpreted by a machine. What does seem clear is that in a short time, AI will soon be an indispensable tool in decision-making, diagnosis, or participation in clinical trials.

Research

Participating in any scientific study and discovering new forms of medical care is a stimulating activity, but it adds to the workload of day-to-day work. The time available to a physician for research within his or her working hours is limited or non-existent. Participating in a clinical study requires submitting a large amount of documentation to the ethics committee of each centre, involves explaining the study to the patient, entering forms, signing consents, and the clinician himself/herself filling in the different variables in computer platforms, all of which makes the task tedious and encourages many doctors to drop out or are not motivated to do research.

In this regard, AI will be able to facilitate methodology, become a fundamental tool for streamlining bureaucratic tasks, recruit data automatically from clinical reports and entering this information into the respective databases from which it will be able to perform statistical calculations and draw conclusions more quickly. All of this saves a lot of time and can serve as an incentive for physicians to participate more in research, an essential task for medicine to be in constant development.^1,5

Engineering/medicine binomial

Engineers and clinicians need each other to develop software to improve healthcare.

In the development of a surgical intervention on a metahuman model (digital representation of a person) for training a partial breast reconstruction with a chest wall perforator flap,⁶ presented at the Breast Cancer Metaverse congress in Lisbon in May 2023 (Fig. 5), we found that the language and communication between doctors and engineers is at times not very fluid, the programming or the terms used by one or the other professional, making this task difficult.

This problem has motivated, in universities, the incorporation of double degree courses in medicine/engineering and biomedical engineering where doctors learn programming language concepts, essential in the construction of algorithms and data manipulation to improve interaction and collaboration with engineers Fig. 1.

Fig. 1.

AVA XR Project: Mannequin simulating which the fusion of digital and physical anatomical structures of the abdominal wall. Created by the joint team of engineers and physicians belonging to the Digital Surgery Lab at Fundação Champalimaud (Lisbon) to develop new innovative technologies in breast cancer surgery.

BreastAItegy exemplifies interdisciplinary collaboration in breast oncology, fusing advanced AI technology with medical expertise. The application uses state-of-the-art natural language processing and image interpretation, but its true potential lies in the quality of the data and clinical context provided. Medical professionals play a crucial role, not only as end users, but as essential contributors to its development. They provide expertise to create comprehensive databases, validate the clinical accuracy of the results, identify areas for improvement, and ensure that recommendations are aligned with current guidelines. This synergy between AI and medical expertise ensures that BreastAItegy is constantly evolving, learning from both data and expert clinical judgement.

Examples of AI application

Here are some examples of how AI plays an essential role in clinical practice within a Breast Unit.

Breast XR®

Sometimes malignant breast lesions are not palpable, which requires placing a seed or a wire^11,12 to guide the surgeon to resect the lesion, the placement of these devices often causes discomfort in patients. Breast XR develops AI algorithms that process images obtained from breast MRI scans performed in different positions, allowing the construction of personalised 3D digital breast models. These models contain precise information regarding the location, shape, and size of the tumour. Thanks to AI, it is possible to visualise how the components of the breast deform with the different positions of the patient. In the operating room, surface cameras recognise the patient's position and the skin surface, allowing the 3D digital model to be aligned. At the same time, the surgeon has an augmented reality headset (Fig. 2) that allows him to visualise these 3D digital models of the breast to help plan and perform the surgery more accurately and less invasively.^7–9 The surgeon, while viewing the tumour through the skin, can visualise and manipulate with sterile gloved hands, reports, images, or holograms that float in front his or her eyes. Utilising augmented reality to visualise breast tumours will enhance the precision of surgeries. This advancement will increase the number of conservative surgeries, eliminate the need for wires or seeds, reduce costs, and spare patients from invasive and uncomfortable procedures.²

Fig. 2.

Breast surgeon performing surgery with Magic Leap augmented reality glasses.

AVA XR

The DIEP (Deep Inferior Epigastric Perforator) flap is the technique considered the current state of the art in breast reconstruction, which consists of replacing the resected breast tissue with the patient's own abdominal tissue. In order to plan this procedure, surgeons need a detailed characterisation of the vessels that are randomly distributed in the abdominal wall.²

The Digital Surgery Lab team has developed software (AVA) to create and validate a three-dimensional digital model of the abdominal wall including vascular anatomy, from images obtained by Computed Tomographic Angiography, fusing digital and physical anatomical structures of the lower abdominal wall.¹⁰ The anchoring and superimposition of this 3D digital model^9,13 is possible thanks by cameras that perform a surface scan of the patient (Microsoft Kinect®). This model is loaded into the augmented reality headset so that the surgeon can visualise the perforating vessels of the lower abdominal wall, facilitating the planning and dissection of these small vessels. In addition to seeing the trajectory of the vessels, the surgeon can see their relationship to other anatomical structures (e.g., the rectus abdominis muscle) and add data visualisation interfaces (Fig. 1).

Three-dimensional visualisation, especially when superimposed on real scenarios, requires the development of a huge amount of computation to determine the exact position of each pixel and its luminance properties. We are talking about an amount of information that can only be processed thanks to AI and the use of advanced heuristic algorithms.

Remote supervision

In May 2022, at the AECIMA Congress in Zaragoza, we witnessed a demonstration of how a surgical procedure can be remotely supervised thanks to 5G¹⁴ and augmented reality. The supervising surgeon (Fig. 3) accompanied a breast surgery from almost 800 km away as if he were looking over the shoulder of the surgeon performing the operation, all thanks to the fact that the surgeon was wearing augmented reality glasses connected to the remAid® software. The 5G supported by Telefónica and Altice enabled audiovisual transmission without appreciable latency, allowing real-time synchronisation of the indications. This remote support allows the surgeon performing a given technique to consult and be advised by a more experienced surgeon.^2,15

Fig. 3.

Remote digital anchoring with surgical incision design for performing breast-conserving surgery with partial breast reconstruction with lateral intercostal artery perforator flap (LICAP). View of the trainee surgeon through HoloLens2® presented at the Zaragoza Medical School.

This is undoubtedly another relevant example of a technology where, in an almost hidden way, AI is essential as it is responsible for crucial tasks to achieve the high speeds offered by 5G by managing in real time network optimisation, reducing latency, and managing traffic (Edge Computing) that enables data processing close to the end user.^3,4

Cinderella

It is difficult for a woman who is about to undergo breast cancer surgery to imagine what the aesthetic impact on her body will be after breast surgery, psychologically, emotionally, and sexually. Cinderella is a project funded by the European Commission through Horizon Europe in 2021, in which AI is applied to predict the aesthetic outcome of a patient undergoing breast cancer surgery. The patient has an App on her smartphone that uses an AI programme in which, based on photographs of the patient herself taken by a robot, it shows images of other patients with a similar profile who have already undergone the same surgery (Fig. 4). This study aims to demonstrate greater satisfaction in patients who used this tool, allowing them to avoid creating false expectations and helping them to participate in the proposed decision, especially when more than one surgical option is proposed.

Fig. 4.

Cinderella study.

Fig. 5.

Application of virtual environments. Left: Quantum Babylon room where the speakers' avatars presented their presentations at the Spanish and Portuguese Association of Breast Surgeons (AEPCIMA) congress held in Lisbon. Right: virtual simulation with metahumans for educational purposes of an intervention.

Fig. 6.

Patient participates in the Flow study, an experience to reduce stress levels before surgery through a combination of virtual reality and digital meditation.

Flow

Patients report high levels of anxiety and stress prior to surgery. This is an unsettling situation as it is an unfamiliar place and unsuitable for the patient's comfort. The aim is to attenuate this discomfort by inviting the patient to participate, just before entering the operating room, in an immersive experience with the use of virtual reality glasses (Fig. 6) in which the patient is transported to relaxing environments where nature and the sounds that we find there envelop her in this environment, reducing the level of stress and anxiety in many cases.^16,17 To achieve this purpose, we also added an 8-min digital meditation session (Tristan Gribbin; www.flow.is).

Another example of AI applied to the creation of virtual reality environments, predictive computations allow computers to process images with a high degree of realism and immersion and to determine in real time the spatial position of the user with respect to certain elements and sounds.

Conclusion

Breast oncology is on the threshold of an unprecedented transformation, driven by the convergence of AI, virtual reality, and augmented reality. From BreastAItegy to augmented reality surgical applications, these innovations promise to revolutionise the diagnosis, treatment, and monitoring of breast cancer in the medium term.

The different examples of practical experience provide insight into how they can be extrapolated to other fields. The synergy between clinical expertise and technological innovation is expanding the horizons of what is possible. However, the success of this revolution depends on seamless collaboration between healthcare professionals and engineers. This challenge begins by reshaping medical education and technical careers, fostering the creation of specialised degrees and interdisciplinary programmes that train a new generation capable of navigating between both fields and ending current communication problems.

Such knowledge would allow AI to maintain a balance between technological innovation and the fundamental principles of medicine, while respecting at all times the right to privacy and patients' rights. Empathy, clinical judgement, and effective communication will continue to be the core of quality care, but complemented by the analytical power of AI it will undoubtedly represent a significant, possibly exponential advance, the pace of which will be set by the ability to communicate and collaborate in a practical way between the medical environment and the technological disciplines involved.

Funding

The Cinderella study received funding of 5.4 million euros from the European Commission through Horizon Europe in 2021 (agreement no. 101057389).

Breast XR received 2.7 million euros through the Portuguese Resilience and Recovery Plan.

Ethical considerations

All consents required by law have been obtained for the publication of any personal data or images of patients, research subjects or other persons appearing in materials submitted. Copies or evidence that such written consents have been obtained.

Contributions from the authors

Concept and design of the article: Rogelio Andrés-Luna, Josep Salvador López.

Data acquisition: Rogelio Andrés-Luna; Pedro Gouveia; David Pinto; Carlos Mavioso; João Anacleto; Eduard Bonci; Marcio Debiasi.

Explanation of the functioning of the AI tool, BreastAItegy, as well as supervision and correct use of descriptions and AI terminology in the text: Josep Salvador López; Juan Pablo Castillo.

Editing and preparation of the manuscript: Rogelio Andrés-Luna; Josep Salvador López; Juan Pablo Castillo; Eduard Bonci; Jose Francisco Alonso.

Manuscript revision: Juan Pablo Castillo, Maria João Cardoso; Fatima Cardoso.

Principal investigators of the studies mentioned: Breast XR: Pedro Gouveia; AVA XR: David Pinto; Cinderella: Maria João Cardoso; Flow: David Pinto and Rogelio Andres-Luna; BreastAItegy: Rogelio Andres-Luna.