The sensorial perception of what is captured is what we know as “image” and consists of a static component and a dynamic process. This continuous process of images capture is essential in surgery. The image is crucial for the surgeon, who requires it for the diagnosis, for the therapeutic process and for postoperative follow-up. In minimally invasive surgery the sequence of images is essential and promotes the appearance of digital video. Digital video is the representation of moving images in the form of encoded digital data, unlike classic analog video, with continuous analog signals.
Beyond what we can consider the “real image” (what we see as part of the existing reality) other realities appear in these decades; the Virtual Reality and Augmented Reality. With these realities we refer in the medical ambitus to the creation or superposition, respectively, of a three-dimensional virtual environment to support healthcare and teaching or research processes. Today, these technologies have already begun to be integrated into various surgical specialties, with predictive surgical planning and intraoperative navigation us their main applications.
When using these digital environments, it is difficult to completely separate virtual reality from augmented reality, often being Mixed Reality. The current developments offer an environment that mixes the best aspects of both, unifying the simulation and requiring a single helmet or glasses to enjoy the sensorial experience. In this fusion of realities it will be possible to simultaneously create a virtual world from scratch to which we can add virtual elements from our real environment.
La percepción sensorial de lo captado es lo que conocemos como “imagen” y consta de un componente estático y un proceso dinámico. Este proceso continuo de captación de imágenes es indispensable en la cirugía. La imagen es vital para el cirujano, que la precisa para el diagnóstico, para el proceso terapéutico y para el seguimiento postoperatorio. En cirugía mínimamente invasiva la secuencia de imágenes es fundamental y nos lleva a la aparición del video digital. El video digital es la representación de imágenes en movimiento en forma de datos digitales codificados, a diferencia del clásico video analógico, con señales analógicas continuas.
Más allá de lo que podemos considerar la “imagen real”, lo que vemos como parte de la realidad existente, aparecen en estas décadas otras realidades; la Realidad Virtual y la Realidad Aumentada. En el ámbito médico nos referimos a la creación o superposición, respectivamente, de un entorno virtual tridimensional de apoyo a procesos asistenciales y docentes o de investigación. A día de hoy, estas tecnologías ya han comenzado a integrarse en diversas especialidades quirúrgicas, siendo la planificación predictiva y la navegación intraoperatoria sus principales aplicaciones.
Cuando se hace uso de estos entornos digitales es difícil separar de forma completa la realidad virtual de aumentada, siendo en muchas ocasiones una Realidad Mixta. En los actuales desarrollos avanzados se ofrece un entorno que mezcla los mejores aspectos de ambas, unificando la simulación y necesitando un único casco o gafas para poder disfrutar de la experiencia sensorial. En esta fusión se podrá de forma simultánea crear un mundo virtual desde cero al que agregar elementos virtuales en nuestro entorno real.
Images, defined as the sensory perception of what is captured in the environment by our organ of vision, are a static component and a fundamental, strategic and essential dynamic process in surgery. Surgeons require images to reach a diagnosis, perform the therapeutic process (fundamentally, surgery), and conduct the postoperative follow-up.
Gone are the days of the negatoscopes hanging on the walls of hospitalization units and session rooms in the surgical wards — those frosted glass light boxes on which X-rays or other negatives were placed for viewing transparencies. But that really was not very long ago; digital transformation is quite recent and has been implemented progressively, with increasing and unstoppable speed. Even today, developing rooms and negatoscopes are still found in many hospitals as an example of a not-too-distant past.
Why is this digital revolution so necessary? This evolution in image digitization presents several advantages, the most important being the capacity and ease of storage, as well as simple subsequent access to the images from a remote repository. No less important is the environmental factor associated with the generation of waste, to which digitization make a positive contribution: X-rays carry heavy metals (silver salts, etc.) that come off the plastic, which is highly toxic to the environment, as are the products used in developing X-rays.
According to the dictionary of the Royal Academy of the Spanish Language (RAE), the term “digital” refers to a device or system “that creates, represents, transports or stores information by combining bits.” A digital image is made up of a matrix of pixels (a × b × c), where a and b are the width and height, and c is the depth of color. This bit depth is the third dimension of the matrix and allows each pixel to have a different color. Digital images are technically defined as the 2-dimensional representation of an image, based on a binary numerical matrix. The images can be fixed or static and mobile or dynamic, depending on the resolution of the device.
There are 2 types of digital images: vector or bitmap. Vector images are made up of simple geometric figures, such as segments or polygons. Each of these figures is mathematically configured by position parameters (start and end-point coordinates), contour, thickness and color, which allow them to be transformed quickly and easily by modifying their size, without losing information or quality in the final image. These types of images are used in graphic design and in the creation of logos and illustrations (Fig. 1). Bitmap images are made up of colored pixels. These colored bits, imperceptible to the human eye, come together visually on our retina to define the image we finally see.
The resolution of the digital image is the characteristic that allows us to discern the details of the image with greater or lesser quality. The resolution is defined, although not exclusively, by the number of pixels per inch of the image. The more pixels per inch the image has, the higher the digital resolution. This is why the pixels may be seen, or not, when the image is enlarged depending on the image resolution (Fig. 2).
The color mode of the digital image defines the number of colors that the pixels can have in the formation of the image, progressively obtaining higher quality images in correlation with their coloring. The simplest are monochrome and grayscale. In the first, the image is only formed with pure white or black pixels. In grayscale mode, up to 256 shades of gray, black and white participate in the image configuration. There are other richer modes, such as the most commonly used “RGB” (Red – Green – Blue) mode, which provides great chromatic richness due to the great variety of colors in the image pixels, or the “CMYK” mode (Cyan – Magenta – Yellow – Black), which is used in printing and only displays the mixture of those colors.
The format of these digital images varies. Also known as an extension, it is the way the images are stored. The most frequently used are: the “JPG/JPEG” format (created in 1992 by the Joint Photographic Experts Group), which is used for graphic and web environments as it enables the use of a low file weight; the “GIF” format, which uses animated images and is used in icons and banners (advertisements that have images, animation and sound); the “PDF” format (Portable Document Format, from the Adobe company), which provides viewing of vector images and bitmaps and is therefore widely used in printing; and the “PNG” (Portable Network Graphics) format, for portable network graphics, which is widely used in web and graphic design as it produces gradients and transparencies.
In surgery, when we talk about images, we naturally think about image sequences, especially in our era of endoscopic surgery, where sequential images are fundamental, and this leads us to digital video. Digital video is the representation of moving images in the form of encoded digital data, unlike classic analog video, which represents images using continuous analog signals.
A digital video format is a set of rules that defines how we store, organize and compress visual information, and sometimes also audio information, in a file. The first attempt to create a digital video compression format dates back to 1984, and it was in 1986 when the Sony brand managed to record a component video signal in digital format in television studios, without being compressed. Unlike analog video, digital video can be copied without degrading its quality and can be stored on magnetic tapes and optical media, as well as on computer media, and can be transmitted by video streaming and viewed on computer or television devices.
Compression formats are necessary for compatibility between different devices and reproduction software, allowing images to be reproduced effectively. The digital video format has transformed the way we produce and reproduce multimedia content. Its configurations include image quality, the size of the file generated, and the ability to play it on different devices. The most common formats are “MP4” (evolution of MPEG – Moving Picture Experts Group), “AVI” (Audio Video Interleave, from Microsoft, 1992), “MOV” (from Apple, 1998) and “MKV” (Matroska video, derived from EBML – Extensible Binary Meta Language), etc. Digital video cameras capture images using interlaced or progressive scanning. The difference lies in the images captured per second, which is higher in interlaced video, despite which progressive-scan video recorders are preferred due to their similarity to cinema.
Videos of surgical procedures are a problem due to the size of the information generated. Problems arise when we have to save the sequence of images, process it and edit it. It is therefore necessary to compress this huge amount of bits through video compression, which is a data reduction process that involves reducing the size of the digital video. During compression, data that have temporally and spatially redundant information are eliminated, such as repeated frames within a succession of images that show no variation.
There are 2 types of compression. In “lossless” compression, there is no loss of information, and the encoded signal preserves the quality of the original signal. Then, there is “lossy” compression, in which the decoded signal is not true to the source signal because part of the information that is considered non-essential has been lost, having assumed that it would not be perceptible to the end user. Once this information is discarded during lossy compression, it is impossible to recover. Compression can be “intraframe”, which unifies information that is similar in the pixels of a frame by removing redundant information, and “interframe”, which eliminates repeated data between frames, showing only when differences are generated in the pixels. The different coding algorithms are called “codecs”, a name derived from the processes of compression and decompression.
The discourse on digital images and digital video leads us to a concept that is progressively penetrating our surgical environments: “digital surgery” or Surgery 4.0. Digital transformation has allowed us to move from open surgery (which we now call “Surgery 1.0”) to endoscopic surgery (2.0) in the 1980s and to robotic surgery (3.0) at the beginning of this century. The application of image digitization, artificial intelligence, computational data analysis and machine learning in the surgical process opens the era of Surgery 4.0, or digital surgery.
The emergence of the digital world in the surgical process has unfailingly improved the quality of the images viewed and stored as well as the process itself due to the ability to merge real and virtual images (simulation, virtual reality, 3D printing, etc.). It also aides decision making by providing a computational analysis of “big data”, that enormous amount of data impossible to be processed by conventional technologies and tools.1–9
This digital revolution and its needs have been included in the current government Recovery, Transformation and Resilience Plan, called Plan España Podemos, which establishes a digital transformation plan for 2021 to 2025.10 Within the plan, a PERTE (Strategic Project for Economic Recovery and Transformation) is specifically established for healthcare, which is called “Vanguard Health”. This plan identifies health as one of the main lines of the strategy for economic recovery and transformation, outlining innovative procedures to improve individualized prevention, diagnosis, treatment and rehabilitation. Its objective is to transform the healthcare sector so that science, innovation and digitalization are used in conjunction to face new health challenges.
Precision medicine occupies a prominent place within the plan, which entails treating patients in the context of their genomic data, clinical data, radiology studies, environmental exposure, lifestyle habits, socioeconomic factors and other relevant factors to have more precise and integrated patient information. In the PERTE for cutting-edge healthcare, certain strategic lines link with digital transformation, such as developing a digital national health system, with an integrated database to improve prevention, diagnosis, treatment, rehabilitation and research. Also, the PERTE proposes enhancing primary healthcare through digital transformation, with the application of advanced technology for all activities that involve relations with citizens and the management of resources anywhere in the country and in all public healthcare regions, emphasizing cybersecurity. This latter strategic line considers the impact of technological advances, such as information technologies and medical robotics, which facilitate the development of new diagnostic, therapeutic, and digital health techniques through innovation in health digitalization and synergy among emerging technologies. Advances made in robotics, artificial intelligence, sensors, 3D modeling and printing, bioprinting and immersive technologies allow us to respond to current and future needs by improving surgical care and surgical training.11
Virtual realityIn general terms, when we talk about virtual reality (VR) and augmented reality (AR) in the medical field, we are referring to either the creation or overlay, respectively, of a three-dimensional virtual environment to support healthcare, teaching or research processes. Today, these technologies have already begun to be integrated into various surgical specialties. Predictive planning and intraoperative navigation are their main applications, as they help accurately determine the anatomical region of interest and provide references for safer surgery. Likewise, they are increasingly used in teaching and clinical simulation in combination with haptic response, thereby providing quality training through the virtual creation of surgical procedures in an immersive and safe environment.12
As required by the Digital Health Strategy of the National Healthcare System of Spain, the implementation of VR and AR is presented as a key item to promote the transformation towards “5P Innovation and Healthcare” (5P Medicine: Personalized, Predictive, Preventive, Participatory and Populational). This incorporation translates into new, higher quality services adapted to patients, which facilitate the activities of clinical professionals. As a result, satisfaction increases, as well as performance and competence in prevention, diagnosis and treatment, all of which benefit the well-being and safety of the population. Thus, 5P Medicine is a paradigm that can only be developed through the application of technology to clinical practice and medical management.
However, there is much to be developed in terms of software and its collaboration, interaction and integration in clinical practice. In addition, surgical training in itself is not sufficient to acquire competence and develop skills; it is also essential to give feedback on performance. Although there has been increased interest in recent years in the development of feedback methodologies in virtual surgical training, their effectiveness in practice has rarely been studied. Furthermore, very few articles have analyzed the impact of these technologies on healthcare processes through prospective randomized studies, and this also holds true for validation metrics in simulators.
Based on what was previously stated, projects are being designed for the development, integration and validation of virtual and augmented reality systems adapted to the needs of the surgical ecosystem.13 These projects mainly focus their basic activities on surgical planning and personalized surgical training. However, the most interesting evolution is in intraoperative navigation and guidance thanks to the application of machine learning based on feedback from artificial intelligence (AI) algorithms.
Strictly speaking, virtual reality creates an environment of scenes and objects that look real but are generated using computer technology, thereby giving the user the sensation of being immersed in said environment. This immersive capability is created by using virtual reality glasses or helmets that situate the user within the simulated environment (Fig. 3). Although this experience seems recent, its origins seem to date back to the 1950s, when devices appeared to view 3D movies in an environment that associated olfactory sensations as well as movement.
Virtual reality is a technology that is currently experiencing great development and projection. Its penetration in different fields is multiplying, and it will be incorporated into the digital transformation of most companies. The improvement of virtual interfaces (connections between elements or components) and certain problems, such as the dizziness experienced by some users and improvements to the quality and definition of the images, are some of the current areas of development. Likewise, AI is being implemented to interconnect virtual environments between user communities thanks to the incorporation of 5 G.
Augmented realityAugmented reality (AR) is a technology in which virtual elements are superimposed on our vision of reality. Unlike virtual reality, where the real environment is hidden so that only the digital content is seen, in augmented reality the digital content is interspersed and overlayed on what we are viewing in our real environment.14 In virtual reality, we cannot see what is in front of us or to the sides, but with augmented reality we will continue to see all of this as well as whatever we want to add in the form of a digital hologram (Fig. 4).
Differences can also be found in the helmets or glasses for both types of reality. In virtual reality, they are opaque and have no glass to see through, so in effect they “blind” us to hide reality, showing us only the virtual environment. Conversely, augmented reality glasses are transparent to continue seeing the real environment.
The term “augmented reality” seems to have originated in 1992 when T Caudell thought of devising screens to guide assembly-line workers in the aeronautical industry to assemble different parts, thereby improving the interpretation of the assembly and installation instructions.15 Although this was the origin, the clearest example of the use of AR was the video game “Pokemon Go” in 2016, which inserted cartoon characters into real daily personal environments.
In the daily application of these types of digital environments, it is difficult to completely separate what is virtual reality from augmented reality, and there is often a mixture of both, or mixed reality. These developments offer an environment that mixes the best facets of both, unifying the experience so that only a single helmet or pair of glasses are needed. Despite being intertwined, they represent different revolutions. While virtual reality allows us to create a virtual world from scratch with whatever we want in a “fantasy” world, what augmented reality does is add virtual elements (additional information; e.g., graphics or images) to the real environment.
According to statements by Tim Cook, CEO of Apple, augmented reality encompasses more than virtual reality, because it gives us the possibility to be present, to communicate and, simultaneously, to enjoy other things visually. It will be the next revolution, as at the time the “Smartphone” was. Together with artificial intelligence and the “internet of things”, this revolution will be a paradigm shift in medicine and, particularly, in surgery.
Conflict of interestThe author declare that there is no conflict of interest.
AcknowledgementTo SIMIM3D platform of the Centro Tecnológico A Coruña for the clinical experience on augmented reality.
