One of the most relevant aspects in assisted reproduction technology is the possibility of characterizing and identifying the most viable oocytes or embryos. In most cases, embryologists select them by visual examination and their evaluation is totally subjective. Recently, due to the rapid growth in the capacity to extract texture descriptors from a given image, a growing interest has been shown in the use of artificial intelligence methods for embryo or oocyte scoring/selection in IVF programmes. This work concentrates the efforts on the possible prediction of the quality of embryos and oocytes in order to improve the performance of assisted reproduction technology, starting from their images. The artificial intelligence system proposed in this work is based on a set of Levenberg-Marquardt neural networks trained using textural descriptors (the local binary patterns). The proposed system was tested on two data sets of 269 oocytes and 269 corresponding embryos from 104 women and compared with other machine learning methods already proposed in the past for similar classification problems. Although the results are only preliminary, they show an interesting classification performance. This technique may be of particular interest in those countries where legislation restricts embryo selection.
One of the most relevant aspects in assisted reproduction technology is the possibility of characterizing and identifying the most viable oocytes or embryos. In most cases, embryologists select them by visual examination and their evaluation is totally subjective. Recently, due to the rapid growth in our capacity to extract texture descriptors from a given image, a growing interest has been shown in the use of artificial intelligence methods for embryo or oocyte scoring/selection in IVF programmes. In this work, we concentrate our efforts on the possible prediction of the quality of embryos and oocytes in order to improve the performance of assisted reproduction technology, starting from their images. The artificial intelligence system proposed in this work is based on a set of Levenberg-Marquardt neural networks trained using textural descriptors (the ‘local binary patterns’). The proposed system is tested on two data sets, of 269 oocytes and 269 corresponding embryos from 104 women, and compared with other machine learning methods already proposed in the past for similar classification problems. Although the results are only preliminary, they showed an interesting classification performance. This technique may be of particular interest in those countries where legislation restricts embryo selection.
Introduction
The method routinely used for selecting the highest quality embryos to transfer is still based on morphological analysis. Many morphological embryo scoring systems have been proposed and reviewed for selecting embryos to transfer (Puissant et al., 1987, Giorgetti et al., 1995). The choice of the most suitable embryo to transfer can be achieved by extended culture of human embryos to the blastocyst stage (Gardner et al., 1998). However, approaches involving embryo selection cannot be implemented in countries with restrictive IVF legislation, for example Switzerland, Germany and Italy (Germond and Senn, 1999, Van der Ven et al., 2002, Benagiano and Gianaroli, 2004) since these techniques involve the loss of embryos cultured in vitro unless oocyte selection is implemented. Several pronuclear morphology scoring systems have been proposed to predict derived embryo quality and implantation or pregnancy success (Scott and Smith, 1998); also used has been a combination of pronuclei and embryo scores (De Placido et al., 2002). Morphological oocyte assessment is still controversial, although oocyte scoring systems have been proposed to help choose the best oocytes to be fertilized (Rienzi et al., 2008).
In most cases, embryologists select the oocytes/embryos by a non-invasive examination based on simple observation focused on morphology and dynamics of their development (third day of culture or blastocyst stage). The examination is usually performed visually and the evaluation is subjective considering the existence of many scoring systems especially for pronuclei or embryos. Therefore, the experience and expertise of the embryologist is of particular importance for the final success rate. In fact a consensus conference (Balaban et al., 2011; Alpha-ESHRE consensus grading scheme) allowed a standardized reporting of the minimum data set required for an accurate description of embryo development. This grading system established common criteria and terminology for grading oocytes, zygotes and embryos for routine use in IVF laboratories. It could be implemented with other tools that technology might introduce in the future.
Alternative methods, including polar body diagnosis (Verlinsky et al., 1990, Gianaroli et al., 2003), metabolomics (Patrizio et al., 2007) and polarization light microscopy (Oldenbourg, 1996, Montag et al., 2007) are at a preliminary stage or are often time consuming in routine IVF. Many studies have investigated the relationship between the timing of embryonic division and embryo quality (Hesters et al., 2008). In order to decrease the subjectivity of these observations, new promising methods, such as time-lapse monitoring systems of embryo development, are rapidly entering into laboratory practice (Cruz et al., 2011). Meseguer et al. (2011) reported a wide and successful trial for the use of morphokinetics as a predictor of embryo implantation. Lemmen et al. (2008) found a correlation between live birth and embryo development analysis with a time-lapse technique.
As for other medical applications, the use of artificial intelligence techniques may offer a possible solution to help embryologists in their work. Other examples of applying artificial intelligence methods to improve success rates of IVF programmes based on embryo or oocyte scoring/selection have been described. A pattern recognition algorithm has been presented to select embryos from images, which classifies the objects into a number of classes (Patrizi et al., 2004). Preliminary studies (Manna et al., 2004) correlating embryo quality with embryo imaging before transfer showed an improvement of manual selection. Morales et al., 2008a, Morales et al., 2008b presented a novel intelligent decision support system for IVF treatment based on a detailed analysis of human embryo morphology and clinical data of patients.
The present paper proposes the application of an advanced machine learning system based on a combination of classifiers for oocyte/embryo quality scoring and a state-of-the-art method for texture representation of images, the local binary pattern (LBP) descriptors (Ojala et al., 2002). As far as is known for the first time, an objective methodology is used in assisted reproduction technology to identify images of viable oocytes and embryos.
The proposed decision support system is evaluated on a data set of oocytes and their derived embryos. The aim at the moment is not to demonstrate that the system is able to select the perfect oocyte and embryo that will implant or to predict with great accuracy the chance of pregnancy in a routine clinical setting. The aim is to explore the possibility of using this new system in larger and more structured studies such as those with single-embryo transfer, including a prospective randomized trial for reaching full clinical relevance.
Section snippets
Study design
The experiments were carried out on two data sets: one includes of 269 photographs of oocytes and the other 269 photographs of the corresponding embryos usually at the 4-cell stage taken 40–50 h after intracytoplasmic sperm injection (ICSI) and immediately before transfer (day 2). The photographs were taken with an inverted microscope (Diaphot-300; Nikon) equipped with Hoffmann interference optics, stain-free objectives and a video camera (Digital SIGHT DS-F (i1); Nikon). The digital photographs
Results
The aim of this section is to validate the proposed approach with the available data set, according to two different testing protocols. The first testing protocol is the ‘leave-one-out-woman’, which uses only the embryos/oocyte where the label is certain: the testing set is composed of all the oocytes/embryos of a given woman (considering only those with a label that is certain). Therefore the results are obtained by considering the performance of 62 experiments (Table 1). The second testing
Discussion
This paper focuses on a new method for embryo and oocyte image classification based on a textural descriptor (local binary pattern) and on a random subspace ensemble of Levenberg–Marquardt neural networks. The results clearly outperform the existing approaches (Patrizi et al., 2004, Manna et al., 2004) and are encouraging, in particular considering that they have been obtained using a ‘small’ training set with very few positive samples (∼0.8 AUC considering the leave-one-out-woman testing
Can a deep learning image analysis model be developed to assess oocyte quality by predicting blastocyst development from images of denuded mature oocytes?
A deep learning model was developed utilizing 37,133 static oocyte images with associated laboratory outcomes from eight fertility clinics (six countries). A subset of data (n = 7807) was allocated to test model performance. External model validation was conducted to assess generalizability and robustness on new data (n = 12,357) from two fertility clinics (two countries). Performance was assessed by calculating area under the curve (AUC), balanced accuracy, specificity and sensitivity. Subgroup analyses were performed on the test dataset for age group, male factor and geographical location of the clinic. Model probabilities of the external dataset were converted to a 0–10 scoring scale to facilitate analysis of correlation with blastocyst development and quality.
The deep learning model demonstrated AUC of 0.64, balanced accuracy of 0.60, specificity of 0.55 and sensitivity of 0.65 on the test dataset. Subgroup analyses displayed the highest performance for age group 38–39 years (AUC 0.68), a negligible impact of male factor, and good model generalizability across geographical locations. Model performance was confirmed on external data: AUC of 0.63, balanced accuracy of 0.58, specificity of 0.57 and sensitivity of 0.59. Analysis of the scoring scale revealed that higher scoring oocytes correlated with higher likelihood of blastocyst development and good-quality blastocyst formation.
The deep learning model showed a favourable performance for the evaluation of oocytes in terms of competence to develop into a blastocyst, and when the predictions were converted into scores, they correlated with blastocyst quality. This represents a significant first step in oocyte evaluation for scientific and clinical applications.
The selection of embryos is a key for the success of in vitro fertilization (IVF). However, automatic quality assessment on human IVF embryos with optical microscope images is still challenging. In this study, we developed a clinical consensus-compliant deep learning approach, named Esava (Embryo Segmentation and Viability Assessment), to quantitatively evaluate the development of IVF embryos using optical microscope images. In total 551 optical microscope images of human IVF embryos of day-2 to day-3 were collected, preprocessed, and annotated. Using the Faster R-CNN model as baseline, our Esava model was constructed, refined, trained, and validated for precise and robust blastomere detection. A novel algorithm Crowd-NMS was proposed and employed in Esava to enhance the object detection and to precisely quantify the embryonic cells and their size uniformity. Additionally, an innovative GrabCut-based unsupervised module was integrated for the segmentation of blastomeres and embryos. Independently tested on 94 embryo images for blastomere detection, Esava obtained the high rates of 0.9940, 0.9121, and 0.9531 for precision, recall, and mAP respectively, and gained significant advances compared with previous computational methods. Intraclass correlation coefficients indicated the consistency between Esava and three experienced embryologists. Another test on 51 extra images demonstrated that Esava surpassed other tools significantly, achieving the highest average precision 0.9025. Moreover, it also accurately identified the borders of blastomeres with mIoU over 0.88 on the independent testing dataset. Esava is compliant with the Istanbul clinical consensus and compatible to senior embryologists. Taken together, Esava improves the accuracy and efficiency of embryonic development assessment with optical microscope images.
The integration of artificial intelligence (AI) and deep learning algorithms into medical care has been the focus of development over the last decade, particularly in the field of assisted reproductive technologies and in vitro fertilization (IVF). With embryo morphology the cornerstone of clinical decision making for IVF, the field of IVF is highly reliant on visual assessments that can be prone to error and subjectivity and be dependent on the level of training and expertise of the observing embryologist. Implementing AI algorithms into the IVF laboratory allows for reliable, objective, and timely assessments of both clinical parameters and microscopy images. This review discusses the ever-expanding applications of AI algorithms within the IVF embryology laboratory, aiming to discuss the many advances in multiple aspects of the IVF process. We will discuss how AI will improve various processes and procedures such as assessing oocyte quality, sperm selection, fertilization assessment, embryo assessment, ploidy prediction, embryo transfer selection, cell tracking, embryo witnessing, micromanipulation, and quality management. Overall, AI provides great potential and promise to improve not only clinical outcomes but also laboratory efficiency, a key focus because IVF clinical volume continues to increase nationwide.
Thus, an efficient computer-assisted method is urgently needed [6]. In earlier studies, machine learning [7,8] and time-lapse image analyses [5,9] have been conducted for the computer-assisted embryo grading. However, previous methods involved complex mathematic algorithms and required manual feature extraction and selection process, which were time-consuming and limited in clinical applications.
Visual inspection of embryo morphology is routinely used in embryo assessment and selection. However, due to the complexity of morphologies and large inter- and intra-observer variances among embryologists, manual evaluations remain to be subjective and time-consuming. Thus, we proposed a light-weighted morphology attention learning network (LWMA-Net) for automatic assistance on embryo grading. The LWMA-Net integrated a morphology attention module (MAM) to seek the informative features and their locations and a multiscale fusion module (MFM) to increase the features flowing in the model. The LWMA-Net was trained with a primary set of 3599 embryos from 2318 couples that were clinically enrolled between Sep. 2016 and Dec. 2018, and generated area under the receiver operating characteristic curves (AUCs) of 96.88% and 97.58% on 4- and 3-category gradings, respectively. An independent test set comprises 691 embryos from 321 couples between Jan. 2019 and Jan. 2021 were used to test the assisted fertility values on the embryo grading. Five experienced embryologists were invited to regrade the embryos in the independent set with and without the aid of the LWMA-Net three months apart. Embryologists aided by our LWMA-Net significantly improved their grading capabilities with average AUCs improved by 4.98%–5.32% on 4- and 3-category grading tasks, respectively, which suggests good potential of our LWMA-Net on assisted human reproduction.
In assisted reproductive technology (ART), embryos produced by in vitro fertilization (IVF) are graded according to their live birth potential, and high-grade embryos are preferentially transplanted. However, rates of live birth following clinical ART remain low worldwide. Grading is based on the embryo shape at a limited number of stages and does not consider the shape of embryos and intracellular structures, e.g., nuclei, at various stages important for normal embryogenesis. Here, we developed a Normalized Multi-View Attention Network (NVAN) that directly predicts live birth potential from the nuclear structure in live-cell fluorescence images of mouse embryos from zygote to across a wide range of stages. The input is morphological features of cell nuclei, which were extracted as multivariate time-series data by using the segmentation algorithm for mouse embryos. The classification accuracy of our method (83.87%) greatly exceeded that of existing machine-learning methods and that of visual inspection by embryo culture specialists. Our method also has a new attention mechanism that allows us to determine which values of multivariate time-series data, used to describe nuclear morphology, were the basis for the prediction. By visualizing the features that contributed most to the prediction of live birth potential, we found that the size and shape of the nucleus at the morula stage and at the time of cell division were important for live birth prediction. We anticipate that our method will help ART and developmental engineering as a new basic technology for IVF embryo selection.
The goal of an IVF cycle is a healthy live-born baby. Despite the many advances in the field of assisted reproductive technologies, accurately predicting the outcome of an IVF cycle has yet to be achieved. One reason for this is the method of selecting an embryo for transfer. Morphological assessment of embryos is the traditional method of evaluating embryo quality and selecting which embryo to transfer. However, this subjective method of assessing embryos leads to inter- and intra-observer variability, resulting in less than optimal IVF success rates. To overcome this, it is common practice to transfer more than one embryo, potentially resulting in high-risk multiple pregnancies. Although time-lapse incubators and preimplantation genetic testing for aneuploidy have been introduced to help increase the chances of live birth, the outcomes remain less than ideal. Utilization of artificial intelligence (AI) has become increasingly popular in the medical field and is increasingly being leveraged in the embryology laboratory to help improve IVF outcomes. Many studies have been published investigating the use of AI as an unbiased, automated approach to embryo assessment. This review summarizes recent AI advancements in the embryology laboratory.
Claudio Manna obtained his MD degree in 1980 and specialized in obstetrics and gynaecology in 1985. He is a university researcher at the University of Rome Tor Vergata and lectures on the assisted reproduction techniques within the gynaecological specialization course. He is responsible for two centres of assisted reproduction in Rome (Genesis and Biofertility). He has produced several multimedia tools for doctors and people. He is Secretary of the Italian branch of the Mediterranean Society for Human Reproductive Medicine and lecturer for the Ministry of Health programme at the school of fertility. His particular interest is the development of informatics in reproductive medicine.
