Published online Apr 11, 2024.
https://doi.org/10.3348/kjr.2023.1306
Let’s imagine two artists engaged in a creative duel - one tirelessly crafting masterpieces, the other scrutinizing every brushstroke for the slightest hint of forgery. This encapsulates the essence of Generative Adversarial Networks (GANs), a powerful class of artificial intelligence (AI) algorithms revolutionizing the field of medical imaging.
What are GANs?
GANs are a class of machine learning algorithms introduced by Goodfellow et al. in 2014 [1]. At the core, GANs are designed to pit two neural networks opposite to the other in a high stakes game of one-upmanship, to engage in an adversarial and dynamic process. First is a generator which strives to create evermore convincing “fakes” or “forgeries.” Second is the discriminator which acts as a discerning critic, tasked with distinguishing the authentic from the artificial. This adversarial approach propels both networks to constantly improve: the generator by mimicking reality with ever-increasing precision, and the discriminator by detecting even the subtlest imperfections [2, 3, 4]. Figure 1 provides a flowchart to explain the iterative process of training GANs. Figure 2 discusses the workflow process in GANs. This schematic diagram illustrates the iterative process in which the generator and discriminator networks engage to generate increasingly realistic data samples.
Fig. 1
Generative Adversarial Network flowchart.
Fig. 2
GAN training workflow. GAN = Generative Adversarial Network, G = generator, D = discriminator
Examples of Uses of GANs in Radiology
GANs have several applications in radiology. One major challenge in medical imaging is the scarcity of large datasets for training AI algorithms. GANs can be advantageous in overcoming this challenge by generating synthetic medical images, mimicking real patient data with diverse representations of pathologies. This data augmentation significantly expands the training pool, leading to more robust and accurate AI models for disease detection. GANs not only create new data but can also enhance existing images by reducing noise and artifacts, improving visual clarity and diagnostic accuracy. Additionally, GANs can standardize images across different CT protocols and vendors, ensuring uniformity in data representation and improving the reliability of medical imaging data. This is particularly crucial for quantifying regional disease patterns in conditions like interstitial lung disease [5, 6].
The versatility of GANs extends beyond single modality enhancement. GANs can also seamlessly transform images from one modality to another. For instance, converting chest radiographs to CT-style images facilitates the detection of pulmonary nodules, demonstrating the transformative impact of GANs on cross-modal medical image analysis. Beyond clinical applications, GANs can also be used for medical education [7, 8]. By generating realistic anatomical structures and pathology scenarios, GANs create AI-driven educational cases and patient models. These virtual simulations serve as valuable tools for training healthcare professionals, allowing them to interact with diverse clinical cases and hone their diagnostic skills.
Several concrete examples for the utilization of GANs in radiology are:
Data Augmentation for AI Training
GANs can assist in the generation of synthetic images which may contain a diverse representation of the target disease mimicking real patient data beyond traditional data augmentation by geometric transformations (such as rotation, flipping, or zooming of existing images) and adjusting parameters (such as brightness, color, contrast, or introducing controlled levels of noise). For instance, early detection is pivotal for improving lung cancer survival rates. Thus, the integration of AI in lung cancer screening a promising avenue. However, acquiring large datasets of screening CT scans containing lung cancer nodules for training AI algorithms can be challenging. In this context, GANs involve a generator for meticulously creating synthetic CT images to simulate the complexity of authentic cancer nodules, whereas a discriminator evaluates and refines the synthetic data realism [9].
Anomaly Detection through Normal Image Generation
Consider a scenario in which a radiologist is examining a chest X-ray for signs of pneumonia. If the X-ray indicates abnormalities such as early signs of pneumonia, the radiologist employs a GAN trained on diverse datasets of normal chest X-rays. The GAN generator network then synthesizes a normal version of the input X-ray. The discriminator critically evaluates the generated image based on its knowledge database of normal X-rays, refines the process iteratively until the generated version convincingly aligns with the characteristics of a typical healthy chest X-ray. The radiologist then gains a precise reference point to identify and analyze the subtle abnormalities present in the original X-ray. GANs are promising in substantially improving anomaly detection, offering a more nuanced and accurate interpretation of abnormalities in medical imaging [10, 11].
GAN-Based Bone Suppression
In chest radiography, the challenge of detecting pulmonary nodules is often impeded by overlapping bones and by inherent limitations in standard imaging techniques. To address this, Bae et al. [12] conducted a study comparing the efficacy of two distinct approaches: bone subtraction imaging using a dual-energy technique (BSt-DE) and bone suppression imaging using deep learning (BSp-DL) based on a GAN. The study aimed to evaluate the impact of these techniques on radiologists’ performance in pulmonary nodule detection. The study utilized CT scans as a reference and subjected chest X-ray (CXR) images to different reading sessions. The results demonstrated that both BSt-DE and BSp-DL outperformed the standard CXR in nodule-wise performance. Further, BSp-DL demonstrated improved detection of nodules overlapping with bones or located in the upper/middle lung zone over standard CXR. The study emphasizes the valuable contribution of GAN-based approaches in improving diagnostic capabilities in chest radiology.
Accelerated MR Reconstruction
MRI scans can be time-consuming, especially when high-resolution images are required. It can increase motion artefacts or discomfort of patients. Consider a scenario where a patient undergoes a high-resolution MRI scan to assess intricate brain structures. GANs which are trained on a diverse dataset of fully sampled high-resolution MRI images can be utilized here. When the under-sampled MRI data from a new patient is uploaded on a GAN, it generates a synthetic, high-resolution image through its trained generator network. The discriminator network then evaluates this image against fully-sampled ground truth data, ensuring that the reconstructed image aligns with the characteristics of a true high-resolution MRI. Radiologists can analyze these accelerated MRI reconstructions, benefiting from both the speed of acquisition and the preservation of image quality. This not only enhances patient comfort during imaging procedures but also opens new possibilities for dynamic and real-time imaging [13, 14].
Noise Reduction in MR Imaging
A novel approach for image denoising involves the utilization of a residual encoder–decoder Wasserstein GAN. This method is designed to explore structural similarities between neighboring slices by adopting a 3D configuration as the fundamental processing unit. For example, for a patient undergoing a prostate MRI scan for the detection and characterization of potential abnormalities (such as prostate cancer), the 3D MRI scans are often affected by noise, which can impede the identification of small lesions or the delineation of prostate anatomy [15]. The GAN, in such a case, incorporates residual auto-encoders and deconvolution operations within its generator network, ensuring effective noise reduction while preserving the nuanced structural intricacies of the prostate. By applying a 3D Wasserstein GAN for denoising, radiologists gain access to clearer and more accurate images, enhancing their ability to identify subtle abnormalities such as early-stage tumors [16].
Computed Tomography Kernel Conversion
The choice of reconstruction kernel significantly impacts the diagnostic quality and visual representation of anatomical structures on CT. This problem was addressed by Choi et al. [17] by utilizing GANs. This approach transforms CT images reconstructed with one kernel into images with different reconstruction kernels. For example, in a scenario where a hospital upgrades its CT scanner, necessitating the conversion of historical images captured with an older sharp kernel to align with the characteristics of a newer, smoother kernel. The GAN is enlisted to perform this multi-domain image-to-image translation, with a distinctive emphasis on modifying the discriminator architecture. In this specific application, the discriminator is trained not only to differentiate between real and generated images but also to discern variations stemming from different CT kernel domains. This GAN-based approach streamlines the process of CT kernel conversion, offering a practical solution for healthcare institutions seeking to upgrade imaging systems while preserving the integrity and comparability of their historical medical data. The GAN-based CT kernel conversion also serves as a valuable tool in the realm of AI research, addressing challenges related to technical heterogeneity and fostering advancements in medical image analysis [18].
Radiation Dose Reduction
GANs offer a promising solution in reducing radiation dose while maintaining imaging quality. Instead of directly using standard-dose scans, which carry higher radiation risks, GANs can generate realistic, high-resolution images from significantly lower-dose scans. The generator takes a noisy, low-dose image as input and attempts to complete it, mimicking the detail and clarity of a standard-dose scan. Meanwhile, the discriminator acts as a critical judge, constantly refining the generator’s output to ensure the final image satisfactorily creates a realistic image. Thereby radiologists can rely on quick, low-dose scans, minimizing radiation exposure for patients. The technique can be beneficial for patients of all ages, ensuring that the advantages of reduced radiation exposure are extended universally [19].
Medical Education
GANs have proven to be invaluable tools in generating realistic anatomical structures and pathology scenarios, serving as a foundation for AI-driven educational cases and patient models.
Clinical Cases
Using GANs, intricate details and complexities of medical scenarios are replicated, providing learners with a dynamic and diverse training environment. For instance, a GAN can generate realistic medical images showcasing different stages of disease progression, allowing healthcare professionals to refine their diagnostic skills in a risk-free virtual setting [20].
Patient Models
In the context of patient care, the development of AI-generated patient models represents a significant advancement. These patient models generated using GANs, serve as valuable tools for predictive modelling, treatment planning, and simulating various clinical scenarios. For instance, a GAN can create patient models based on different age groups, comorbidities, and medical histories, offering a comprehensive training ground for healthcare practitioners to refine their decision-making skills and tailor treatments to specific patient profiles [21, 22].
Challenges in GAN Utilization
GANs, like any AI model, can inherit biases present in the training data. This can lead to discriminatory or incorrect image generation if the training data itself is skewed. Mitigating data bias requires careful data selection and curation practices. Robust and satisfactorily representative data needs to be used for training to ensure that feature analysis is done with adequate level of detail. Generating realistic synthetic data raises concerns about potential misuse. GANs could be misused to create fake medical images for insurance fraud or spread misinformation. Robust security measures and ethical guidelines are essential to address these vulnerabilities. Moreover, GANs remain a “black box,” making it difficult to understand how the output is achieved. At this stage, it is more accurate to say that GANs can work as supportive measures for radiologists on a case-by-case basis rather than as reliable tools [23].
Additionally, GANs are associated with several unique hurdles which need to be carefully considered. One is a phenomenon known as “mode collapse,” which is an inherent problem in GANs where the generator fails to produce diverse outputs and instead converges to a limited set of patterns. This limitation can impede the model ability to capture the full variability from the training data [24]. For example, for generation of synthetic CT images from MRIs, mode collapse could manifest as an overemphasis on a specific tissue type, neglecting the diversity present in the training dataset. Additionally, for generating chest X-rays, mode collapse may result in the generation of images which predominantly feature healthy lungs, thus, overlooking the pathological variations. Unlike traditional generative models, GANs lack straightforward metrics for assessing the output quality. This complexity makes it challenging to gauge the fidelity and diversity of the generated medical images objectively. To overcome this, alternative evaluation methods, such as the Turing test or visual scoring by domain experts like radiologists become essential.
Conflicts of Interest:The author has no potential conflicts of interest to disclose.
Funding Statement:None
References
-
Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative adversarial networks. arXiv [Preprint]. 2014 [accessed on December 28, 2023].Available at: https://doi.org/10.48550/ARXIV.1406.2661 .
-
-
Kasthurirathne SN, Dexter G, Grannis SJ. Generative adversarial networks for creating synthetic free-text medical data: a proposal for collaborative research and re-use of machine learning models. AMIA Jt Summits Transl Sci Proc 2021;2021:335–344.
-
-
Park C, Jeong HK, Henao R, Kheterpal M. Current landscape of generative adversarial networks for facial deidentification in dermatology: systematic review and evaluation. JMIR Dermatol 2022;5:e35497
-
-
Motamed S, Khalvati F. Multi-class generative adversarial networks: improving one-class classification of pneumonia using limited labeled data. Annu Int Conf IEEE Eng Med Biol Soc 2021;2021:3817–3822.
-
-
Choi C, Jeong J, Lee S, Lee SM, Kim N. CT kernel conversion using multi-domain image-to-image translation with generator-guided contrastive learning. In: Greenspan H, Madabhushi A, Mousavi P, Salcudean S, Duncan J, Syeda-Mahmood T, et al., editors. Medical image computing and computer assisted intervention. MICCAI 2023. Lecture notes in computer science, vol 14229. Cham: Springer; 2023. pp. 344-354.
-
-
Hwang HJ, Kim H, Seo JB, Ye JC, Oh G, Lee SM, et al. Generative adversarial network-based image conversion among different computed tomography protocols and vendors: effects on accuracy and variability in quantifying regional disease patterns of interstitial lung disease. Korean J Radiol 2023;24:807–820.
-
-
Vey BL, Gichoya JW, Prater A, Hawkins CM. The role of generative adversarial networks in radiation reduction and artifact correction in medical imaging. J Am Coll Radiol 2019;16(9):1273–1278.
-
MeSH Terms
Figures
1 / 2
