2.1. Magnetic Resonance Imaging (MRI)

Using radio waves and magnetic fields, magnetic resonance imaging (MRI) is a non-invasive imaging technique that produces precise images of inside structures. Magnetic resonance imaging, or MRIs, are superior to CT scans in terms of quality. It works especially well for imaging soft tissues, such as the brain, muscles, and connective tissues. MRIs are also painless and safe because magnetic fields and radio waves do not have any negative effect on the patients’ are used to visualize the internal structure of the brain, spinal cord, bones, heart, blood vessels and many internal body parts. MRI scanners are sensitive to metal so patients with pacemakers cannot be subjected to an MRI scanner. Strong magnetic fields and radio waves are used in magnetic resonance imaging (MRI) to create images; ionizing radiation is not used, as is the case with other imaging techniques like CT and X-rays. During the procedure, the patient is placed inside a sizable, cylindrical magnet, where the body’s hydrogen atoms are momentarily realigned by the magnetic field. Then, by using radio waves to produce weak signals, cross-sectional images are produced and detected

2.2. Computed Tomography (CT)

CT scan called computerized tomography scan makes detailed images of the internal structures of the body utilizing X-rays and Computers that are not same as X-rays because it generates cross-sectional images of the body. CT scans are non-invasive so they are painless. Computed Tomography (CT), a medical imaging technology, uses X-ray measurements taken from different angles across the body to construct cross sectional images, or slices, of specific body sections. Compared to ordinary X-rays, these pictures offer more detailed information, enabling a thorough look of inside structures. During the procedure, the patient lies on a motorized table that glides into a CT scanner, which resembles a tunnel. Narrow X-ray beams are emitted by an X-ray tube inside the scanner, which revolves around the subject and penetrates the body. The X-rays are picked up by detectors on the other side, and a computer uses the data to create a precise three-dimensional picture of the interior structures. They are commonly used for imaging the chest, abdomen, and pelvis, and are particularly effective for detecting bone fractures, tumours, and internal bleeding.

Fig. 1. Different medical images ^[8].

2.3. Ultrasound

High frequency sound waves are used in ultrasounds, also known as sonograms, to create images of inside organs and structures. Because ultrasound doesn’t use ionizing radiation like X-rays or CT scans do, it’s a safer alternative–especially for expectant mothers and growing foetuses. In order to guarantee flawless transmission of the waves, a transducer is applied to the skin after a gel has been applied. The transducer then records the reverberating echoes after these waves bounce off interior structures. It is frequently used to assess soft tissues, blood flow, and cardiology in addition to obstetrics. Ultrasound produces real-time images of unborn babies to monitor their growth. It aids in tracking foetal development and identifying anomalies in obstetrics. It evaluates heart health and spots conditions such heart valve disorders in cardiology. Ultrasound is used in emergency care to promptly assess internal injuries and direct treatments such as fluid drainage or needle biopsies. They are also used in the diagnosis of different diseases.

2.4. Positron Emission Tomography (PET)

An internal organ image in three dimensions is obtained from a Positron Emission Tomography (PET) scan. It can be focused on a particular part of the body to visualize how well a body part is functioning. PET imaging uses radioactive tracers to visualize metabolic processes in the body. They are used to produce high resolution images of the brain and also predict the progress of cancer. PET scans are used for patients having already been diagnosed with cancer disease because PET scans can clearly demonstrate to how much extent cancer has extended or how well the patient has responded to chemotherapy. PET scan is also used for planning surgery of brain or heart etc.

As a feature of tumour cells, elevated metabolic activity is highlighted in oncology PET scans that help identify malignant tissues. PET scans are used in cardiology to evaluate heart tissue viability and blood flow, which helps in coronary artery disease diagnosis and treatment. PET scans are used in neurology to investigate brain activity and identify diseases including epilepsy, Alzheimer’s disease, and other neurological problems. DL models, when applied to PET images, are able to analyse the spatial distribution of radiotracer uptake, which reflects the underlying metabolic activity, and hence learn complicated patterns associated with various diseases.

2.5. X-ray

One of the most popular and extensively utilized diagnostic techniques in medicine is X-ray imaging, which uses electromagnetic radiation to create images of the body’s internal components. Different tissues in the body absorb X-rays to different degrees as they travel through it. Denser objects, like bones, absorb more X-rays and show white on radiographs, while softer tissues absorb less and appear in shades of Gray. One of the earliest and most widely used imaging modalities is X-ray imaging. Its main uses are to view bone structures and find infections or fractures. X-rays are also utilized by dentists and orthodontists to have a clear view of teeth. Tumor’s on bones can also be detected using X-rays used to guide surgeons during surgery. X-rays are essential for diagnosing diseases like pneumonia, heart difficulties, and digestive system disorders in addition to skeletal imaging. Digital X-ray technology advancements have improved radiation exposure and image quality, increasing patient safety and diagnostic precision. Even though there are very little hazards involved with radiation exposure, X-ray imaging is an essential tool in modern healthcare since its advantages for early disease identification and therapy greatly outweigh any possible disadvantages.

3.1. Convolutional Neural Networks (CNNs)

Given their capacity to automatically and adaptively learn the spatial hierarchies of features from input images, CNNs have emerged as the industry standard for image analysis applications. They are particularly effective for classification, detection, and segmentation tasks.

3.1.1 Classification

Image classification has been a busy research topic in the domains of computer vision and medical imaging. Because classification is regarded as an essential stage in computer-aided diagnosis (CAD), many research analysts have tried to use deep learning’s advantages for this task in medical imaging. Classification involves assigning a label to an image based on its contents. In MIA, CNNs have been used to classify various diseases, such as identifying different types of tumours in MRI scans. A Farhan , Shangming Yang ^[7] propose a novel Hybrid Deep Learning Algorithm (HDLA) framework for automatic lung disease classification from chest X-ray images. The model comprises of several phases, such as automatic feature extraction, detections, and pre-processing of chest X-ray images. shown in Fig. 2. They had mixed the advantages of CNN for feature drawing with the problem

Fig. 2. Basic block diagram of classification in ^[8].

3.1.3 Segmentation

In deep learning, Segmentation is the task of dividing the input image into a smaller part known as segments. Segmentation involves partitioning an image into meaningful regions, such as separating a tumor from surrounding tissues. Segmentation creates a pixel-wise mask for each object within the image. Segmentation can be broadly classified as Semantic Segmentation and Instance Segmentation. While distinct objects of the same class are given different masks in instance segmentation, all pixels belonging to a given class are represented by the same label in semantic segmentation medical image analysis these objects usually include different organs, pathologies, tissues or some other biological structure. Medical image analysis has made extensive use of image segmentation to divide up images from various imaging modalities, such as computed tomography (CT), ultrasound, MRI, PET, X-ray, and magnetic resonance imaging (MRI).

Fig. 3. Architecture for segmentation of medical images as proposed in ^[4].

In order to diagnose breast cancer, T.-C. Chiang et al. ^[3] suggested an architecture based on 3D CNNs and prioritized candidate aggregation utilizing the Automated Whole Breast Ultrasound (ABUS) modality. According to the experimental evaluation, the suggested method achieves a sensitivity of up to 95%, with an average of 3.62 false positives per patient. According to the authors, its design provides promising performance and is faster and more general than state of-the-art techniques. To illustrate U-Net’s ongoing potential for use in enhancing brain tumour segmentation performance, this study examines the various innovations and advancements in the U-Net design in addition to contemporary trends. This was proposed by Yousef et al. to segment brain tumor from CT images. Based on the U-Net architecture, they created a novel feature fusion strategy that effectively embeds high level features conveying semantic information with low-level features carrying picture data. They achieved this by applying an attention mechanism. When compared to other current approaches, the authors’ evaluation of this strategy on the BraTS 2020 dataset yielded good results.

3.2. Recurrent Neural Networks (RNNs)

Because RNNs are inherently good at preserving temporal relationships, they are especially well-suited to handle sequential data. RNNs have been used in the medical field to evaluate time-series data, including patient monitoring data, electronic health records (EHRs), and electrocardiograms (ECGs). Applications include tracking the course of diseases, managing chronic disorders, and forecasting patient outcomes. Because RNNs can handle sequential data, they are perfect for modelling temporal dynamics in medical data. RNNs are designed to manage sequential data, which qualifies them for analysis of time-series data in medical imaging, such as tracking disease progression over time. Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs) are popular RNN variants used in MIA.

Online medical pre-diagnosis assistance systems have received a lot of interest as telemedicine has grown. These systems are designed to offer first diagnosis suggestions based on medical data and information supplied by the patient. The paper by Zhou, X., Li, Y. Liang, W. ^[6] presents a novel CNN-RNN based intelligent recommendation system for online medical pre-diagnosis support. The proposed system combines the spatial feature extraction capabilities of CNNs with the sequential data processing strengths of RNNs to provide accurate pre-diagnostic recommendations.

The effectiveness of EEG data processing in the diagnosis of neurological diseases is greatly impacted by the removal of superfluous signals ^[7]. This paper proposed a novel dynamic filtering approach to identify and preprocess the most informative sub-bands related to a given neurological disorder. It does this by using a Recurrent Neural Network with a Gated Recurrent Unit (RNN-GRU) and Finite and Infinite Impulse Response (FIR and IIR) filters. Because it requires more hidden layers than traditional neural network topologies, this RNN with GRU combination gives a significantly stronger capacity to learn fitting and extract features from extremely complicated EEG data recordings, allowing for better harmonization of the diagnosis process Based on an offline diagnostic procedure using the Bonn and MIT datasets, the suggested diagnosis system achieves an average classification accuracy of 100percentage for epilepsy, and uses the KAU dataset to deliver an average accuracy of 99.5% for autism.

3.3. Generative Adversarial Networks (GANs)

Two neural networks, a discriminator and a generator, compete with one another to form GANs. They are used in MIA for data augmentation, image reconstruction, and generating realistic medical images for training other DL models. The networks that can produce lifelike synthetic images in any field are called Generative Adversarial Networks (GANs). By training the deep model with both the created and real data, the generated images can be utilized to address the problem of data scarcity in the medical field.

4.1. Impact of Image Quality

Input image quality has a major impact on DL models. Better, more accurately labelled images typically result in improved model performance. However, the performance of deep learning models may be affected by noisy, blurry, or low-resolution photos.

As an illustration: MRI and CT scans: When it comes to high-resolution MRI and CT scans, DL models frequently exhibit remarkable performance. They could, however, have trouble with photos that have artifacts or are of poorer quality. Ultrasound Imaging: DL models may have difficulties when dealing with ultrasound pictures, which can differ in terms of quality and consistency. When compared to CT or MRI scans, the performance could be less stable.

4.2. Effectiveness Across Disease Types

The kind of disease being studied can affect how well DL models work. Due to distinguishing imaging characteristics, certain diseases might be simpler to identify or categorize using DL, while others might pose greater difficulties: Lung Nodules: Because lung nodules in CT scan pictures are well-defined, DL models have demonstrated great accuracy in identifying and categorizing lung nodules. Brain Tumours: Although DL models have achieved great progress in brain tumour segmentation, the nature and location of the tumour within the brain can affect the model’s performance.

4.3. Modality-Specific Performance

The performance of DL models can be affected by several imaging modalities. Every modality has distinct qualities that can impact the effectiveness of the model: CT and MRI: Because of their high-resolution and detailed imaging capabilities, DL models frequently perform well in CT and MRI examinations. CNNs, for example, perform well when it comes to separating liver lesions from these modalities. PET Scans: Due to the decreased spatial resolution of PET images, DL models, while useful for PET image processing, may not always outperform CT or MRI.

4.4. Instances of Performance Variability

Research indicates that DL models can detect liver lesions with high accuracy from CT and MRI images, but their performance may be compromised by inferior pictures or non-standard imaging procedures. Diagnosis of Breast Cancer While DL techniques have shown great effectiveness with mammography, they may not work as well with ultrasound imaging due to differences in picture quality.

4.5. Dataset Source and Authenticity

In clinical settings where imaging instruments like MRI, CT, PET, or ultrasound are used in normal patient care, datasets from real-time systems are usually obtained directly from these settings. As a realistic depiction of what models can experience in real-world applications, these datasets mirror the imaging methods and protocols that are already used in hospitals and clinics.

The NIH Chest X-ray Dataset is a comprehensive collection of pictures used in routine practice that was gathered from clinical imaging procedures carried out at the NIH Clinical Centre. The Alzheimer’s Disease Neuroimaging Initiative (ADNI) Dataset is a collection of MRI and PET scans that have been gathered for Alzheimer’s research. The MRI pictures of brain tumours gathered from various universities are part of the BraTS Dataset (Brain Tumour Segmentation), which is used as a standard for segmentation algorithms.

Multiple radiologists have annotated CT images in the LIDC-IDRI Dataset (Lung Image Database Consortium and Image Database Resource Initiative), which offers a trustworthy ground truth for model training and assessment.

5.1. Liver Lesion Classification and Segmentation

Recent studies have demonstrated the value of CNNs in the classification and segmentation of liver lesions from CT and MRI images. These models have achieved high accuracy, sensitivity, and specificity, contributing to improved diagnosis and treatment planning. While ultrasonography is the most often used screening method, magnetic resonance imaging (MRI) and computed tomography (CT) are more effective for liver disease diagnosis and staging.. Yu, W et.al. ^[24] suggest a unique network that uses multi-phase CT scans to segregate liver lesions. We designed a cross-modal feature guiding module and a multi-scale feature fusion module to take advantage of the importance of reciprocal information from many phases.

5.2. Lung Nodule Detection and Classification

DL models have been extensively used for detecting and classifying lung nodules in CT scans. Techniques such as RCNN and U-Net have shown promising results in identifying malignant nodules, aiding in early diagnosis and reducing false positives. Awan, T., Khan, K. B ^[2] created Xray GAN is a synthetic X-ray image generator that provides a way to produce high-quality and varied X-ray images. In order to enhance thoracic disease identification, the study examines characteristics extracted from chest radiographs utilizing the innovative Xray GAN. Xray GAN is a unique technique that uses extracted picture features from a separate multiscale feature learning module as input labels for both the generator and discriminator. It exploits a form of the auxiliary classifier generative adversarial network (ACGAN). Furthermore, in order to guarantee model stability, our generative adversarial network (GAN) uses unique loss functions that keep weights constant via gradient adjustment. Two different kinds of datasets are used as inputs in our study: a self collected dataset and the publicly accessible NIH dataset. The suggested Xray GAN has produced encouraging results, especially in terms of better image production quality and improved accuracy.

5.3. Brain Tumor Classification and Detection

CNNs and RNNs have been applied to MRI images for classifying and detecting brain tumours. These models have enhanced the accuracy of tumour detection, allowing for precise treatment planning and monitoring. Automated brain tumour segmentation is essential for assisting with brain disease diagnosis and tracking the development of those conditions. In the field of brain tumor segmentation, magnetic resonance imaging (MRI) is currently a commonly used method that can produce pictures with several modalities. Using multi-modal pictures is essential to improving the efficacy of brain tumor segmentation in ^[12].

5.4. Breast Cancer Detection

Breast cancer has been identified through the use of DL methods in ultrasound and mammography pictures. Specifically, CNNs have demonstrated great accuracy in detecting malignant tumours, which helps with early detection and improves patient outcomes. Breast imaging is important for early detection and treatment to help patients with breast cancer have better outcomes. Deep learning has made significant strides in the last ten years in the analysis of breast cancer imaging, and it has enormous potential for deciphering the intricate context and wealth of data associated with various breast imaging modalities. In light of the swift advancements in deep learning technology and the growing gravity of breast cancer, it is imperative to synthesize previous achievements and pinpoint forthcoming obstacles that require attention.

6.1. Multi-Modal Imaging

Integrating information from several imaging modalities–such as CT, PET, and MRI–allows for a more thorough comprehension of the features of the disease. DL models can integrate these data sources to improve diagnostic accuracy and treatment planning. The global COVID pandemic and the introduction of novel strains have made it more critical than ever to promptly and effectively detect COVID-19 cases ^[9]. Nasir and et. al. introduces a novel dual-mode multi-modal method for identifying COVID-19 patients. This has been accomplished by combining the CT and X-ray image of the chest with the clinical notes that came with the scan. The dataset is extended by the use of data augmentation techniques. There have been five main kinds of image and text models used, including transfer learning. The binary cross entropy loss function and the Adam optimizer are used to build each of these models. Existing pre-trained models like VGG16, ResNet50, InceptionResNetV2, and MobileNetV2 are also used to test the multi-modal. The resulting multi-modal yields a 97.8% accuracy rate.

6.2. Radiomics and Genomics Integration

Integrating radiomics (quantitative features extracted from medical images) with genomic data provides a deeper insight into disease biology. Deep Learning models can analyse these combined data sets to predict disease outcomes and personalize treatment strategies.

7.1. Data Availability and Quality

High-quality, annotated datasets are essential for training DL models. However, obtaining such datasets can be challenging due to privacy concerns and the need for expert annotations. Efforts to create publicly available annotated datasets and the use of data augmentation techniques can help address this challenge.

Since medical photographs include comprehensive information about a patient’s health, they are by nature sensitive. Strict laws like the GDPR (General Data Protection Regulation) in Europe, the HIPAA (Health Insurance Portability and Accountability Act) in the United States, and other systems around the world make maintaining patient confidentiality a primary issue in the healthcare industry. These rules restrict the ease with which medical datasets can be shared for research and model training by requiring that patient data be anonymised and securely kept.

Hospitals and other medical facilities frequently implement stringent access controls on patient data as a result of these privacy concerns. This limits access to the extensive annotated datasets required for building reliable deep learning models. Furthermore, the process of developing a model is complicated by the possibility that shared data is inadequate or lacks the thorough annotations necessary for efficient training.

A high level of expertise is needed to annotate medical photographs. It is imperative for medical specialists such as radiologists and pathologists to meticulously label images, highlighting noteworthy findings like tumours, lesions, or other pathological characteristics. This procedure is expensive, labour-intensive, and time-consuming. It is also difficult to quickly build big, annotated datasets due to the restricted availability of these expertise.

Publicly accessible datasets are very useful, but they frequently contain restrictions. These datasets might not be comprehensive enough to cover all diseases, imaging modalities, or patient demographics, which would limit their use in various contexts. Furthermore, there may be differences in the annotation quality and the datasets may not be representative of the overall population, which could cause problems with the generalization of the model.

7.2. Model Interpretability

The interpretability and reliability of DL models in clinical contexts are questioned due to their? black-box? character. Developing techniques to provide insights into model decisions and ensuring that models are robust and reliable are important areas of ongoing research.

The instruments that clinicians and other healthcare workers use must be trusted, particularly when such tools are being used to make crucial decisions about diagnosis or treatment. Adoption of deep learning models in clinical contexts may be hampered by their incapacity to offer comprehensible explanations for their predictions. Clinicians could be reluctant to use these models if they lack interpretability, especially if the model’s forecast conflicts with their own clinical opinion. Since it is challenging to verify a model’s safety and effectiveness when the underlying decision-making process is opaque, deep learning models’ lack of interpretability may make this process more complex.

The construction and enhancement of deep learning models also depend on interpretability. Understanding the causes of a model’s errors is crucial for troubleshooting and improving the model when it makes an incorrect prediction. Because of the interpretability problem, researchers may find it difficult to pinpoint and solve the root causes of the problem, which could result in models that are less accurate and more prone to mistakes.

Researchers are increasingly using Explainable AI (XAI) strategies to overcome the interpretability difficulty. The goal of these techniques is to increase the transparency and comprehensibility of deep learning models’ decision-making process. Gradient-weighted Class Activation Mapping (Grad-CAM) is one technique that enables users to see the regions of a picture that have the most influence on the model’s prediction. Likewise, SHAP (SHapley Additive explanations) values offer a means of measuring the part played by every feature in a model’s choice.

By incorporating interpretability techniques into deep learning models, it is possible to create a collaborative environment where human knowledge and machine intelligence may be combined to help close the gap between AI systems and physicians. These techniques can help physicians better understand and trust the AI’s suggestions by offering insights into the model’s decision-making process, which will eventually improve patient outcomes.

There are real difficulties in integrating DL models into current healthcare processes. We talk about recent initiatives to provide decision support tools and user-friendly interfaces that promote smooth integration. Examples include the creation of software solutions that give clinicians real-time feedback and automated workflow integration.

7.3. Generalizability

DL models trained on specific datasets may not generalize well to other datasets or populations. Ensuring that models are robust and can generalize across different imaging modalities and patient populations is crucial for their clinical adoption.

DL models are frequently trained on datasets that are particular to medical disorders or certain imaging modalities (e.g., MRI, CT, PET). These models might work incredibly well on the training dataset, but they might not work so well on photos from other populations or sources. For instance, due to variations in imaging processes, equipment, and patient demographics, a model trained on CT scans from one institution might not perform as well on scans from another.

Training DL models on broad datasets encompassing a wide range of imaging modalities, patient demographics, and clinical contexts is one of the best approaches to enhance the models’ generalization. The model learns to handle novel and varied instances in real-world applications by being exposed to a wide range of data during training.

Deep learning models–in particular, deep neural networks–are sometimes viewed as "black boxes," which denotes that it is difficult to understand how they make decisions. Because healthcare practitioners could be hesitant to trust models they don’t completely comprehend, this lack of openness may impede clinical adoption to some extent. Deep learning models, particularly those with sophisticated architectures, can be computationally demanding, needing significant hardware resources for both training and inference. It may be challenging to widely implement these models in environments with limited resources because of this.

To allow models to adapt to new datasets with little retraining, researchers have also investigated transfer learning and multi-domain learning methodologies. Using large-scale, diversified datasets and sophisticated data augmentation techniques, for instance, can help create more resilient models that function well in a variety of clinical contexts.

Fig. 4. Multi-modal Architecture proposed in ^[9].