UMPM-Net: A Multimodal Deep Learning Framework for Colorectal Cancer Prognosis

Colorectal cancer (CRC) is one of the most common cancers worldwide, for which early prediction of patient outcomes is important for personalised treatment planning and to improve survival. However, current prediction systems are limited in the fact that they are based on single-modality data and cannot capture the complex interactions between clinical, imaging, and genomic factors. To overcome these shortcomings, this study presents a Unified Multimodal Predictive Modeling Network (UMPM-Net) which incorporates Electronic Health Records (EHRs), medical imaging biomarkers and genomic signatures in a deep learning model. The proposed approach uses U-Net for accurate tumor segmentation, EfficientNet for imaging feature extraction and DenseNet for classification and survival prediction. The multimodal features are then fused at the feature level as a way of obtaining complementary information across the modalities. Experiments were performed on a multi-institutional dataset of a CRC with 1200 distinct patients with paired imaging, EHR and genomic profiles. Quantitative results show better performance than five state-of-the-art models with Dice score 0.92 for segmentation, accuracy score 0.90, ROC-AUC score 0.95 for outcome classification and concordance index (C-index) score 0.80 for predicting survival. The ablation analysis further confirms that the fusion of imaging, clinical and genomic modalities has a significant impact on the predictive accuracy and clinical interpretability. In conclusion, the proposed UMPM-Net is a comprehensive, explainable and robust multimodal learning framework that is capable of accurate, interpretable, and clinically relevant prognostication for colorectal cancer.