AI-Guided Anatomical Landmark and Abnormality Detection for Autonomous Endoscopy Examination
Md Shakhawat Hossain1,2, Munim Ahmed2, Md Sahilur Rahman2, Mahreen Tabassum3, Fariha Karim3, Md Aulad Hossain4, Razib Hayat Khan1, Razib Hayat Khan2, M. M. Mahbubul Syeed1,2 & Mohammad Faisal Uddin1,2 Show affiliations
- Department of Computer Science and Engineering, Independent University, Bangladesh, Dhaka, Bangladesh
- RIoT Research Center, Independent University, Bangladesh, Dhaka, Bangladesh
- Department of Computer Science and Engineering, American International University-Bangladesh, Dhaka, Bangladesh
- Department of Gastroenterology, Bangabandhu Sheikh Mujib Medical University Hospital, Dhaka, Bangladesh
Journal of Computer Science, Volume 20, Issue 8, pp. 858–871 · doi.org/10.3844/jcssp.2024.858.871
Abstract
Endoscopy is the routine medical procedure to observe tumors in the human Gastrointestinal (GI) tract by inserting an endoscope, a thin, flexible, tube-like instrument with a light source and camera. Traditionally, an endoscopist performs the endoscopy, orients the endoscope within these structures and navigates this through the help of familiar anatomical landmarks to reach the abnormalities and mark them. Identifying landmarks and abnormalities is critical for the maneuver and the success of endoscopy, which is related to the patient’s comfort, injury and accurate diagnosis. The manual naked-eye-observation maneuver and examination are highly challenging, take a long time and often cause discomfort to the patients and the endoscopists. As a result, several AI-based landmark detection methods have been proposed recently to facilitate autonomous endoscopy examination. However, these methods lack accuracy and consider only limited landmarks. This study presents a Data-efficient image Transformer (DeiT)-based method to detect anatomical landmarks and anomalies for autonomous endoscopy. The proposed method detected 23 landmarks and anomalies from the entire GI tract with 99% accuracy and precision, outperforming the state-of-the-art (91%). Moreover, this method took only 0.045 sec to identify a landmark. The phi coefficient (0.997) indicated a strong positive association between the proposed method and clinical ground truth. Strong association, high accuracy and rapid speed ensured the reliability of the proposed method for autonomous endoscopy examination.
References
- Aabakken, L., Barkun, A. N., Cotton, P. B., Fedorov, E., Fujino, M. A., Ivanova, E., Kudo, S., Kuznetzov, K., De Lange, T., Matsuda, K., Moine, O., Rembacken, B., Rey, J., Romagnuolo, J., Rösch, T., Sawhney, M., Yao, K., & Waye, J. D. (2014). Standardized endoscopic reporting. Journal of Gastroenterology and Hepatology, 29(2), 234–240. https://doi.org/10.1111/jgh.12489CrossrefGoogle Scholar
- Aliyi, S., Dese, K., & Raj, H. (2023). Detection of gastrointestinal tract disorders using deep learning methods from colonoscopy images and videos. Scientific African, 20, e01628. https://doi.org/10.1016/j.sciaf.2023.e01628CrossrefGoogle Scholar
- Ayyoubi Nezhad, S., Khatibi, T., & Sohrabi, M. (2022). Proposing Novel Data Analytics Method for Anatomical Landmark Identification from Endoscopic Video Frames. Journal of Healthcare Engineering, 2022(1), 8151177. https://doi.org/10.1155/2022/8151177CrossrefGoogle Scholar
- Borgli, H., Thambawita, V., Smedsrud, P. H., Hicks, S., Jha, D., Eskeland, S. L., Randel, K. R., Pogorelov, K., Lux, M., Nguyen, D. T. D., Johansen, D., Griwodz, C., Stensland, H. K., Garcia-Ceja, E., Schmidt, P. T., Hammer, H. L., Riegler, M. A., Halvorsen, P., & De Lange, T. (2020). HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific Data, 7(1), 283. https://doi.org/10.1038/s41597-020-00622-yCrossrefGoogle Scholar
- Bour, A., Castillo-Olea, C., Garcia-Zapirain, B., & Zahia, S. (2019). Automatic colon polyp classification using Convolutional Neural Network: A Case Study at Basque Country. 2019 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), 1–5. https://doi.org/10.1109/isspit47144.2019.9001816CrossrefGoogle Scholar
- Che, K., Ye, C., Yao, Y., Ma, N., Zhang, R., Wang, J., & Meng, M. Q. H. (2021). Deep learning-based biological anatomical landmark detection in colonoscopy videos. ArXiv, 2108.02948. https://doi.org/10.48550/arXiv.2108.02948CrossrefGoogle Scholar
- Chicco, D., & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21, 1–13. https://doi.org/10.1186/s12864-019-6413-7CrossrefGoogle Scholar
- Del Moral, P., Nowaczyk, S., & Pashami, S. (2022). Why Is Multiclass Classification Hard? IEEE Access, 10, 80448–80462. https://doi.org/10.1109/access.2022.3192514CrossrefGoogle Scholar
- Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255. https://doi.org/10.1109/cvprw.2009.5206848CrossrefGoogle Scholar
- Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ArXiv, 2010.11929. https://doi.org/10.48550/arXiv.2010.11929CrossrefGoogle Scholar
- Ferlay, J., Soerjomataram, I., Dikshit, R., Eser, S., Mathers, C., Rebelo, M., Parkin, D. M., Forman, D., & Bray, F. (2015). Cancer incidence and mortality worldwide: Sources, methods and major patterns in GLOBOCAN 2012. International Journal of Cancer, 136(5), E359–E386. https://doi.org/10.1002/ijc.29210CrossrefGoogle Scholar
- He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778. https://doi.org/10.1109/cvpr.2016.90CrossrefGoogle Scholar
- Hirasawa, T., Aoyama, K., Tanimoto, T., Ishihara, S., Shichijo, S., Ozawa, T., Ohnishi, T., Fujishiro, M., Matsuo, K., Fujisaki, J., & Tada, T. (2018). Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images. Gastric Cancer, 21, 653–660. https://doi.org/10.1007/s10120-018-0793-2CrossrefGoogle Scholar
- Hossain, M. S., Nakamura, T., Kimura, F., Yagi, Y., & Yamaguchi, M. (2018). Practical image quality evaluation for whole slide imaging scanner. Biomedical Imaging and Sensing Conference, 203–206. https://doi.org/10.1117/12.2316764CrossrefGoogle Scholar
- Hossain, M. S., Rahman, M. M., Syeed, M. M., Uddin, M. F., Hasan, M., Hossain, M. A., Ksibi, A., Jamjoom, M. M., Ullah, Z., & Samad, M. A. (2023). DeepPoly: Deep Learning-Based Polyps Segmentation and Classification for Autonomous Colonoscopy Examination. IEEE Access, 11, 95889–95902. https://doi.org/10.1109/access.2023.3310541CrossrefGoogle Scholar
- Hossain, M. S., Syeed, M. M., Fatema, K., & Uddin, M. F. (2022). The Perception of Health Professionals in Bangladesh toward the Digitalization of the Health Sector. International Journal of Environmental Research and Public Health, 19(20), 13695. https://doi.org/10.3390/ijerph192013695CrossrefGoogle Scholar
- Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely Connected Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4700–4708. https://doi.org/10.1109/cvpr.2017.243CrossrefGoogle Scholar
- Iwagami, H., Ishihara, R., Aoyama, K., Fukuda, H., Shimamoto, Y., Kono, M., Nakahira, H., Matsuura, N., Shichijo, S., Kanesaka, T., Kanzaki, H., Ishii, T., Nakatani, Y., & Tada, T. (2021). Artificial intelligence for the detection of esophageal and esophagogastric junctional adenocarcinoma. Journal of Gastroenterology and Hepatology, 36(1), 131–136. https://doi.org/10.1111/jgh.15136CrossrefGoogle Scholar
- Kaminski, M. F., Regula, J., Kraszewska, E., Polkowski, M., Wojciechowska, U., Didkowska, J., Zwierko, M., Rupinski, M., Nowacki, M. P., & Butruk, E. (2010). Quality Indicators for Colonoscopy and the Risk of Interval Cancer. New England Journal of Medicine, 362(19), 1795–1803. https://doi.org/10.1056/nejmoa0907667CrossrefGoogle Scholar
- Luo, H., Xu, G., Li, C., He, L., Luo, L., Wang, Z., Jing, B., Deng, Y., Jin, Y., Li, Y., Li, B., Tan, W., He, C., Seeruttun, S. R., Wu, Q., Huang, J., Huang, D., Chen, B., Lin, S. B., … Xu, R. H. (2019). Real-time artificial intelligence for detection of upper gastrointestinal cancer by endoscopy: A multicentre, case-control, diagnostic study. The Lancet Oncology, 20(12), 1645–1654. https://doi.org/10.1016/s1470-2045(19)30637-0CrossrefGoogle Scholar
- Misawa, M., Kudo, S. E., Mori, Y., Hotta, K., Ohtsuka, K., Matsuda, T., Saito, S., Kudo, T., Baba, T., Ishida, F., Itoh, H., Oda, M., & Mori, K. (2021). Development of a computer-aided detection system for colonoscopy and a publicly accessible large colonoscopy video database (with video). Gastrointestinal Endoscopy, 93(4), 960–967. https://doi.org/10.1016/j.gie.2020.07.060CrossrefGoogle Scholar
- Nishitha, R., Amalan, S., Sharma, S., Preejith, S. P., & Sivaprakasam, M. (2022). Image Quality Assessment for Interdependent Image Parameters Using a Score-Based Technique for Endoscopy Applications. 2022 IEEE International Symposium on Medical Measurements and Applications (MeMeA), 1–6. https://doi.org/10.1109/memea54994.2022.9856448CrossrefGoogle Scholar
- Ozawa, T., Ishihara, S., Fujishiro, M., Kumagai, Y., Shichijo, S., & Tada, T. (2020). Automated endoscopic detection and classification of colorectal polyps using convolutional neural networks. Therapeutic Advances in Gastroenterology, 13, 175628482091065. https://doi.org/10.1177/1756284820910659CrossrefGoogle Scholar
- Pogorelov, K., Randel, K. R., Griwodz, C., Eskeland, S. L., de Lange, T., Johansen, D., Spampinato, C., Dang-Nguyen, D.-T., Lux, M., Schmidt, P. T., Riegler, M., & Halvorsen, P. (2017). KVASIR: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection. Proceedings of the 8th ACM on Multimedia Systems Conference, 164–169. https://doi.org/10.1145/3083187.3083212CrossrefGoogle Scholar
- Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Proceedings of the IEEE International Conference on Computer Vision, 618–626. https://doi.org/10.1109/iccv.2017.74CrossrefGoogle Scholar
- Shakhawat, H. M., Nakamura, T., Kimura, F., Yagi, Y., & Yamaguchi, M. (2020). Automatic Quality Evaluation of Whole Slide Images for the Practical Use of Whole Slide Imaging Scanner. ITE Transactions on Media Technology and Applications, 8(4), 252–268. https://doi.org/10.3169/mta.8.252CrossrefGoogle Scholar
- Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. ArXiv, 1409.1556. https://doi.org/10.48550/arXiv.1409.1556CrossrefGoogle Scholar
- Suzuki, H., Yoshitaka, T., Yoshio, T., & Tada, T. (2021). Artificial intelligence for cancer detection of the upper gastrointestinal tract. Digestive Endoscopy, 33(2), 254–262. https://doi.org/10.1111/den.13897CrossrefGoogle Scholar
- Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–9. https://doi.org/10.1109/cvpr.2015.7298594CrossrefGoogle Scholar
- Tan, M., & Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning, 6105–6114.CrossrefGoogle Scholar
- Tomar, N. K., Jha, D., Ali, S., Johansen, H. D., Johansen, D., Riegler, M. A., & Halvorsen, P. (2021). DDANet: Dual Decoder Attention Network for Automatic Polyp Segmentation. In A. Del Bimbo, R. Cucchiara, S. Sclaroff, G. Maria Farinella, T. Mei, M. Bertini, H. Jair Escalante, & R. Vezzani (Eds.), Pattern Recognition. ICPR International Workshops and Challenges (1st ed., Vol. 12668, pp. 307–314). Springer, Cham. https://doi.org/10.1007/978-3-030-68793-9_23CrossrefGoogle Scholar
- Touvron, H., Sablayrolles, A., Douze, M., Cord, M., & Jegou, H. (2021). Grafit: Learning fine-grained image representations with coarse labels. Proceedings of the IEEE/CVF International Conference on Computer Vision, 874–884. https://doi.org/10.1109/iccv48922.2021.00091CrossrefGoogle Scholar
- Tran, T.-H., Nguyen, P.-T., Tran, D.-H., Manh, X.-H., Vu, D.-H., Ho, N.-K., Do, K.-L., Nguyen, V.-T., Nguyen, L.-T., Dao, V.-H., & Vu, H. (2021). Classification of anatomical landmarks from upper gastrointestinal endoscopic images. 2021 8th NAFOSTED Conference on Information and Computer Science (NICS), 278–283. https://doi.org/10.1109/nics54270.2021.9701513CrossrefGoogle Scholar
Licence and reuse
Open access. This article is licensed under a Creative Commons Attribution 4.0 International Licence (CC BY 4.0), as printed on the article, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original authors and the source. No permission is needed.
How to reuse this articleResponses (0)
Have a comment on this research? Sign in to respond. Responses are moderated and published with your name and affiliation; the authors are invited to reply. How we use your dataWrite a response
About this article
Cite this article
Hossain, M. S., Ahmed, M., Rahman, M. S., Tabassum, M., Karim, F., Hossain, M. A., Hayat Khan, R., Khan, R. H., Syeed, M. M. M. & Uddin, M. F. (2024). AI-Guided Anatomical Landmark and Abnormality Detection for Autonomous Endoscopy Examination. Journal of Computer Science, 20(8), 858–871. https://doi.org/10.3844/jcssp.2024.858.871
- Licence
- CC BY 4.0
- Issue
- Volume 20, Issue 8
- Policies
- Editorial independenceUse of generative AIData availabilityAppeals and complaints
- Version
- Version of record