A CNN-KNN Based Recognition of Online Handwritten Symbols within Physics Expressions Using Contour-Based Bounding Box (CBBS) Segmentation Technique

Ujwala Kolte; Sachin Naik; Vidya Kumbhar

doi:10.3844/jcssp.2024.783.792

Research Article · Open access · Published 7 May 2024

A CNN-KNN Based Recognition of Online Handwritten Symbols within Physics Expressions Using Contour-Based Bounding Box (CBBS) Segmentation Technique

Ujwala Kolte¹, Sachin Naik¹ & Vidya Kumbhar² Show affiliations

Journal of Computer Science, Volume 20, Issue 7, pp. 783–792 · doi.org/10.3844/jcssp.2024.783.792

6,278 accesses3,248 downloads2 citations (Crossref)Metrics

Cite Check for updates

Abstract

The task of recognizing symbols poses a significant challenge owing to the wide variability in human handwriting. Complexity in terms of the structural representation of symbols used in physics expressions is a major challenge in the recognition process The emergence of online handwriting, fueled by the widespread adoption of handheld digital devices, particularly in educational contexts, highlights the critical importance of precise symbol recognition, especially in the teaching and learning process. In contemporary literature, there is a notable emphasis on LaTex sequencing, symbol recognition and parsing. However, deep learning continues to yield promising results in this domain. The convenience of user input provides benefits to e-learning applications. In this study, we propose three approaches for the recognition of physics symbols within physics expressions (1) A proposed Java user interface for taking input from the user, as convenience of user input provides benefits to e-learning applications. (2) Contour-based bounding box segmentation algorithm, which deals with broken symbols within physics expressions. (3) For recognition, we propose a Convolution Neural Network-K-Nearest Neighbor (CNN-KNN) recognition model, as CNN plays an important role in extracting features, which are further provided as input to the K-NN classifier using the dropout method. Combining these three approaches into a symbol recognition model provides state-of-arts results. Handwritten physics symbols were collected from 20 different writers and each writer has written 5 types of physics expressions under different categories like electric flux, Maxwell’s equations, inductance and pointing vector and moment of Interia. There were 25 classes identified from the 780 samples collected from the users. The recognition rate is identified using (1) Using CNN model, which shows an accuracy of 91.48 and (2) Using the proposed hybrid CNN-KNN model the accuracy reported is 98.06.

KeywordsCNN Contour Based Bounding Box Segmentation (CBBS)K-NN Physics Expression Symbol Recognition

References

Artieres, T., Marukatat, S., & Gallinari, P. (2007). Online Handwritten Shape Recognition Using Segmental Hidden Markov Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(2), 205–217. https://doi.org/10.1109/tpami.2007.38Crossref Google Scholar
Baró, A., Riba, P., Calvo-Zaragoza, J., & Fornés, A. (2019). From Optical Music Recognition to Handwritten Music Recognition: A baseline. Pattern Recognition Letters, 123, 1–8. https://doi.org/10.1016/j.patrec.2019.02.029Crossref Google Scholar
Bresler, M., Prua, D., & Hlavác, V. (2013). Modeling Flowchart Structure Recognition as a Max-Sum Problem. 2013 12th International Conference on Document Analysis and Recognition, 1215–1219. https://doi.org/10.1109/icdar.2013.246Crossref Google Scholar
Coquenet, D., Chatelain, C., & Paquet, T. (2023). End-to-End Handwritten Paragraph Text Recognition Using a Vertical Attention Network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 508–524. https://doi.org/10.1109/tpami.2022.3144899Crossref Google Scholar
Huang, B., Du, C., Zhang, Y., & Kechadi, M. T. (2006). A Hybrid HMM-SVM Method for Online Handwriting Symbol Recognition. Sixth International Conference on Intelligent Systems Design and Applications, 887–891. https://doi.org/10.1109/isda.2006.61Crossref Google Scholar
Jeon, H.-K., & Yang, C.-S. (2021). Enhancement of Ship Type Classification from a Combination of CNN and KNN. Electronics, 10(10), 1169. https://doi.org/10.3390/electronics10101169Crossref Google Scholar
Keshari, B., & Watt, S. (2007). Hybrid Mathematical Symbol Recognition Using Support Vector Machines. Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), 859–863. https://doi.org/10.1109/icdar.2007.4377037Crossref Google Scholar
Kherdekar, V. A., Naik, S. A., & Bafna, P. (2023). A Segmentation-based Token Identification for Recognition of Audio Mathematical Expression. International Journal of Advanced Computer Science and Applications, 14(7), 298–303. https://doi.org/10.14569/ijacsa.2023.0140733Crossref Google Scholar
Lai, S.-H., Lepetit, V., Nishino, K., & Sato, Y. (Eds.). (2017). Computer Vision – ACCV 2016 (Vol. 10112). Springer Cham. https://doi.org/10.1007/978-3-319-54184-6Crossref Google Scholar
Le, A. D., Nguyen, H. D., Indurkhya, B., & Nakagawa, M. (2019). Stroke Order Normalization for Improving Recognition of Online Handwritten Mathematical Expressions. International Journal on Document Analysis and Recognition (IJDAR), 22(1), 29–39. https://doi.org/10.1007/s10032-019-00315-2Crossref Google Scholar
Li, Z., Wang, X., Liu, Y., Jin, L., Huang, Y., & Ding, K. (2024). Improving Handwritten Mathematical Expression Recognition via Similar Symbol Distinguishing. IEEE Transactions on Multimedia, 26, 90–102. https://doi.org/10.1109/tmm.2023.3260648Crossref Google Scholar
Long, J., Shelhamer, E., & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3431–3440. https://doi.org/10.1109/cvpr.2015.7298965Crossref Google Scholar
Michael, J., Labahn, R., Grüning, T., & Zöllner, J. (2019). Evaluating Sequence-to-Sequence Models for Handwritten Text Recognition. 2019 International Conference on Document Analysis and Recognition (ICDAR), 1286–1293. https://doi.org/10.1109/icdar.2019.00208Crossref Google Scholar
Miyao, H., & Maruyama, M. (2007). An Online Handwritten Music Symbol Recognition System. International Journal of Document Analysis and Recognition (IJDAR), 9(1), 49–58. https://doi.org/10.1007/s10032-006-0026-9Crossref Google Scholar
Nguyen, C. T., Truong, T.-N., Nguyen, H. T., & Nakagawa, M. (2021). Global Context for Improving Recognition of Online Handwritten Mathematical Expressions. In J. Lladós, D. Lopresti, & S. Uchida (Eds.), Document Analysis and Recognition – ICDAR 2021 (Vol. 12822, pp. 617–631). Springer, Cham. https://doi.org/10.1007/978-3-030-86331-9_40Crossref Google Scholar
Ptucha, R., Petroski Such, F., Pillai, S., Brockler, F., Singh, V., & Hutkowski, P. (2019). Intelligent Character Recognition Using Fully Convolutional Neural Networks. Pattern Recognition, 88, 604–613. https://doi.org/10.1016/j.patcog.2018.12.017Crossref Google Scholar
Putri, D. U. K., Pratomo, D. N., & Azhari, A. (2023). Hybrid Convolutional Neural Networks-Support Vector Machine Classifier with Dropout for Javanese Character Recognition. TELKOMNIKA (Telecommunication Computing Electronics and Control), 21(2), 346–353. https://doi.org/10.12928/telkomnika.v21i2.24266Crossref Google Scholar
Qiu, Q., Tan, Y., Ma, K., Tian, M., Xie, Z., & Tao, L. (2023). Geological Symbol Recognition on Geological Map Using Convolutional Recurrent Neural Network with Augmented Data. Ore Geology Reviews, 153, 105262. https://doi.org/10.1016/j.oregeorev.2022.105262Crossref Google Scholar
Sanida, T., Sideris, A., Tsiktsiris, D., & Dasygenis, M. (2022). Lightweight Neural Network for COVID-19 Detection from Chest X-ray Images Implemented on an Embedded System. Technologies, 10(2), 37. https://doi.org/10.3390/technologies10020037Crossref Google Scholar
Sejuti, Z. A., & Islam, M. S. (2023). A Hybrid CNN–KNN Approach for Identification of COVID-19 with 5-Fold Cross Validation. Sensors International, 4, 100229. https://doi.org/10.1016/j.sintl.2023.100229Crossref Google Scholar
Shuvo, S. N., Hasan, F., Ahmed, M. U., Hossain, S. A., & Abujar, S. (2021). MathNET: Using CNN Bangla Handwritten Digit, Mathematical Symbols, and Trigonometric Function Recognition. In S. Borah, R. Pradhan, N. Dey, & P. Gupta (Eds.), Soft Computing Techniques and Applications (Vol. 1248, pp. 515–523). Springer, Singapore. https://doi.org/10.1007/978-981-15-7394-1_47Crossref Google Scholar
Smith, W. A. P., & Pillatt, T. (2023). You Only Look for a Symbol Once: An Object Detector for Symbols and Regions in Documents. In G. A. Fink, R. Jain, K. Kise, & R. Zanibbi (Eds.), Document Analysis and Recognition - ICDAR 2023 (Vol. 14191, pp. 227–243). Springer, Cham. https://doi.org/10.1007/978-3-031-41734-4_14Crossref Google Scholar
Sundaram, S., & Ramakrishnan, A. G. (2014). Performance enhancement of online handwritten Tamil symbol recognition with reevaluation techniques. Pattern Analysis and Applications, 17(3), 587–609. https://doi.org/10.1007/s10044-013-0353-7Crossref Google Scholar
Tang, P., Hui, S. C., & Fu, C.-W. (2013). Online chemical symbol recognition for handwritten chemical expression recognition. 2013 IEEE/ACIS 12th International Conference on Computer and Information Science (ICIS), 535–540. https://doi.org/10.1109/icis.2013.6607894Crossref Google Scholar
Urala, K. B., Ramakrishnan, A. G., & Mohamed, S. (2014). Recognition of open vocabulary, online handwritten pages in Tamil script. 2014 International Conference on Signal Processing and Communications (SPCOM), 1–6. https://doi.org/10.1109/spcom.2014.6984002Crossref Google Scholar
Xin, G., Cuiyun, L., Jihong, P., & Weixin, X. (2003). HMM based online hand-drawn graphic symbol recognition. 6th International Conference on Signal Processing, 2002, 1067–1070. https://doi.org/10.1109/icosp.2002.1179973Crossref Google Scholar
Zhu, R., & Wang, Y. (2012). Application of Improved Median Filter on Image Processing. Journal of Computers, 7(4), 838–841. https://doi.org/10.4304/jcp.7.4.838-841Crossref Google Scholar

Licence and reuse

Open access. This article is licensed under a Creative Commons Attribution 4.0 International Licence (CC BY 4.0), as printed on the article, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original authors and the source. No permission is needed.

How to reuse this article

Responses (0)

Have a comment on this research? Sign in to respond. Responses are moderated and published with your name and affiliation; the authors are invited to reply. How we use your dataWrite a response

About this article

Cite this article

Kolte, U., Naik, S. & Kumbhar, V. (2024). A CNN-KNN Based Recognition of Online Handwritten Symbols within Physics Expressions Using Contour-Based Bounding Box (CBBS) Segmentation Technique. Journal of Computer Science, 20(7), 783–792. https://doi.org/10.3844/jcssp.2024.783.792

Download RIS Download BibTeX

Licence: CC BY 4.0
Issue: Volume 20, Issue 7
Policies: Editorial independence Use of generative AI Data availability Appeals and complaints
Version: Version of record