Rakshith M. D.; Harish H. Kenchannavar

Hybrid Deep Optimal Network for Recognizing Emotions Using Facial Expressions at Real Time

PDF (1593KB), PP.47-58

Views: 0 Downloads: 0

Author(s)

Rakshith M. D. ^1,* Harish H. Kenchannavar ²

1. Department of Computer Science and Engineering, Canara Engineering College, Benjanapadavu, Visvesvaraya Technological University, Belagavi, 590018, India

2. Department of Information Science and Engineering, Gogte Institute of Technology, Belagavi, Visvesvaraya Technological University, Belagavi, 590018, India

* Corresponding author.

DOI: https://doi.org/10.5815/ijisa.2024.03.04

Received: 9 Dec. 2023 / Revised: 14 Jan. 2024 / Accepted: 9 Feb. 2024 / Published: 8 Jun. 2024

Index Terms

Facial Emotion Recognition, Adaptive Median Filter, Aquila Optimizer, Feature Extraction

Abstract

Recognition of emotions by utilizing facial expressions is the progression of determining the various human facial emotions to infer the mental condition of the person. This recognition structure has been employed in several fields but more commonly applied in medical arena to determine psychological health problems. In this research work, a new hybrid model is projected using deep learning to recognize and classify facial expressions into seven emotions. Primarily, the facial image data is obtained from the datasets and subjected to pre-processing using adaptive median filter (AMF). Then, the features are extracted and facial emotions are classified through the improved VGG16+Aquila_BiLSTM (iVABL) deep optimal network. The proposed iVABL model provides accuracy of 95.63%, 96.61% and 95.58% on KDEF, JAFFE and Facial Expression Research Group 2D Database (FERG-DB) which is higher when compared to DCNN, DBN, Inception-V3, R-152 and Convolutional Bi-LSTM models. The iVABL model also takes less time to recognize the emotion from the facial image compared to the existing models.

Cite This Paper

Rakshith M. D., Harish H. Kenchannavar, "Hybrid Deep Optimal Network for Recognizing Emotions Using Facial Expressions at Real Time", International Journal of Intelligent Systems and Applications(IJISA), Vol.16, No.3, pp.47-58, 2024. DOI:10.5815/ijisa.2024.03.04

Reference

[1]Isha Talegaonkar, Kalyani Joshi, Shreya Valunj, Rucha Kohok, and Anagha Kulkarni, “Real time facial expression recognition using deep learning”, in Proceedings of International Conference on Communication and Information Processing (ICCIP), 2019. DOI:10.2139/ssrn.3421486.
[2]M. Kalpana Chowdary, Tu N. Nguyen, and D. Jude Hemanth, “Deep learning-based facial emotion recognition for human–computer interaction applications”, Neural Computing and Applications, pp. 1-18, 2021. DOI: 10.1007/s00521-021-06012-8.
[3]Syed Aley Fatima, Ashwani Kumar, and Syed Saba Raoof, “Real time emotion detection of humans using mini-Xception algorithm”, In IOP Conference Series: Materials Science and Engineering, Vol. 1042, No. 1, pp. 012-027. IOP Publishing, 2021. DOI:10.1088/1757-899X/1042/1/012027.
[4]Yahia Said, and Mohammad Barr, “Human emotion recognition based on facial expressions via deep learning on high-resolution images,” Multimedia Tools and Applications, Vol. 80, No. 16, pp. 25241-25253, 2021.DOI: 10.1007/s11042-021-10918-9.
[5]Luu-Ngoc Do, Hyung-Jeong Yang, Hai-Duong Nguyen, Soo-Hyung Kim, Guee-Sang Lee, and In-Seop Na, “Deep neural network-based fusion model for emotion recognition using visual data”, The Journal of Supercomputing, Vol. 77, No. 1, pp. 10773-10790, 2021.DOI: 10.1007/s11227-021-03690-y.
[6]Edeh Michael Onyema, Piyush Kumar Shukla, Surjeet Dalal, Mayuri Neeraj Mathur, Mohammed Zakariah, and Basant Tiwari, “Enhancement of patient facial recognition through deep learning algorithm: ConvNet”, Journal of Healthcare Engineering, pp.1-8, 2021. DOI: 10.1155/2021/5196000.
[7]Shrey Modi, and Mohammed Husain Bohara, “Facial emotion recognition using convolution neural network”, in 2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS), pp. 1339-1344, IEEE, 2021. DOI: 10.1109/ICICCS51141.2021.9432156.
[8]James Ren Lee, Linda Wang, and Alexander Wong, “Emotionnet Nano: An efficient deep convolutional neural network design for real-time facial expression recognition”, Frontiers in Artificial Intelligence, Vol.3, pp. 609-673, 2021. DOI:10.3389/frai.2020.609673.
[9]Moutan Mukhopadhyay, Aniruddha Dey, RabindraNath Shaw, and Ankush Ghosh, “Facial emotion recognition based on textural pattern and convolutional neural network”, in 2021 IEEE 4th International Conference on Computing, Power and Communication Technologies (GUCON), pp. 1-6. IEEE, 2021.
[10]Jun Liu, Yanjun Feng, and Hongxia Wang, “Facial expression recognition using pose-guided face alignment and discriminative features based on deep learning”, IEEE Access, Vol. 9, pp. 69267-69277, 2021. DOI: 10.1109/ACCESS.2021.3078258.
[11]P.K. Das, A. Pradhan, S. Meher, “Detection of Acute Lymphoblastic Leukemia using Machine Learning Techniques”, in book: Machine Learning, Deep Learning and Computational Intelligence for Wireless Communication, Springer, pp. 425.437, 2021. DOI:10.1007/978-981-16-0289-4_32.
[12]Binh T.Nguyen, Min H.Trinh, Tan V.Phan, Hien D.Nguyen, “An efficient real-time emotion detection using camera and facial landmarks.” In: 2017 seventh international conference on information science and technology (ICIST); 2017. DOI: 10.1109/icist.2017.7926765.
[13]C. Loconsole, C.R. Miranda, G. Augusto, A.Frisoli, V. Orvalho, “Real-time emotion recognition novel method for geometrical facial features extraction.” Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP), Vol.1, pp.378-385, 2014. DOI: 10.5220/0004738903780385.
[14]Giuseppe Palestra, Adriana Pettinicchio, Marco Del Coco, Pierluigi Carcagn, Marco Leo, Cosimo Distante, “Improved performance in facial expression recognition using 32 geometric features.” In: Proceedings of the 18th international conference on image analysis and processing. ICIAP, pp. 518–528, 2015. DOI:10.1007/978-3-319-23234-8_48.
[15]Aya Hassouneh, A. M. Mutawa, and M. Murugappan, “Development of a real-time emotion recognition system using facial expressions and EEG based on machine learning and deep neural network methods,” Informatics in Medicine Unlocked, vol. 20, 2020. DOI: 10.1016/j.imu.2020.100372.
[16]Mehmet Akif Ozdemir, Berkay Elagoz, Aysegul Alaybeyoglu, Reza Sadighzadeh, and Aydin Akan, “Real time emotion recognition from facial expressions using CNN architecture.” In 2019 medical technologies congress (tiptekno), pp. 1-4. IEEE, 2019. DOI:10.1109/TIPTEKNO.2019.8895215.
[17]M. Jeong, B.C. Ko, “Driver’s facial expression recognition in real-time for safe driving”, Sensors ,18:4270, pp.1-17, 2018.
[18]M.A.H Akhand, S. Roy, N. Siddique, M.A.S. Kamal, M.A.S. and T. Shimamura, “Facial emotion recognition using transfer learning in the deep CNN,” Electronics, Vol.10, No.9,1036, pp.1-19, 2021. DOI:10.3390/electronics10091036.
[19]H. Deeb, A. Sarangi, D. Mishra and S.K. Sarangi, “Human facial emotion recognition using improved black hole based extreme learning machine”, Multimedia Tools and Applications, Vol.81, No.17, pp. 24529-24552, 2022. DOI:10.1007/s11042-022-12498-8.
[20]N. Kumari and R. Bhatia, “Efficient facial emotion recognition model using deep convolutional neural network and modified joint trilateral filter”, Soft Computing, pp.1-19, 2021.
[21]X. Sun, P. Xia and F. Ren, “Multi-attention based deep neural network with hybrid features for dynamic sequential facial expression recognition”, Neurocomputing, Vol. 444, pp. 378-389, 2021. DOI:10.1016/j.neucom.2019.11.127.
[22]R. Vedantham and E.S. Reddy, “A robust feature extraction with optimized DBN-SMO for facial expression recognition”, Multimedia Tools and Applications, Vol. 79, No.29, pp.21487-21512, 2020. DOI: 10.1007/s11042-020-08901-x.
[23]D. Liang, H. Liang, Z. Yu and Y. Zhang, “Deep convolutional BiLSTM fusion network for facial expression recognition”, The Visual Computer, Vol. 36, No.3, pp.499-508, 2020. DOI: 10.1007/s00371-019-01636-3.
[24]Shaik Asif Hussain and Ahlam Salim Abdallah Al Balushi, “A real time face emotion classification and recognition using deep learning model,” Journal of physics: Conference series. Vol. 1432, No.1, IOP Publishing, 2020. DOI: 10.1088/1742-6596/1432/1/012087.
[25]Deepak Kumar Jain, Zhang Zhang, and Kaiqi Huang. “Multi angle optimal pattern-based deep learning for automatic facial expression recognition”, Pattern Recognition Letters, Volume 139, pp.157-165, 2020. DOI:10.1016/j.patrec.2017.06.025.
[26]Pritam Khan, Yasin Khan, Sudhir Kumar, Mohammad S. Khan, Amir H. Gandomi, “HVD-LSTM based recognition of epileptic seizures and normal human activity”, Computers in Biology and Medicine, 136, 104684, 2021. DOI: 10.1016/j.compbiomed.2021.104684.
[27]A. G. Dastider, F. Sadik, and S.A. Fattah, “An integrated autoencoder-based hybrid CNN-LSTM model for COVID-19 severity prediction from lung ultrasound”, Computers in biology and medicine, Vol.132, 104296, 2021. DOI: 10.1016/j.compbiomed.2021.104296.
[28]S.T. Chavali, C.T. Kandavalli, T.M. Sugash, and R. Subramani, “Smart Facial Emotion Recognition with Gender and Age Factor Estimation”, Procedia Computer Science, Vol. 218, pp.113-123, 2023. DOI: 10.1016/j.procs.2022.12.407.
[29]K. Bhangale, P. Ingle, R. Kanase, and D. Desale, “Multi-view multi-pose robust face recognition based on VGGNet”, in book: Second International Conference on Image Processing and Capsule Networks: ICIPCN, pp. 414-421, 2021, Springer International Publishing. DOI: 10.1007/978-3-030-84760-9_36.
[30]Bhaskara Rao Jana, Haritha Thotakura, Anupam Baliyan, Majji Sankararao, Radhika Gautamkumar Deshmukh, and Santoshachandra Rao Karanam, “Pixel density based trimmed median filter for removal of noise from surface image”, Applied Nanoscience, Vol. 13, No. 2, pp. 1017-1028, 2023. 10.1007/s13204-021-01950-0.
[31]G.S. Nijaguna, Ananda Babu J, B. D. Parameshachari, Rocío Pérez de Prado, and Jaroslav Frnda, “Quantum Fruit Fly algorithm and ResNet50-VGG16 for medical diagnosis”, Applied Soft Computing, Vol. 136, 2023: 110055. DOI: 10.1016/j.asoc.2023.110055
[32]B. Gao, Yuan Shi, Fengqiu Xu, and Xianze Xu. “An improved Aquila optimizer based on search control factor and mutations.” Processes, Vol. 10, No. 8, 2022: 1451. DOI: 10.3390/pr10081451
[33]D.Lundqvist, A. Flykt & A. Öhman, “The Karolinska Directed Emotional Faces - KDEF, CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, ISBN 91-630-7164-9,1998.
[34]Michael J. Lyons, Miyuki Kamachi, Jiro Gyoba, “Coding Facial Expressions with Gabor Wavelets (IVC Special Issue)”, DOI: 10.5281/zenodo.4029679.
[35]D. Aneja, A. Colburn, G. Faigin, L. Shapiro, B. Mones, “Modeling stylized character expressions via deep learning,” Asian Conference on Computer Vision, pp. 136–153, 2016. DOI:10.1007/978-3-319-54184-6_9.
[36]Nimra Bukhari, Shabir Hussain, Muhammad Ayoub, Yang Yu, and Akmal Khan, “Deep learning based framework for emotion recognition using facial expression”, Pakistan Journal of Engineering and Technology, Vol.5, No. 3, pp.51-57, 2022. DOI:10.51846/vol5iss3pp51-57.
[37]W. Xu and Rayan S. Cloutier, “A facial expression recognizer using modified ResNet-152”, EAI Endorsed Transactions on Internet of Things, Vol. 7, No. 28, 2022. DOI:10.4108/eetiot.v7i28.685.

International Journal of Intelligent Systems and Applications (IJISA)