I. J. of Electrical & Electronics Research
Research Article | 
A New Hybrid Approach for Efficient Emotion Recognition using Deep Learning
Author(s) : Mayur Rahul1, Namita Tiwari2, Rati Shukla3, Devvrat Tyagi4, Vikash Yadav5
Published In : International Journal of Electrical and Electronics Research (IJEER) Volume 10, Issue 1
Publisher : FOREX Publication
Published : 30 March 2022
e-ISSN : 2347-470X
Page(s) : 18-22
Abstract
Facial emotion recognition has been very popular area for researchers in last few decades and it is found to be very challenging and complex task due to large intra-class changes. Existing frameworks for this type of problem depends mostly on techniques like Gabor filters, principle component analysis (PCA), and independent component analysis(ICA) followed by some classification techniques trained by given videos and images. Most of these frameworks works significantly well image database acquired in limited conditions but not perform well with the dynamic images having varying faces and images. In the past years, various researches have been introduced framework for facial emotion recognition using deep learning methods. Although they work well, but there is always some gap found in their research. In this research, we introduced hybrid approach based on RNN and CNN which are able to retrieve some important parts in the given database and able to achieve very good results on the given database like EMOTIC, FER-13 and FERG. We are also able to show that our hybrid framework is able to accomplish promising accuracies with these datasets.
Keywords: Recurrent neural networks , Convolutional neural networks , Classification methods, PCA, ICA, EMOTICA, FER-13.
Mayur Rahul, AP, DoCA, UIET, CSJM Univ., Kanpur, UP, India
Namita Tiwari, AP, DoM, SoS, CSJM Univ., Kanpu, UP, India
Rati Shukla, MNNIT, Prayagraj, Allahabad, UP, India
Devvrat Tyagi, AP, ABES Engg. College, Ghaziabad, UP, India
Vikash Yadav, Lec., DoTE, UP, India; Email: vikas.yadav.cs@gmail.com
[1] Jane, E.; Jackson, H.J.; Pattison, P.E. Emotion recognition via facial expression and affective prosody in schizophrenia: A methodological review. Clin. Psychol. Rev. 2002, 22, 789–832. [Cross Ref]
[2] Deepali, A.; Colburn, A.; Faigin, G.; Shapiro, L.; Mones, B. Modeling stylized character expressions via deep learning. In Asian Conference on Computer Vision; Springer: Cham, Switzerland, 2016; pp. 136–153. [Cross Ref]
[3] Chloé, C.; Vasilescu, I.; Devillers, L.; Richard, G.; Ehrette, T. Fear-type emotion recognition for future audio-based surveillance systems. Speech Commun. 2008, 50, 487–503. [Cross Ref]
[4] Rahul Mayur, Yadav Vikash et al, “Zernike Moments based Facial Expression Recognition using Two staged Hidden Markov Model”, Advances in Computer Communication & Computational Sciences Proceedings of IC4S-2018”, Vol. 924, pp. 661-670, May 22, 2019. [Cross Ref]
[5] N. Tiwari, S. Padhye, Analysis on the generalization of Proxy Signature, Security and Communication Network, Wiley, 2013 Vol. 6, pp. 549-556. [Cross Ref]
[6] Meng, Q.; Hu, X.; Kang, J.; Wu, Y. On the effectiveness of facial expression recognition for evaluation of urban sound perception. Sci. Total Environ. 2020, 710, 135484. [Cross Ref]
[7] Marco, L.; Carcagnì, P.; Distante, C.; Spagnolo, P.; Mazzeo, P.L.; Rosato, A.C.; Petrocchi, S. Computational assessment of facial expression production in ASD children. Sensors 2018, 18, 3993. [Cross Ref]
[8] Ali, M.; Chan, D.; Mahoor, M.H. Going deeper in facial expression recognition using deep neural networks. In Proceedings of the IEEE 2016 IEEEWinter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA, 7–10 March 2016. [Cross Ref]
[9] Petrantonakis, C.P.; Hadjileontiadis, L.J. Emotion recognition from EEG using higher order crossings. IEEE Trans. Inf. Technol. Biomed. 2010, 14, 186–197. [Cross Ref]
[10] Wu, C.-H.; Chuang, Z.-J.; Lin, Y.-C. Emotion recognition from text using semantic labels and separable mixture models. ACM Trans. Asian Lang. Inf. Process. TALIP 2006, 5, 165–183. [Cross Ref]
[11] Courville, P.L.C.; Goodfellow, A.; Mirza, I.J.M.; Bengio, Y. FER-2013 Face Database; Universit de Montreal: Montréal, QC, Canada, 2013.
[12] LeCun, Y. Generalization and network design strategies. Connect. Perspect. 1989, 119, 143–155.
[13] Rahul Mayur, Yadav Vikash et al, “Gabor Filter and ICA based Facial Expression Recognition using Two Layered Hidden Markov Model”, Advances in Computational Intelligence and Communication Technology Proceedings of CICT-2019”, Vol. 1086, pp. 511-518, June 19, 2020. [Cross Ref]
[14] Singh Swarnima & Yadav Vikash, “Face Recognition using HOG Feature Extraction and SVM Classifier”, International Journal of Emerging Trends in Engineering Research (IJETER), Vol. 8, No. 9, pp. 6437-6440, September 2020. [Cross Ref]
[15] Cohn, F.J.; Zlochower, A. A computerized analysis of facial expression: Feasibility of automated discrimination. Am. Psychol. Soc. 1995, 2, 6.
[16] Ekman, P.; Friesen, W.V. Constants across cultures in the face and emotion. J. Personal. Soc. Psychol. 1971, 17, 124.
[17] Friesen, E.; Ekman, P.; Friesen, W.; Hager, J. Facial Action Coding System: A Technique for the Measurement of Facial Movement; Psychologists Press: Hove, UK, 1978. [Cross Ref]
[18] Alex, K.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 2012, 25, 1097–1105. [Cross Ref]
[19] Pooya, K.; Paine, T.; Huang, T. Do deep neural networks learn facial action units when doing expression recognition? In Proceedings of the IEEE International Conference on Computer Vision Workshops, Santiago, Chile, 7–13 December 2015. [Cross Ref]
[20] Liu, P.; Han, S.; Meng, Z.; Tong, Y. Facial expression recognition via a boosted deep belief network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 1805–1812. [Cross Ref]
[21] Barsoum, E.; Zhang, C.; Ferrer, C.C.; Zhang, Z. Training deep networks for facial expression recognition with crowd-sourced label distribution. In Proceedings of the 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan, 12–16 November 2016. [Cross Ref]
[22] Han, Z.; Meng, Z.; Khan, A.-S.; Tong, Y. Incremental boosting convolutional neural network for facial action unit recognition. Adv. Neural Inf. Process. Syst. 2016, 29, 109–117.
[23] Meng, Z.; Liu, P.; Cai, J.; Han, S.; Tong, Y. Identity-aware convolutional neural network for facial expression recognition. In Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition, Washington, DC, USA, 30 May–3 June 2017; pp. 558–565. Sensors 2021, 21, 3046 16 of 16. [Cross Ref]
[24] Marrero Fernandez, P.D.; Guerrero Pena, F.A.; Ren, T.; Cunha, A. Feratt: Facial expression recognition with attention net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern RecognitionWorkshops, Long Beach, CA, USA, 16–20 June 2019. [Cross Ref]
[25] Wang, K.; Peng, X.; Yang, J.; Lu, S.; Qiao, Y. Suppressing uncertainties for large-scale facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020. [Cross Ref]
[26] Peng, K.W.; Yang, X.; Meng, D.; Qiao, Y. Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans. Image Process. 2020, 29, 4057–4069. [Cross Ref]
[27] Gan, Y.; Chen, J.; Yang, Z.; Xu, L. Multiple attention network for facial expression recognition. IEEE Access 2020, 8, 7383–7393.
[28] Arpita, G.; Arunachalam, S.; Balakrishnan, R. Deep self-attention network for facial emotion recognition. Proc. Comput. Sci. 2020, 17, 1527–1534. [Cross Ref]
[29] Shan, L.; Deng,W. Deep facial expression recognition: A survey. IEEE Trans. Affect. Comput. 2020.
[30] Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. 2015. Longterm recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2625-2634. [Cross Ref]
[31] R. Kosti, J.M. Álvarez, A. Recasens and A. Lapedriza, "Context based emotion recognition using emotic dataset", IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2019. [Cross Ref]
[32] I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, Y. Zhou, C. Ramaiah, F. Feng, R. Li, X. Wang, D. Athanasakis, J. Shawe-Taylor, M. Milakov, J. Park, R. Ionescu, M. Popescu, C. Grozea, J. Bergstra, J. Xie, L. Romaszko, B. Xu, Z. Chuang, and Y. Bengio. Challenges in representation learning: A report on three machine learning contests. Neural Networks, 64:59--63, 2015. Special Issue on "Deep Learning of Representations" [Cross Ref]
[33] D. Aneja, A. Colburn, G. Faigin, L. Shapiro, and B. Mones. Modeling stylized character expressions via deep learning. In Proceedings of the 13th Asian Conference on Computer Vision. Springer, 2016. [Cross Ref]
[34] Grahlow M, Rupp CI, Derntl B (2022) The impact of face masks on emotion recognition performance and perception of threat. PLoS ONE 17(2): e0262840. https://doi.org/10.1371/journal.pone.0262840 [Cross Ref]
[35] Wang Kay Ngai, Haoran Xie, Di Zou, Kee-Lee Chou, Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources, Information Fusion, Volume 77, 2022, Pages 107-117, ISSN 1566-2535, https://doi.org/10.1016/j.inffus.2021.07.007. [Cross Ref]
[36] Minaee, S.; Minaei, M.; Abdolrashidi, A. Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network. Sensors 2021, 21, 3046. https:// doi.org/10.3390/s21093046 [Cross Ref]
[37] Shashank M Gowda and H N Suresh (2022), Facial Expression Analysis and Estimation Based on Facial Salient Points and Action Unit (AUs). IJEER 10(1), 7-17. DOI: 10.37391/IJEER.100102. [Cross Ref]
Mayur Rahul, Namita Tiwari, Rati Shukla, Devvrat Tyagi and Vikash Yadav (2022), A New Hybrid Approach for Efficient Emotion Recognition using Deep Learning. IJEER 10(1), 18-22. DOI: 10.37391/IJEER.100103.