EmotiEffLib: Library for Efficient Emotion Analysis and Facial Expression Recognition

EmotiEffLib (ex-HSEmotion) is a lightweight library for emotion and engagement recognition in photos and videos. It can be used in Python and C++. It provides flexibility with backend support for Pytorch and ONNX, enabling efficient real-time analysis across various platforms.

This repository contains two implementations of EmotiEffLib: Python and C++.

Documentation

Full documentation is available here.

Installing

Detailed building and installing instruction provided in the pages related to each library: Python and C++.

Quick start guide

Python interface of EmotiEffLib

• One image emotion recognition notebook
• Predict emotions on video notebook
• Predict engagement and emotions on video notebook

C++ interface of EmotiEffLib

• One image emotion recognition notebook
• Predict emotions on video notebook
• Predict engagement and emotions on video notebook

Usage

Detailed examples of using the Python and C++ modules are provided in the Tutorials.

If you want to run EmotiEffCppLib then prepare the models for inference with C++ library:

python models/prepare_models_for_emotieffcpplib.py

Also, in the folder training_and_examples you can find a number of examples of usage our models and training process. This folder also contains an example of mobile application for recognizing user emotions.

In order to run our code on the datasets, please prepare them firstly using our TensorFlow notebooks: train_emotions.ipynb, AFEW_train.ipynb and VGAF_train.ipynb.

NOTE!!! The models were updated so that they should work with timm library of version 0.9.*. However, for v0.1 version, please be sure that EfficientNet models for PyTorch are based on old timm 0.4.5 package, so that exactly this version should be installed by the following command:

pip install timm==0.4.5

News

• Our models let our team HSEmotion took the 1st place in the Fine-Grained Violence Detection and the 2nd places (runner-up) in the Expression Recognition and Action Unit Detection challenges during the 10-th Affective Behavior Analysis in-the-wild (ABAW) Competition
• • Our models let our team HSEmotion took the 1st places in the Expression Recognition and Ambivalence/Hesitancy Recognition Challenges and the 3rd places in the Action Unit Detection and Emotional Mimicry Intensity Estimation challenges during the 8-th Affective Behavior Analysis in-the-wild (ABAW) Competition
• Our models let our team HSEmotion took the second place in the Compound Expression Recognition Challenge and the 3rd place in the Action Unit Detection during the 6-th ABAW Competition
• The paper "Facial Expression Recognition with Adaptive Frame Rate based on Multiple Testing Correction" has been accepted as Oral talk at ICML 2023. The source code to reproduce the results of this paper are available at this repository, see subsections "Adaptive Frame Rate" at abaw3_train.ipynb and train_emotions-pytorch-afew-vgaf.ipynb
• Our models let our team HSE-NN took the first place in the Learning from Synthetic Data (LSD) Challenge and the 3rd place in the Multi-Task Learning (MTL) Challenge in the fourth ABAW Competition
• Our models let our team HSE-NN took the 3rd place in the multi-task learning challenge, 4th places in Valence-Arousal and Expression challenges and 5th place in the Action Unite Detection Challenge in the 3rd ABAW Competition. Our approach is presented in the paper accepted at CVPR 2022 ABAW Workshop.

Details

All the models were pre-trained for face identification task using VGGFace2 dataset. In order to train PyTorch models, SAM code was borrowed.

We upload several models that obtained the state-of-the-art results for AffectNet dataset. The facial features extracted by these models lead to the state-of-the-art accuracy of face-only models on video datasets from EmotiW 2019, 2020 challenges: AFEW (Acted Facial Expression In The Wild), VGAF (Video level Group AFfect), EngageWild; and ABAW CVPR 2022 and ECCV 2022 challenges: Learning from Synthetic Data (LSD) and Multi-task Learning (MTL).

Here are the performance metrics (accuracy on AffectNet, AFEW and VGAF), F1-score on LSD, on the validation sets of the above-mentioned datasets and the mean inference time for our models on Samsung Fold 3 device with Qualcomm 888 CPU and Android 12:

Please note, that we report the accuracies for AFEW and VGAF only on the subsets, in which MTCNN detects facial regions. The code contains also computation of overall accuracy on the complete testing set, which is slightly lower due to the absence of faces or failed face detection.

Additional models trained using Neural Architecture Search (SuperNet and two subnetworks) are available here

Research papers

If you use our models, please cite the following papers:

@inproceedings{savchenko2023facial,
  title = 	 {Facial Expression Recognition with Adaptive Frame Rate based on Multiple Testing Correction},
  author =       {Savchenko, Andrey},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning (ICML)},
  pages = 	 {30119--30129},
  year = 	 {2023},
  editor = 	 {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  url={https://proceedings.mlr.press/v202/savchenko23a.html}
}

@article{savchenko2022classifying,
  title={Classifying emotions and engagement in online learning based on a single facial expression recognition neural network},
  author={Savchenko, Andrey V and Savchenko, Lyudmila V and Makarov, Ilya},
  journal={IEEE Transactions on Affective Computing},
  year={2022},
  publisher={IEEE},
  url={https://ieeexplore.ieee.org/document/9815154}
}

License

The code of EmotiEffLib is released under the Apache-2.0 License. There is no limitation for both academic and commercial usage.