Electrocardiogram (ECG) has been proven to be a reliable source of information for emotion recognition systems [1–4]. Automated ECG analysis can identify the affective states of users such as happiness, sadness, and stress, among others. Understanding and quantifying the emotional states of humans can have significant effects on intelligent human-machine systems. ECG and other physiological signals have been used in several affective computing applications. For example, [5] performed stress detection using ECG, electromyography (EMG), and galvanic skin response (GSR), during driving tasks. In [6], a dynamic difficulty adjustment mechanism for computer games was proposed to provide tailored gaming experience to individual users by analysing ECG and GSR. An ECG-based deep multitask learning framework was proposed in [3,7] for adaptive simulation. The aim was to provide personalised training experience to individual users based on their

level of expertise and cognitive load.

Although ECG has considerable potential for affective computing, we often lack sufficient labelled data in order to train deep supervised models. To tackle this problem, we propose a deep learning solution based on self-supervised learning [8]. Self-supervised learning is a representation learning approach, in which models are trained using automatically generated labels instead of human annotated labels. There are a number of advantages to self-supervised learning. First, the feature manifolds learnt using this approach are often invariant to inter-instance and intra-instance variations [9] by learning more generalized features rather than task-specific ones. As a result, these models can be reused for different tasks within the same domain. Moreover, self-supervised models require less amount of human-annotated labels to achieve high classification performance.

In this paper we propose ECG-based emotion recognition using multi-task self-supervised learning for the first time. We use two publicly available datasets, SWELL [10] and AMIGOS [11]. First, to train our network with automatically generated labels, we perform 6 different signal transformation tasks. Next, the 6 transformed signals along with the original ones are used to train a multi-task convolutional neural network (CNN) in a self-supervised manner. The proposed CNN architecture consists of 3 convolutional blocks as shared layers followed by 2 task-specific dense layers. As the next step, we use the pre-trained model for emotion classification. To do so, we transfer the weights of the pre-trained network to a new network and train a simple fully-connected layer, and test the framework on the two datasets. Our analysis shows that our self-supervised model is better or competitive in comparison to the same network when trained in fully supervised fashion. Finally, we set a new state-of-the-art for arousal and valence detection for SWELL and AMIGOS datasets.

Self-supervised learning is becoming popular in the field of computer vision [9, 12], natural language processing [13], speech and signal processing [14, 15], and others. Human activity recognition using self-supervised learning was performed in [15], where different signal transformation tasks were carried out as pretext tasks to generate automatic labels. A wide variety of activity recognition tasks such as walking, sitting, and jogging were performed using 6 different publicly available datasets. It was shown in [15] that self-supervised learning helps convolution networks learn high-level features. Another work [12] performed 3D pose estimation using self-supervised learning. Training a model to estimate 3D poses requires large amounts of training data and is highly resourcedependent. To overcome these problems, the authors used available 2D pose data and performed epipolar geometry to calculate 3D poses in self-supervised manner. The obtained 3D poses were used to train a model to perform 3D pose estimation. In [16], a self-supervised learning method was used for action recognition. The model was first trained using a 3D convolution neural network to predict the order of shuffled video clips. Subsequently, the pre-trained model was fine-tuned using nearest neighbour technique for action recognition.

3.1. Self-supervised Learning

Let and be two categories of tasks, namely pretext and downstream tasks, where is trained using automatically generated pseudo-labels while is trained using true labels . Now, let be an example tuple of inputs and pseudo-labels for , where 0 denotes the original signal and 1, 2, .., N correspond to the number of performed signal transformations. Our goal is to obtain a feature manifold F that can easily distinguish between the classes. To do so, we define a model is the set of trainable parameters and is the predicted probability of the transformation task. Accordingly, we find the optimum parameters by minimizing the weighted average of the individual losses of the signal transformation network The total loss is represented as where is the loss coefficient of the task. Consequently, we can use the feature manifold F to perform , since it contains useful information regarding the original signals . In order to perform using the learned feature manifold F, we use a simple model , where is the set of trainable parameters, and is the probability vector of classes. We then calculate the optimum value of by minimizing the cross entropy loss, which can be mathematically expressed as , where M is total number of classes. Figure 1 presents an overview of our pipeline, where self-supervised training is performed followed by transfer learning for emotion recognition.

3.2. Signal Transformation Tasks

We train our self-supervised network on the pretext tasks to enable the network to learn spatiotemporal features and abstract

Fig. 1. The proposed self-supervised architecture is presented. First, a multi-task CNN is trained using automatically generated labels to recognize signal transformations. Then, the weights are transferred to the emotion recognition network, where fully connected layers are trained to classify emotions.

representations of the data. For our developed signal transformation recognition network, 6 different transformations are performed [15,17]. These transformations are as follows:

• Noise addition: Random Gaussian noise is added to the ECG signal.

• Scaling: The magnitude of the ECG Signal is scaled by

• Negation: The amplitude of the ECG signal is multiplied by , causing a vertical flip of the original signal.

• Horizontal flipping: The ECG signal is flipped horizontally along the time axis.

• Permutation: ECG segments are divided into 10 subsegments and shuffled, randomly perturbing their temporal locations.

• Time-warping: Random segments of ECG signals are stretched and squeezed along the x axis.

The above-mentioned transformations are performed on the original ECG signals and stacked to create the input matrix while the corresponding labels of the transformations 0, 1, ..., 6 are stacked to create the corresponding output vector.The parameters for the signal transformations, for instance the scaling factor and amount of noise, are selected empirically with the goal of maximizing the final emotion recognition performance.

3.3. Network Architecture

Signal Transformation Recognition Network: Our multi-task signal transformation recognition network consists of 3 convolutional blocks and 2 dense layers. The convolutional layers are shared for the different tasks while the dense layers are task-specific as shown in Figure 1. Each convolutional block consists of D convolution layers with ReLu activation functions followed by a max-pooling layer of size 8. In the convolutional layers, we gradually increase the number of filters, from 32 to 64 and 128. The kernel size is gradually decreased after each convolutional block from 32 to 16 and 8 respectively. Finally, at the end of the convolutional layers, global max-pooling is performed. The dense layers that follow consist of 2 fully-connected layers with 128 hidden nodes followed by a sigmoid layer. We use 60% dropout in the dense layers and L2 regularization with to overcome possible overfitting. The summary of the network architecture is presented in Table 1.

Emotion Recognition Network: In this step, we develop a simple emotion recognition network with identical convolution layers to the signal transformation recognition network and 2 dense layers with 64 hidden nodes, followed by a sigmoid layer. We then transfer the weights from the convolutional layers of the signal transformation recognition network to the convolution layers of this network. The ECG signals and emotion labels are then used as inputs and outputs for training this network. It should be noted that the weights transferred from the convolutional layers of the signal transformation recognition network are frozen and hence not re-trained (only the dense layer is trained). We keep the fully-connected layers of the network simple in order to be able to evaluate the performance of our approach with regards to the self-supervised learning of the signal transformation tasks.

4.1. Datasets

We use two public datasets, SWELL [10] and AMIGOS [11]. The SWELL dataset was collected from 25 participants with the aim of understanding the mental stress and emotional attributes of employees in a typical office environment under different working conditions. Each of these conditions were designed for a duration of 30-45 minutes. The AMIGOS dataset was collected from 40 participants, where emotional video clips were shown to participants individually and in groups to elicit affective reactions. In both the dataset, participants’ self-assessed affect scores were recorded on a scale of 1-9 for both arousal and valence. The SWELL dataset has been recorded using MOBI devices (TMSI) [18] with selfadhesive electrodes, at a sampling frequency of 2048 Hz while the AMIGOS has been collected using Shimmer sensors [19] at a sampling frequency of 256 Hz. We perform very little pre-processing on the ECG signals. Since the two datasets are recorded at different sampling rates, SWELL ECG signals are first downsampled to 256 Hz. Then we remove ECG baseline wander for both datasets by applying a high-pass IIR filter with a pass-band frequency of 0.8 Hz.

4.2. Model Training

The ECG signals are segmented into a fixed window size of 10 seconds. Each segment is used to generate 6 transformed variations used to train the signal transformation recognition

Table 1. The architecture of the signal transformation recogni- tion network is presented.

network. Our proposed architecture is implemented using TensorFlow on an Nvidia 2070 Ti GPU. To train both networks (signal transformation recognition and emotion recognition), Adam optimizer [20] is used with a learning rate of 0.001 and batch size of 128. The signal transformation recognition network is trained for 30 epochs, while the emotion recognition network is trained for 100 epochs, as steady states are reached with different number of epochs. Similar to [11,21–23] the affective attributes (output labels) for both datasets are converted to binary classes by setting a threshold equal to the mean. We use 10-fold cross-validation to evaluate the performance of our proposed model.

4.3. Performance and Comparison

Figure 2 shows that at 13 epochs, the signal transformation recognition model reaches a steady-state when learning the signal transformation tasks. Clear differences among the steady-state loss of the different transformations are observed, pointing to varied difficulties in the self-supervised training tasks. These tasks allow for learning of ECG-specific representations that, when transferred to the emotion recognition network, aid in classification of emotions with higher accuracy.

Tables 2 and 3 show the performance of our self-supervised approach for emotion classification. The results show that for classification of arousal and valence, our model achieves accuracies of 96% and 95.6% with SWELL, while accuracies of 85.1% and 84% are achieved with AMIGOS. To further evaluate the performance of our model, we compare the results with a fully-supervised version of the emotion recognition network when trained only using the labeled dataset. See Tables 2 and 3. The comparison shows that the self-supervised approach performs competitive to or better than the fully-supervised method, indicating the effectiveness of our method.

Next, we compare our results with prior work on emotion recognition performed on these two datasets. It should be noted that prior works on the SWELL dataset using the ECG modality have mostly focused on stress detection as opposed to classification of arousal and valence. In [24], this dataset was used to perform binary classification of stress levels using support vector machines (SVM), reporting a baseline accuracy

Fig. 2. Individual training losses, and total output loss vs. epoch curves are presented for the signal transformation recognition network. The average and standard deviation values are obtained from the 10 folds of the training phase.

of 64.1% when using ECG and GSR. Similarly in [23], an SVM classifier was used to perform stress detection, reporting an accuracy of 86.36%. A similar task was performed in [22] using Bayesian belief network (BBN), reporting an accuracy of 92.6%. While estimation of arousal and valence has been performed in [25], the problem was formulated as regression, thus preventing a valid comparison with our classification approach. As a result, we also performed stress detection on this dataset and achieved an accuracy of 98.3% and 98.4% in self-supervised and fully-supervised methods respectively. Table 2 compares our self-supervised model and past supervised works on the same dataset, showing that the proposed model performs with higher accuracy.

For the AMIGOS dataset, baseline classification results were provided in [11], where classification of arousal and valence was carried out using a Gaussian Naive Bayes clas-sifier, reporting F1 scores of 54.5% and 55.1% for the two tasks respectively. In [21], a CNN was used to perform clas-sification, reporting accuracies of 81% and 71% for arousal and valence respectively. Other works such as [26] have also performed emotion recognition on AMIGOS. However, different validation schemes are utilized. Table 3 presents our results in comparison to prior work, once again showing that the self-supervised approach outperforms prior work on the same dataset.

Next, in order to evaluate the impact of the self-supervised method on the amount of labeled data required to adequately train a model, we utilize only 1% of the labeled data for training both the self-supervised and fully-supervised classification methods. We first use the entire unlabeled dataset to train the signal transformation recognition network using the automatically generated labels (signal transformations). Then, 1% of the labeled data per-user per-class is used to train and test the emotion recognition network after transferring the weights from the self-supervised network. Next, we also used the same 1%-dataset to train a separate CNN without the self-supervised step. Figure 3 shows the results where the fully-supervised

Table 2. The results of our self-supervised method on the SWELL dataset are presented and compared to prior work as well as the emotion recognition network without the self-supervised step.

Table 3. The results of our self-supervised method on the AMIGOS dataset are presented and compared to prior work as well as the emotion recognition network without the self-supervised step.

Fig. 3. Performance of our method with and without the self- supervised learning step using 1% of the labels in the datasets are presented.

network trained using the very small dataset often performs considerably worse than our self-supervised model.

We performed emotion recognition using ECG signals with self-supervised learning. First, a network was trained in self-supervised manner to recognize a number of simple signal transformations. The weights from the convolutional layers of this network were then transferred to an emotion recognition network and the fully-connected layers were trained using the original dataset. Our approach achieved state-of-the-art performance on SWELL and AMIGOS datasets and showed that using significantly less data, the self-supervised approach can yield acceptable results.

For future work, we will further analyze different components of the self-supervised architecture and determine how individual transformation recognition tasks contribute towards the final emotion recognition outcome. Lastly, the contribution of individual network layers to the classification of downstream tasks will be studied.

[1] Rosalind W. Picard, Elias Vyzas, and Jennifer Healey, “Toward machine emotional intelligence: Analysis of affective physiological state,” IEEE Transactions on Pattern Analysis & Machine Intelligence, , no. 10, pp. 1175–1191, 2001.

[2] Mimma Nardelli, Gaetano Valenza, Alberto Greco, Antonio Lanata, and Enzo Pasquale Scilingo, “Recognizing emotions induced by affective sounds through heart rate variability,” IEEE Transactions on Affective Computing, vol. 6, no. 4, pp. 385–394, 2015.

[3] Pritam Sarkar, Kyle Ross, Aaron J Ruberto, Dirk Rodenburg, Paul Hungler, and Ali Etemad, “Classification of cognitive load and expertise for adaptive simulation using deep multitask learning,” arXiv preprint arXiv:1908.00385, 2019.

[4] Udi Nussinovitch, Keren Politi Elishkevitz, Keren Katz, Moshe Nussinovitch, Shlomo Segev, Benjamin Volovitz, and Naomi Nussinovitch, “Reliability of ultra-short ecg indices for heart rate variability,” Annals of Noninvasive Electrocardiology, vol. 16, no. 2, pp. 117–122, 2011.

[5] Jennifer A Healey and Rosalind W Picard, “Detecting stress during real-world driving tasks using physiological sensors,” IEEE Transactions on Intelligent Transportation Systems, vol. 6, no. 2, pp. 156–166, 2005.

[6] Changchun Liu, Pramila Agrawal, Nilanjan Sarkar, and Shuo Chen, “Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback,” International Journal of Human-Computer Interaction, vol. 25, no. 6, pp. 506–529, 2009.

[7] Kyle Ross, Pritam Sarkar, Dirk Rodenburg, Aaron Ruberto, Paul Hungler, Adam Szulewski, Daniel Howes, and Ali Etemad, “Toward dynamically adaptive simulation: Multimodal classifi-cation of user expertise using wearable devices,” Sensors, vol. 19, no. 19, pp. 4270, 2019.

[8] Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y Ng, “Self-taught learning: transfer learning from unlabeled data,” in Proceedings of the 24th International Conference on Machine Learning. ACM, 2007, pp. 759–766.

[9] Xiaolong Wang, Kaiming He, and Abhinav Gupta, “Transitive invariance for self-supervised visual representation learning,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1329–1338.

[10] Saskia Koldijk, Maya Sappelli, Suzan Verberne, Mark A Neer- incx, and Wessel Kraaij, “The swell knowledge work dataset for stress and user modeling research,” in Proceedings of the 16th International Conference on Multimodal Interaction. ACM, 2014, pp. 291–298.

[11] Juan Abdon Miranda Correa, Mojtaba Khomami Abadi, Nicu Sebe, and Ioannis Patras, “Amigos: A dataset for affect, personality and mood research on individuals and groups,” IEEE Transactions on Affective Computing, pp. 1–1, 2018.

[12] Muhammed Kocabas, Salih Karagoz, and Emre Akbas, “Selfsupervised learning of 3d human pose using multi-view geometry,” arXiv preprint arXiv:1903.02330, 2019.

[13] Fei Wu and Daniel S Weld, “Open information extraction using wikipedia,” in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2010, pp. 118–127.

[14] Marco Tagliasacchi, Beat Gfeller, Félix de Chaumont Quitry, and Dominik Roblek, “Self-supervised audio representation learning for mobile devices,” arXiv preprint arXiv:1905.11796, 2019.

[15] Aaqib Saeed, Tanir Ozcelebi, and Johan Lukkien, “Multi-task self-supervised learning for human activity detection,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 3, no. 2, pp. 61, 2019.

[16] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang, “Self-supervised spatiotemporal learning via video clip order prediction,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.

[17] Terry Taewoong Um, Franz Michael Josef Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kuli´c, “Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks,” arXiv preprint arXiv:1706.00527, 2017.

[20] Diederik P Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.

[21] Luz Santamaria-Granados, Mario Munoz-Organero, Gustavo Ramirez-Gonzalez, Enas Abdulhay, and NJIA Arunkumar, “Using deep convolutional neural network for emotion detection on a physiological signals dataset (amigos),” IEEE Access, vol. 7, pp. 57–67, 2018.

[22] Prabal Verma and Sandeep K Sood, “A comprehensive framework for student stress monitoring in fog-cloud iot environment: m-health perspective,” Medical & Biological Engineering & Computing, vol. 57, no. 1, pp. 231–244, 2019.

[23] S Sriramprakash, Vadana D Prasanna, and OV Ramana Murthy, “Stress detection in working people,” Procedia Computer Science, vol. 115, pp. 359–366, 2017.

[24] Saskia Koldijk, Mark A Neerincx, and Wessel Kraaij, “Detecting work stress in offices by combining unobtrusive sensors,” IEEE Transactions on Affective Computing, vol. 9, no. 2, pp. 227–239, 2016.

[25] Ane Alberdi, Asier Aztiria, Adrian Basarab, and Diane J Cook, “Using smart offices to predict occupational stress,” International Journal of Industrial Ergonomics, vol. 67, pp. 13–26, 2018.

[26] Ross Harper and Joshua Southern, “A bayesian deep learning framework for end-to-end prediction of emotion from heartbeat,” arXiv preprint arXiv:1902.03043, 2019.