Emotion cause extraction (ECE) [Lee et al., 2010] aims to extract possible causes for a given emotion clause. While ECE has attracted an increasing attention due to its theoretical and practical significance, it requires that the emotion signals should be given. In practice emotion annotation is rather labor intensive, limiting the applicability of ECE in practical settings. To address the limitation of ECE, the emotion-cause pair extraction (ECPE) task was recently proposed in [Xia and Ding, 2019]. Unlike ECE [Lee et al., 2010], ECPE aims to extract emotions and causes without any given emotion
(a) An example document and extracted emotion-cause pairs Emotion Cause Pair (C4, C2) (C4, C3)
Figure 1: Extracting emotion-cause pairs of the given document via learning to link.
signals, and thus better aligns with real-world applications. An illustrative example is given in Figure 1(a), showing that clause 4 serves as the emotion and clauses 2 and 3 are the corresponding causes. Typically, ECPE is formulated as extracting emotion-cause pairs, e.g., (clause 4, clause 2) and (clause 4, clause 3), directly from provided documents.
ECPE is a challenging task as it requires the extraction of emotions, causes and emotion-cause pairs. Existing work in ECPE mainly focuses on how to collaboratively extract emotions and causes and combine them in an appropriate way. Thus, a two-stage method [Xia and Ding, 2019] is typically adopted, which divides pair extraction into two steps: firstly detecting emotions and causes, and then pairing them based on the likelihood of cartesian products between them. Such pipeline method approach is intuitive and straightforward. However, two critical issues arise. One is the error propagation from the first step to the second. The other issue is that, owing to the step-by-step structure, the two-stage models are often computationally expensive.
To tackle the afore-mentioned issues, we propose an end-to-end multi-task learning model for predicting emotion-cause pairs, namely E2EECPE, which connects emotions and causes within one single stage. More specifically, benefit-ting from end-to-end architectures, E2EECPE resolves the issue of error propagation while substantially reduces the computation time compared with the state-of-the-art models. Particularly, our model views emotion-cause pair extraction as a link prediction problem, as shown in Figure 1(b), and predicts whether there exists a directional link from the emotion to cause. Furthermore, we incorporate into the model two auxiliary tasks, namely emotion extraction and cause extraction, which are oriented to further enhance the expressiveness of the intermediate emotion representation and cause representation.
Extensive experiments are carried out on a benchmarking ECPE dataset. The experimental results demonstrate the effectiveness and efficiency of the proposed E2EECPE model, in comparison with a variety of state-of-the-art baselines. Moreover, further ablation study indicates that the auxiliary tasks are beneficial.
First of all, our work is related to extracting causes based on emotions explicitly presented in documents, i.e., ECE [Lee et al., 2010]. Earlier work views ECE as a word-level sequence tagging problem and tries to solve it with corresponding tagging techniques. Therefore, primary efforts have been made on discovering refined linguistic features [Chen et al., 2010; Lee et al., 2013], yielding improved performance. In line with other tagging related tasks such as named entity recognition (NER), support vector machines (SVMs) [Gui et al., 2014] and conditional random fields (CRFs) [Lafferty et al., 2001] have been used for ECE. More recently, instead of concentrating on word-level cause detection, clause-level extraction [Gui et al., 2016] is put forward in that the impact of individual words in a cause can span over the whole sequence in the clause.
With the emergence and development of deep representation learning, neural models has also been utilized in ECE. [Cheng et al., 2017] leverages long short-term memory networks (LSTMs) [Hochreiter and Schmidhuber, 1997] to promote the context awareness of clause modelling. [Gui et al., 2017] views the information extraction problem as the retrieval task in question answering (QA) and examines the effect of memory networks [Sukhbaatar et al., 2015] for extraction. Likewise, taking advantage of attention mechanism [Bahdanau et al., 2014], [Li et al., 2018] employs a co-attention based model and achieves the state-of-the-art performance.
In light of recent advances in multi-task learning, joint extraction of emotions and causes is investigated [Chen et al., 2018] to exploit the mutual information between two correlated tasks. However, these works do not explicitly combine two tasks into one. Thereafter, [Xia and Ding, 2019] argues that, while co-extraction of emotions and causes is important, emotion-cause pair extraction (ECPE) is a more challenging problem that is worth putting more emphasis on. Nevertheless, [Xia and Ding, 2019] adopts a two-stage approach, which performs emotion and cause extraction first and then pairs the extracted emotions and causes. As discussed in the
Figure 2: An overview of E2EECPE. EE, CE are short for emotion extraction, cause extraction respectively.
previous section, such two stage approach suffers from error propagation and high computation cost.
Our work aims to tackle these challenges in ECPE. Rather than processing emotion-cause pair extraction as a two stage task (as used in the existing work), we consolidate two stages into a unified multi-task learning framework, and further consider it as a link prediction task which could be solved in an end-to-end manner.
To solve ECPE in end-to-end fashion, we take inspiration from the link prediction problem in graph learning, which aims at predicting potential edges between unconnected vertices in a graph. Essentially, if we consider emotion-cause pairs in a document as triplets in a graph, then the extraction of such pairs is a sort of link prediction from a graph that is at first armed with no edges but only vertices. In order to achieve above procedure, we borrow the idea of learning a graph-based dependency parser [Dozat and Manning, 2016] and adapt it to our target task. Coupling link prediction with auxiliary emotion extraction and cause extraction tasks, our model is capable of jointly, and more effectively, extracting emotions, causes, and emotion-cause pairs.
3.1 Problem Formulation
Generally, for any provided document D, we could split it into a sequence of clauses based on punctuation, i.e., D = , where
could further be decomposed into words, i.e.,
. Here, |D| is the number of clauses in the document and
is the number of words in the i-th clause. ECPE aims to extract a set of |P| emotion-cause pairs
from the document D, where
rep- resents the emotion clause and the cause clause in the k-th pair, respectively.
3.2 Overall Architecture
An overview of the proposed E2EECPE approach is shown in Figure 2. The bottom layer is a clause encoder (Section 3.3) and a document modelling layer (Section 3.4) which transform the word embeddings into the contextualized clause representations. The middle part consists of auxiliary tasks (Section 3.7), i.e., emotion extraction and cause extraction. The top most part is a biaffine attention layer (Section 3.5) which first encodes interaction between the emotion representation and cause representation, and then outputs a postion-weighted pair matrix for pair extraction.
3.3 Embedding & Clause Encoder
With the purpose of integrating words into clause-level neural models, we embed each word in a clause into lowdimensional vectors [Bengio et al., 2003], by which we could represent each word in the clause with its vector representation2 , where
and
is the dimen- sionality of the embedding.
After that, we need to attain contextualized representations of clauses. Owing to the recognized performance and local context awareness of the convolutional neural networks (CNNs) on text classification benchmarks [Kim, 2014], we adopt CNNs as the backbone of our clause encoder.
For an embedded clause , we apply one- dimensional convolution operations with kernels of different sizes over the word sequence:
where denotes the t-th convolution operation and
is the output of the operation.
is the number of filters employed in one convolution operation and
used here is actually
.
Then max-pooling is used to distill the features for concatenation. Hence, we finally get context-aware features for the clause:
where is the convoluted feature and
means vector concatenation. |t| is the total number of convolution operations.
3.4 Document Modelling
Since we have sequential clauses in a document, the influ-ences brought by document-level structures become a crucial part that we should fit into our model. A straightforward idea is to leverage temporal relations among clauses with LSTMs.
Specifically, provided with the encoded clause representations , we employ a bidirectional LSTM to update clause-level features and get
:
where and
denote the forward and backward unidirectional LSTMs, respectively.
is the dimensionality of hidden states for a unidirectional LSTM.
3.5 Biaffine Attention
Motivated by advances in link prediction [Schlichtkrull et al., 2018], we can directly compute the similarity scores among vertex representations (clause representations in our task), e.g., for any representations of vertex p and q, to make predictions. However, the above predictions are only concerned with undirectional circumstances since
, which is not adequate for emotion-cause pair extrac- tion. To solve the problem, we utilize biaffine transform to complete the filling of adjacent matrices, which are called pair matrices in our work. This idea is similar to dependency parsing [Dozat and Manning, 2016] that is also directional.
Emotion & Cause Representation According to biaffine attention mechanism, each vertex in the graph should have two independent representations, i.e., one is for pointing out and the other for pointed in. In doing so, the pair matrix output by the transformation is asymmetric and direction-aware.
The emotion representation and cause representation are separately offered as below:
where and
are two sets of trainable weights and biases, respectively for the emotion and cause representations.
Biaffine Transform Then, we implement biaffine transform on the collected emotion and cause representations. In other words, with the purpose of merging these two kinds of representations into our aimed pair matrices, we fold emotion-cause dynamics into two components. On the one hand, we need to perform a bilinear like operation on each possible pair of emotion and cause. On the other hand, we believe bilinear transform is not enough to deal with such complicated interactions, and thus we facilitate it by injecting bias.
More specifically, we calculate each entry in the expected pair matrix as follows:
where and
are learnable parameters of affine transform, while
indicates an entry of the pair matrix in the p-th row, q-th column.
Constrained by the inherent property of an adjacent matrix, we further activate the pair matrix with the sigmoid function :
3.6 Asymmetric Position Weight Matrix
A trivial observation on the co-occurrence patterns of emotions and causes is that the emotion and the cause in a unique pair appear near each other in term of their absolute positions in the document. Thus, position embeddings are introduced to directly encode positions into vectors [Xia and Ding, 2019]. Different from the existing approaches, our work is based on graph learning, and thereby can not be aided by manipulation of embeddings. Instead, we apply proximity weights on features as in [Zhang et al., 2019], but extent it to matrices.
Moreover, we notice that in reality people are more likely to inform the causes before expressing emotions, which will be verified with statistics of dataset, implying we should assign asymmetric position weight matrices, instead of symmetric ones, to the original matrix representations.
Specifically, the entries in the lower triangular of position weight matrix for the given document is of more significance:
where is a small number for smoothing.
Finally, we obtain the features for indicating pair links as follows:
where denotes element-wise multiplication.
3.7 Multi-task Setting
As aforementioned, emotion extraction and cause extraction can be viewed as auxiliary tasks to augment the emotion and cause representations for constructing more expressive pair matrix. Hence we develop a multi-task paradigm, which shares the fundamental part of network structure for the main task with auxiliary tasks. To achieve this goal, we first acquire features dedicated for classification with following procedure:
where ,
and
,
are again two sets of trainable weights and biases, respectively.
Subsequently, predictions are produced by two fully connected layers followed by softmax normalization layers:
where ,
and
,
are weights and biases for learning.
3.8 Training Objective
Eventually, the whole structure can be trained by standard gradient descent. Accordingly, the objective function is a combination of cross entropy with -norm regularization, formulated as below:
3.9 Inference
With well trained model, we can infer emotion-cause pairs by comparing each entry in with a predefined threshold
where is the inference result matrix with binary (1-0) indicators.
4.1 Dataset
We carry out experiments on a publicly available dataset for emotion-cause pair extraction, released by [Xia and Ding, 2019]. Consisting of news crawled from web, the dataset is referred to as NEWS in the rest of the paper. It is randomly split into ten folds. In our experiments, to evaluate our trained model, each fold is further divided into two parts, namely train set and test set which respectively take 90% and 10% of the data. Table 1 shows some basic statistics of the dataset. A key observation is that most documents only contain one emotion-cause pair therein, implying the sparsity of the pair matrix. Therefore the issue of label imbalance will be elaborated in following discussions. Moreover, a large amount of emotion-cause pairs have the emotion and the cause within 1 relative offset, suggesting the necessity of using proximity constraints (exactly what position weight matrix does) in the predicted pair matrix.
4.2 Parameter Settings
For all our experiments, pre-trained word vectors on Weibo (a Chinese micro-bloging website) using Word2Vec [Mikolov et al., 2013] are leveraged to initialize the word embeddings. The dimensionality of embeddings (i.e., ) is set to 200. We use 4 convolutional layers (i.e., |t|) whose kernel sizes are {2,3,4,5} for the clause encoder and the number of filters for all the convolutional layers (i.e.,
) is 50, for capturing gramlevel features. In order to avoid overfitting, we apply dropout to embeddings and outputs of the clause encoder, yielding 0.5 probability of randomized zeroes on features. The dimensionality for hidden states of a unidirectional LSTM (i.e.,
) is 300. The dimensionalities for all fully connected layers in the main task and auxiliary tasks (i.e.,
) are 100. Moreover, the batch size and learning rate are determined through grid parameter search, which are 32 and 10-3, respectively. The
Table 1: Statistics of the dataset. EC stands for emotion-cause, and relative offset indicates the absolute distance between the emotion and the cause of a pair in the document.
coefficient for -norm regularization (i.e.,
) is 10-5 . Based on a pilot study, we find the best value for the threshold (i.e.,
) in the inference stage is 0.3, which will be detailed in next section. The coefficient for the trade-off in objective function (i.e.,
) is 1. In addition, the smoothing term in the calculation of position weight matrix (i.e.,
) is 1. Furthermore, Adam is used as the optimizer and all trainable parameters are randomly initialized with uniform distribution [He et al., 2015].
4.3 Baselines & Evaluation Metrics
Our approach3 is compared with a range of strong baselines, which are the state-of-the-art methods proposed by [Xia and Ding, 2019] for emotion-cause pair extraction. These baselines are either one-stage or two-stage models.
The two-stage models are listed below. They first extract emotions and causes with multi-task architectures independently or interactively, then classify the cartesian products of emotions and causes extracted in the first stage into pairs or non-pairs.
• Indep firstly considers emotion extraction and cause extraction as independent tasks and extract emotions and causes with multi-task learning, then pairs the extracted emotions and causes with a classifier.
• Inter-CE and Inter-EC typically follow the procedure of Indep, however, assist emotion extraction and cause extraction with directed interaction modelling. The one-stage models neglect the second stage in the two-stage models, and consider the cartesian products as predictions. They are listed as follows.
• Indep w/o filter removes the classifier of Indep. • Inter-CE w/o filter removes the classifier of Inter-CE. • Inter-EC w/o filter removes the classifier of Inter-EC. Precision, recall, and macro F1 measures are adopted as effectiveness metrics in our experiments. Meanwhile, runtime is used as a measure of efficiency. The final results are obtained by averaging the ten folds results.
5.1 Results in Effectiveness
We perform a comparison of E2EECPE with one-stage and two-stage baseline models to quantitatively understand in what ways E2EECPE is more effective than the baselines.
Table 2 gives the results in terms of precision, recall and macro-F1 measures. The comparison results demonstrate that our model E2EECPE consistently outperforms the baselines for the main task (emotion-cause pair extraction) with regard to recall and F1, indicating the representation power and the effectiveness of our model. Nevertheless, we also observe that our model performs less well in precision than the two-stage baseline models. With additional observation that the baseline models are performing poorly on recall, we conjecture the existing models suffer from predicting only few testing instances as pairs.
Furthermore, E2EECPE is superior on the two auxiliary tasks (emotion extraction and cause extraction). We attribute the improvement to multi-task structure in our model which combines auxiliary tasks and the main task. Apart from that, the one-stage models yield lower results than E2EECPE on cause extraction, suggesting that error propagation is a comparably severe issue in the existing models but is alleviated in our model.
5.2 Runtime Analysis
To confirm that our end-to-end model is more efficient than the two-stage models, we perform runtime analysis among the two-stage baseline models and ours. Concretely, we display the average running time consumed by models on a epoch in different folds.
Figure 3: Runtime analysis (s).
Results listed in Figure 3 suggests that our model is 6-7 times faster than two-stage models, indicating the efficiency of our end-to-end model.
5.3 Ablation Study
To understand the efficacy of auxiliary tasks and position weight matrix, we conduct an ablation study on E2EECPE. Specifically, we separately ablate auxiliary tasks and position weight matrix from E2EECPE, and call them E2EECPE w/o auxiliary and E2EECPE w/o position, respectively.
The results in Table 3 show a significant performance drop of E2EECPE w/o auxiliary and a relatively minor drop of
Table 2: Comparison results of emotion extraction, cause extraction, and emotion-cause pair extraction with precision, recall, and F1-measure as metrics. The results in bold are the best performing ones under each column. The results of emotion extraction and cause extraction for one-stage and two-stage models are exactly the same because one-stage models are ablated ones of two-stage models. † indicates results that are significantly better than best performing baseline Inter-EC with paired t-test (p is smaller than 0.05).
Table 3: Ablation study results. The results in bold are the best performing ones under each column.
Table 4: Effect of threshold. The results in bold are the best performing ones under each column.
E2EECPE w/o position compared with E2EECPE, verifying the remarkable benefit of the multi-task learning schema. Meanwhile, the results that E2EECPE w/o position only differs slightly from E2EECPE based on all metrics, indicate the fact that imposing position information is still of importance.
5.4 Effect of Threshold in Inference Stage
Inference based on pair matrix is powerful, yet we do not exactly know what threshold (i.e., ) is the most suitable one for its expressiveness. It is therefore helpful to explore the effect of the threshold by altering it and examining the results.
From Table 4, we conclude that 0.3 is the most appropriate one for our studied task. With increases of , drops of F1 are noted, implying potential loss of extracted pairs. In addition, we also speculate that the reason why the best value is not around 0.5 (the expectation of random variables ranging uniformly from 0 to 1) is that the element-wise multiplication of a position weight matrix with the sigmoid-activated pair matrix produces a smaller expectation (as upper bound decreases).
Table 5: The results for verifying the issue of label imbalance.
5.5 Issue of Label Imbalance
In order to measure the impact brought by label imbalance, typically in the form of pair matrix sparsity, we remove the examples containing more than one pair for test set in each fold to make up a Hard dataset, then record the mean results across ten folds correspondingly.
We can observe in Table 5 that our model encounters a failure on the Hard dataset with decreases on precision and F1 measure, suggesting that further investigation is needed to solve this problem.
The emotion-cause pair extraction task is a new and more realistic task that seeks to identify emotion-cause pairs in documents. However, previous two-stage models are inherently limited by the idea of solving this task via two stages. To this end, we propose an end-to-end model that regards the oriented problem as predicting directional links between emotions and causes via biaffine attention. Additionally, we also aid the model with auxiliary tasks and position weight matrix. Experimental results prove the superiority of our model over other baselines.
Based on the work in this paper, we believe there are some promising directions yet to be explored. On the one hand, more fancy models such as graph neural networks are expected to be developed to incorporate with learned position information instead of refined one. On the other hand, the label imbalance issue should be addressed with task-specific tactics.
[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation
by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
[Bengio et al., 2003] Yoshua Bengio, R´ejean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003.
[Chen et al., 2010] Ying Chen, Sophia Yat Mei Lee, Shoushan Li, and Chu-Ren Huang. Emotion cause detection with linguistic constructions. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 179–187. Association for Computational Linguistics, 2010.
[Chen et al., 2018] Ying Chen, Wenjun Hou, Xiyao Cheng, and Shoushan Li. Joint learning for emotion classification and emotion cause detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 646–651, 2018.
[Cheng et al., 2017] Xiyao Cheng, Ying Chen, Bixiao Cheng, Shoushan Li, and Guodong Zhou. An emotion cause corpus for chinese microblogs with multipleuser structures. ACM Transactions on Asian and LowResource Language Information Processing (TALLIP), 17(1):6, 2017.
[Dozat and Manning, 2016] Timothy Dozat and Christo- pher D Manning. Deep biaffine attention for neural dependency parsing. arXiv preprint arXiv:1611.01734, 2016.
[Gui et al., 2014] Lin Gui, Li Yuan, Ruifeng Xu, Bin Liu, Qin Lu, and Yu Zhou. Emotion cause detection with linguistic construction in chinese weibo text. In Natural Language Processing and Chinese Computing, pages 457– 464. Springer, 2014.
[Gui et al., 2016] Lin Gui, Dongyin Wu, Ruifeng Xu, Qin Lu, and Yu Zhou. Event-driven emotion cause extraction with corpus construction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1639–1649, 2016.
[Gui et al., 2017] Lin Gui, Jiannan Hu, Yulan He, Ruifeng Xu, Lu Qin, and Jiachen Du. A question answering approach for emotion cause extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1593–1602, 2017.
[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, 2014.
[Lafferty et al., 2001] John D Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields:
Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 282–289. Morgan Kaufmann Publishers Inc., 2001.
[Lee et al., 2010] Sophia Yat Mei Lee, Ying Chen, and ChuRen Huang. A text-driven rule-based system for emotion cause detection. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 45–53. Association for Computational Linguistics, 2010.
[Lee et al., 2013] Sophia Yat Mei Lee, Ying Chen, Chu-Ren Huang, and Shoushan Li. Detecting emotion causes with a linguistic rule-based approach 1. Computational Intelligence, 29(3):390–416, 2013.
[Li et al., 2018] Xiangju Li, Kaisong Song, Shi Feng, Daling Wang, and Yifei Zhang. A co-attention neural network model for emotion cause analysis with emotional context awareness. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4752–4757, 2018.
[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.
[Schlichtkrull et al., 2018] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pages 593–607. Springer, 2018.
[Sukhbaatar et al., 2015] Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In Advances in neural information processing systems, pages 2440–2448, 2015.
[Xia and Ding, 2019] Rui Xia and Zixiang Ding. Emotion- cause pair extraction: A new task to emotion analysis in texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1003– 1012, Florence, Italy, July 2019. Association for Computational Linguistics.
[Zhang et al., 2019] Chen Zhang, Qiuchi Li, and Dawei Song. Syntax-aware aspect-level sentiment classification with proximity-weighted convolution network. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1145–1148, 2019.