Through engagement in argumentative dialogue, interlocutors present arguments with the goals of winning the debate or contributing to the joint construction of knowledge. Especially modeling the knowledge co-construction process requires understanding of both the substance of viewpoints and how the substance of an argument connects with what it is arguing against. Prior work on argumentation in the NLP community, however, has focused mainly on the first goal and has often reduced the concept of a viewpoint as a discrete side (e.g., pro vs against, or liberal vs conservative), missing more nuanced and complex details of viewpoints. In addition, while the strength of the argument and the side it represents have been addressed relatively often, the dialogical aspects of argumentation have received less attention.
To bridge the gap, we present a model that jointly considers an Opinion Holder’s (OH’s) expressed viewpoint with a challenger’s argument in order to predict if the argument succeeded in altering the OH’s view. The first component of the architecture, vulnerable region detection, aims to identify important parts in the OH’s reasoning that are key to impacting their viewpoint. The intuition behind our model is that addressing certain parts of the OH’s reasoning often has little impact in changing the OH’s view, even if the OH realizes the reasoning is flawed. On the other hand, some parts of the OH’s reasoning are more open to debate, and thus, it is reasonable for the model to learn and attend to parts that have a better chance to change an OH’s view when addressed.
The second component of the architecture, interaction encoding, aims to identify the connection between the OH’s sentences and the challenger’s sentences. Meaningful interaction in argumentation may include agreement/disagreement, topic relevance, or logical implication. Our model encodes the interaction between every pair of the OH’s and the challenger’s sentences as interaction embeddings, which are then aggregated and used for prediction. Intuitively, the interactions with the most vulnerable regions of the OH’s reasoning are most critical. Thus, in our complete model, the interaction embeddings are weighted by the vulnerability scores computed in the first component.
We evaluate our model on discussions from the Change My View forum on Reddit, where users (OHs) post their views on various issues, participate in discussion with challengers who try to change the OH’s view, and acknowledge when their views have been impacted. Particularly, we aim to answer the following questions:
• RQ1. Does the architecture of vulnerable region detection and interaction encoding help to predict changes in view?
• RQ2. Can the model identify vulnerable sentences, which are more likely to change the OH’s view when addressed? If so, what properties constitute vulnerability?
• RQ3. What kinds of interactions between arguments are captured by the model?
We use our model to predict whether a challenger’s argument has impacted the OH’s view and compare the result with several baseline models. We also present a posthoc analysis that illuminates the model’s behavior in terms of vulnerable region detection and meaningful interaction.
For the remainder of the paper, we position our work in the literature (Section 2) and examine the data (Section 3). Then we explain our model design (Section 4). Next, we describe the experiment settings (Section 5), discuss the results (Section 6), and conclude the paper (Section 7).
Argumentation theories have identified important dialogical aspects of (non-)persuasive argumentation, which motivate our attempt to model the interaction of OH’s and challenger’s arguments. Persuasive arguments build on the hearer’s accepted premises (Walton, 2008) and appeal to emotion effectively (Aristotle and Kennedy, 2007). From a challenger’s perspective, effective strategies for these factors could be derived from the OH’s background and reasoning. On the other hand, nonpersuasive arguments may commit fallacies, such as contradicting the OH’s accepted premises, diverting the discussion from the relevant and salient points suggested by the OH, failing to address the issues in question, misrepresenting the OH’s reasoning, and shifting the burden of proof to the OH by asking a question (Walton, 2008). These fallacies can be identified only when we can effectively model how the challenger argues in relation to the OH’s reasoning.
While prior work in the NLP community has studied argumentation, such as predicting debate winners (Potash and Rumshisky, 2017; Zhang et al., 2016; Wang et al., 2017; Prabhakaran et al., 2013) and winning negotiation games (Keizer et al., 2017), this paper addresses a different angle: predicting whether an argument against an OH’s reasoning will successfully impact the OH’s view. Some prior work investigates factors that underlie viewpoint changes (Tan et al., 2016; Lukin et al., 2017; Hidey et al., 2017; Wei et al., 2016), but none target our task of identifying the specific arguments that impact an OH’s view.
Changing an OH’s view depends highly on argumentation quality, which has been the focus of much prior work. Wachsmuth et al. (2017) reviewed theories of argumentation quality assessment and suggested a unified framework. Prior research has focused mainly on the presentation of an argument and some aspects in this framework without considering the OH’s reasoning. Specific examples include politeness, sentiment (Tan et al., 2016; Wei et al., 2016), grammaticality, factuality, topic-relatedness (Habernal and Gurevych, 2016b), argument structure (Nic- ulae et al., 2017), topics (Wang et al., 2017), and argumentative strategies (e.g., anecdote, testimony, statistics) (Al Khatib et al., 2017). Some of these aspects have been used as features to predict debate winners (Wang et al., 2017) and view changes (Tan et al., 2016). Habernal and Gurevych (2016a) used crowdsourcing to develop an ontology of reasons for strong/weak arguments.
The persuasiveness of an argument, however, is highly related to the OH’s reasoning and how the argument connects with it. Nonetheless, research on this relationship is quite limited in the NLP community. Existing work uses word overlap between the OH’s reasoning and an argument as a feature in predicting the OH’s viewpoint (Tan et al., 2016). Some studies examined the relationship between the OH’s personality traits and receptivity to arguments with different topics (Ding and Pan, 2016) or degrees of sentiment (Lukin et al., 2017).
The most relevant to our work is the related task by Tan et al. (2016). Their task used the same discussions from the Change My View forum as in our work and examined various stylistic features (sentiment, hedging, question marks, etc.) and word overlap features to identify discussions that impacted the OH’s view. However, our task is different from theirs in that they made predictions on
(b) Delta ratios in discussions by topic. (e.g., a discussion has a 10% ratio if 10% of the OH’s replies have a
Figure 1: Discussion characteristics by topic.
initial comments only, while we did so for all comments replied to by the OH in each discussion. Our task is more challenging because comments that come later in a discussion have a less direct connection to the original post. Another challenge is the extreme skew in class distribution in our data, whereas Tan et al. (2016) ensured a balance between the positive and negative classes.
The Change My View forum has received attention from recent studies. For example, ad hominem (attacking an arguer) arguments have been studied, along with their types and causes (Habernal et al., 2018). Another study annotated semantic types of arguments and analyzed the relationship between semantic types and a change in view (Hidey et al., 2017). Although this work did not look at the interaction between OHs and specific challengers, it provides valuable insight into persuasive arguments. Additionally, the semantic types may potentially allow our model to better model complex interaction in argumentation.
Our study is based on discussions from the Change My View (CMV) forum2 on Reddit. In this forum, users (opinion holders, OHs) post their views on a
CMV: DNA tests (especially for dogs) are bullshit. For my line of work (which is not the DNA testing), … I have NEVER seen a DNA test return that a dog is purebred, or even anywhere close to purebred. … these tests are consistently way off on their results. … My mother recently had a DNA test done showing she is 1/4 black. I believe this is also incorrect since she knows who her parents and grandparents are, and none of them are black. …
Challenger 1 I'm not sure what exactly these particular DNA tests are looking at, but they are probably analyzing either SNPs or VNTRs. There's nothing stopping a SNP from mutating at any given generation, or a VNTR from shrinking or expanding due to errors during DNA replication. … The take-home
message is that DNA testing isn't complete bullshit, but it does have limitations.
Challenger 2 Knowing your grandparents "aren't black" doesn't really rule out being 25% African American, genetically, because genes combine during
information is that race is only barely genetic. It's mostly a social construct.
Figure 2: A discussion from Change My View.
wide range of issues and invite other users (challengers) to change their expressed viewpoint. If an OH gains a new insight after reading a comment, he/she replies to that comment with a symbol and specifies the reasons behind his/her view change. DeltaBot monitors the forum and marks comments that received a
, which we will use as labels indicating whether the comment successfully changed the OH’s view.
CMV discussions provide interesting insights into how people accept new information through argumentation, as OHs participate in the discussions with the explicit goal of exposing themselves to new perspectives. In addition, the rules and moderators of this forum assure high quality discussions by requiring that OHs provide enough reasoning in the initial post and replies.
We use the CMV dataset compiled by Tan et al. (2016)3. The dataset is composed of 18,363 discussions from January 1, 2013–May 7, 2015 for training data and 2,263 discussions from May 8– September 1, 2015 for test data.
Qualitative analysis We conducted qualitative analysis to better understand the data. First, to see
Figure 3: Architecture of Attentive Interaction Model.
if there are topical effects on changes in view, we examined the frequency of view changes across different topics. We ran Latent Dirichlet Allocation (Blei et al., 2003) with 20 topics, taking each discussion as one document. We assigned each discussion the topic that has the highest standardized probability. The most discussed topics are government, gender, and everyday life (Figure 1a). As expected, the frequency of changes in view differs across topics (Figure 1b). The most malleable topics are food, computers & games, clothing, art, education, and everyday life. But even in the food domain, OHs give out a in less than 10% of their replies in most discussions.
In order to inform the design of our model, we sampled discussions not in the test set and compared comments that did and did not receive a . A common but often unsuccessful argumentation strategy is to correct detailed reasons and minor points of the OH’s reasoning—addressing those points often has little effect, regardless of the validity of the points. On the contrary, successful arguments usually catch incomplete parts in the OH’s reasoning and offer another way of looking at an issue without threatening the OH. For instance, in the discussion in Figure 2, the OH presents a negative view on DNA tests, along with his/her reasoning and experiences that justify the view. Challenger 1 addresses the OH’s general statement and provides a new fact, which received a
. On the other hand, Challenger 2 addresses the OH’s issue about race but failed to change the OH’s view.
When a comment addresses the OH’s points, its success relies on various interactions, including the newness of information, topical relatedness, and politeness. For example, Challenger 1 provides new information that is topically dissimilar to the OH’s original reasoning. In contrast, Challenger 2’s argument is relatively similar to the OH’s reasoning, as it attempts to directly correct the OH’s reasoning. These observations motivate the design of our Attentive Interaction Model, described in the next section.
Our Attentive Interaction Model predicts the probability of a comment changing the OH’s original view, , given the OH’s initial post and the comment. The architecture of the model (Figure 3) consists of detecting vulnerable regions in the OH’s post (sentences important to address to change the OH’s view), embedding the interactions between every sentence in the OH’s post and the comment, summarizing the interactions weighted by the vulnerability of OH sentences, and predicting
The main idea of our model is the architecture for capturing interactions in vulnerable regions, rather than methods for measuring specific argumentation-related features (e.g., agreement/disagreement, contraction, vulnerability, etc.). To better measure these features, we need much richer information than the dataset provides (discussion text and s). Therefore, our proposed architecture is not to replace prior work on argumentation features, but rather to complement it at a higher, architectural level that can potentially integrate various features. Moreover, our architecture serves as a lens for analyzing the vulnerability of OH posts and interactions with arguments.
Formal definition of the model (Figure 3 (A) and (B)) Denote the OH’s initial post by th sentence, and
is the number of sentences. The sentences are encoded via an RNN, yielding a hidden state for the ith sentence
dimensionality of the hidden states. Similarly, for a comment
, hidden states of the sentences
, are computed.
Vulnerable region detection (Figure 3 (A)) Given the OH’s sentences, the model computes the vulnerability of the ith sentence using a feedforward neural network). From this vulnerability, the attention weight of the sentence is calculated as
Interaction encoding (Figure 3 (C)) The model computes the interaction embedding of every pair of the OH’s ith sentence and the comment’s jth sentence,
where is the dimensionality of interaction embeddings, and h is an interaction function between two sentence embeddings. h can be a simple inner product (in which case
), a feedforward neural network, or a more complex network. Ideally, each dimension of
indicates a particular type of interaction between the pair of sentences.
Interaction summary (Figure 3 (D)) Next, for each of the OH’s sentences, the model summarizes what types of meaningful interaction occur with the comment’s sentences. That is, given all interaction embeddings for the OH’s ith sentence, , the model conducts max pooling for each dimension,
where th dimension of
. Intuitively, max pooling is to capture the existence of an interaction and its highest intensity for each of the OH’s sentences—the interaction does not have to occur in all sentences of the comment. Since we have different degrees of interest in the interactions in different parts of the OH’s post, we take the attention-weighted sum of
to obtain the final summary vector
Prediction (Figure 3 (E)) The prediction component consists of at least one feedforward neural network, which takes as input the summary vector and optionally the hidden state of the last sentence in the comment
works may be used to integrate other features as input, such as TFIDF-weighted n-grams of the comment. The outputs of the networks are concatenated and fed to the final prediction layer to compute
. Using a single network that takes different kinds of features as input does not perform well, because the features are in different spaces, and linear operations between them are probably not meaningful.
Loss The loss function is composed of binary cross-entropy loss and margin ranking loss. Assume there are total initial posts written by OHs, and the lth post has
binary cross-entropy of the lth post and its tth comment measures the similarity between the predicted
and the true
where is the true
of the comment and
is the probability predicted by our model with parameters
. Since our data is skewed to negatives, the model may overpredict
adjust this bias, we use margin ranking loss to drive the predicted probability of positives to be greater than the predicted probability of negatives to a certain margin. The margin ranking loss is defined on a pair of comments
MRL
where is a margin. Combining the two losses, our final loss is
For the expectation in the ranking loss, we consider all pairs of comments in each minibatch and take the mean of their ranking losses.
Table 1: Data statistics. (CD: cross-domain test)
Our task is to predict whether a comment would receive a , given the OH’s initial post and the comment. We formulate this task as binary prediction of
. Since our data is highly skewed, we use as our evaluation metric the AUC score (Area Under the Receiver Operating Characteristic Curve), which measures the probability of a positive instance receiving a higher probability of
than a negative instance.
5.1 Data Preprocessing
We exclude (1) DeltaBot’s comments with no content, (2) comments replaced with [deleted], (3) system messages that are included in OH posts and DeltaBot’s comments, (4) OH posts that are shorter than 100 characters, and (5) discussions where the OH post is excluded. We treat the title of an OH post as its first sentence. After this, every comment to which the OH replies is paired up with the OH’s initial post. A comment is labeled as if it received a
otherwise. Details are described in Appendix B.
The original dataset comes with training and test splits (Figure 1a). After tokenization and POS tagging with Stanford CoreNLP (Manning et al., 2014), our vocabulary is restricted to the most frequent 40,000 words from the training data. For a validation split, we randomly choose 10% of training discussions for each topic.
We train our model on the seven topics that have the highest ratios (Figure 1b). We test on the same set of topics for in-domain evaluation and on the other 13 topics for cross-domain evaluation. The main reason for choosing the most malleable topics is that these topics provide more information about people learning new perspectives, which is the focus of our paper. Some statistics of the resulting data are in Table 1.
5.2 Inputs
We use two basic types of inputs: sentence embeddings and TFIDF vectors. These basic inputs are by no means enough for our complex task, and most prior work utilizes higher-level features (politeness, sentiment, etc.) and task-specific information. Nevertheless, our experiment is limited to the basic inputs to minimize feature engineering and increase replicability, but our model is general enough to incorporate other features as well.
Sentence embeddings Our input sentences x are sentence embeddings obtained by a pretrained sentence encoder (Conneau et al., 2017) (this is different from the sentence encoder in our model). The pretrained sentence encoder is a BiLSTM with max pooling trained on the Stanford Natural Language Inference corpus (Bowman et al., 2015) for textual entailment. Sentence embeddings from this encoder, combined with logistic regression on top, showed good performance in various transfer tasks, such as entailment and caption-image retrieval (Conneau et al., 2017).
TFIDF A whole post or comment is represented as a TFIDF-weighted bag-of-words, where IDF is based on the training data. We consider the top 40,000 n-grams (n = 1, 2, 3) by term frequency.
Word Overlap Although integration of hand-crafted features is behind the scope of this paper, we test the word overlap features between a comment and the OH’s post, introduced by Tan et al. (2016), as simple proxy for the interaction. For each comment, given the set of its words C and that of the OH’s post O, these features are defined as
5.3 Model Setting Network configurations For sentence encod-
ing, Gated Recurrent Units (Cho et al., 2014) with hidden state sizes 128 or 192 are explored. For attention, a single-layer feedforward neural network (FF) with one output node is used. For interaction encoding, we explore two interaction functions: (1) the inner product of the sentence embeddings and (2) a two-layer FF with 60 hidden nodes and three output nodes with a concatenation of the sentence embeddings as input. For prediction, we explore (1) a single-layer FF with either one output node if the summary vector is the only input or 32 or 64 output nodes with ReLU activation if the hidden state of the comment’s last sentence is used as input, and optionally (2) a single-layer FF with 1 or 3 output nodes with ReLU activation for the TFIDF-weighted n-grams of the comment. The final prediction layer is a single-layer FF with one output node with sigmoid activation that takes the outputs of the two networks above and optionally the word overlap vector. The margin
for the ranking margin loss is 0.5. Optimization is performed using AdaMax with the initial learning rate 0.002, decayed by 5% every epoch. Training stops after 10 epochs if the average validation AUC score of the last 5 epochs is lower than that of the first 5 epochs; otherwise, training runs 5 more epochs. The minibatch size is 10.
Input configurations The prediction component of the model takes combinations of the inputs: MAX ((the last hidden state of the sentence encoder
weighted n-grams of the comment), and WDO (word overlap).
5.4 Baseline
The most similar prior work to ours (Tan et al., 2016) predicted whether an OH would ever give a in a discussion. The work used logistic regression with bag-of-words features. Hence, we also use logistic regression as our baseline to predict
. Simple logistic regression using TFIDF is a relatively strong baseline, as it beat more complex features in the aforementioned task.
Model configurations Different regularization methods (L1, L2), regularization strengths (), and class weights for positives (1, 2, 5) are explored. Class weights penalize false-negatives differently from false-positives, which is appropriate for the skewed data.
Input configurations The model takes combinations of the inputs: TFIDF (TFIDF-weighted n-grams of the comment), TFIDF (+OH) (concatenation of the TFIDF-weighted n-grams of the comment and the OH’s post), WDO (word overlap), and SENT (the sum of the input sentence embeddings of the comment).
Table 2 shows the test AUC scores for the baseline and our model in different input configurations. For each configuration, we chose the optimal parameters based on validation AUC scores.
RQ1. Does the architecture of vulnerable region detection and interaction encoding help to predict changes in view? Both interaction
Table 2: AUC scores. (ID: in-domain AUC (%), CD: cross-domain AUC (%), LR: logistic regression, AIM: Attention Interaction Model, (A)IM: AIM without attention.) *: p < 0.05 using the DeLong test compared to LR with TFIDF.
information learned by our model and surfacelevel n-grams in TFIDF have strong predictive power, and attending to vulnerable regions helps. The highest score is achieved by our model (AIM) with both MAX and TFIDF as input (72.0%). The performance drops if the model does not use interaction information—(A)IM with HSENT (69.6%)—or vulnerability information— (A)IM with MAX+TFIDF (69.5%).
TFIDF by itself is also a strong predictor, as logistic regression with TFIDF performs well (70.9%). There is a performance drop if TFIDF is not used in most settings. This is unsurprising because TFIDF captures some topical or stylistic information that was shown to play important roles in argumentation in prior work (Tan et al., 2016; Wei et al., 2016). Simply concatenating both comment’s and OH’s TFIDF features does not help (69.5%), most likely due to the fact that a simple logistic regression does not capture interactions between features.
When the hand-crafted word overlap features are integrated to LR, the accuracy is increased slightly, but the difference is not statistically sig-nificant compared to LR without these features nor to the best AIM configuration. These features do not help AIM (70.9%), possibly because the information is redundant, or AIM requires a more deliberate way of integrating hand-crafted features.
For cross-domain performance, logistic regres-
sion with TFIDF performs best (69.6%). Our interaction information does not transfer to unseen topics as well as TFIDF. This weakness is alleviated when our model uses TFIDF in addition to MAX, increasing the cross-domain score (from 67.5% to 69.4%). We expect that information about vulnerability would have more impact within domain than across domains because it may learn domain-specific information about which kinds of reasoning are vulnerable.
The rest of the section reports our qualitative analysis based on the best model configuration.
RQ2. Can the model identify vulnerable sentences, which are more likely to change the OH’s view when addressed? If so, what properties constitute vulnerability? Our rationale behind vulnerable region detection is that the model is able to learn to pay more attention to sentences that are more likely to change the OH’s view when addressed. If the model successfully does this, then we expect more alignment between the attention mechanism and sentences that are actually addressed by successful comments that changed the OH’s view.
To verify if our model works as designed, we randomly sampled 30 OH posts from the test set, and for each post, the first successful and unsuccessful comments. We asked a native English speaker to annotate each comment with the two most relevant sentences that it addresses in the OH post, without knowledge of how the model computes vulnerability and whether the comment is successful or not.
After this annotation, we computed the average attention weight of the two selected sentences for each comment. We ran a paired sample t-test and confirmed that the average attention weight of sentences addressed by successful comments was sig-nificantly greater than that of sentences addressed by unsuccessful comments (p < 0.05). Thus, as expected in the case where the attention works as designed, the model more often picks out the sentences that successful challengers address.
As to what the model learns as vulnerability, in most cases, the model attends to sentences that are not punctuation marks, bullet points, or irrelevant to the topic (e.g., can you cmv?). A successful example is illustrated in Figure 4. More successful and unsuccessful examples are included in Appendix C.
RQ3. What kinds of interactions between arguments are captured by the model? We first use existing argumentation theories as a lens for interpreting interaction embeddings (refer to Section 2). For this, we sampled 100 OH posts with all their comments and examined the 150 sentence pairs that have the highest value for each dimension of the interaction embedding (the dimensionality of interaction embeddings is 3 for the best performing configuration). 22% of the pairs in a dimension capture the comment asking the OH a question, which could be related to shifting the burden of proof. In addition, 23% of the top pairs in one dimension capture the comment pointing out that the OH may have missed something (e.g., you don’t know the struggles ...). This might represent the challengers’ attempt to provide premises that are missing in the OH’s reasoning.
As providing missing information plays an important role in our data, we further examine if this attempt by challengers is captured in interaction embeddings even when it is not overtly signaled (e.g., You don’t know ...). We first approximate the novelty of a challenger’s information with the topic similarity between the challenger’s sentence and the OH’s sentence, and then see if there is a correlation between topic similarity and each dimension of interaction embeddings (details are in Appendix D). As a result, we found only a small but significant correlation (Pearson’s between topic similarity with one of the three dimensions.
Admittedly, it is not trival to interpret interaction embeddings and find alignment between embedding dimensions and argumentation theories. The neural network apparently learns complex interactions that are difficult to interpret in a human sense. It is also worth noting that the top pairs contain many duplicate sentences, possibly because the interaction embeddings may capture sentence-specific information, or because some types of interaction are determined mainly by one side of a pair (e.g., disagreement is manifested mostly on the challenger’s side).
TFIDF We examine successful and unsuccessful styles reflected in TFIDF-weighted n-grams, based on their weights learned by logistic regression (top n-grams with the highest and lowest weights are in Appendix E). First, challengers are more likely to change the OH’s view when talking about themselves than mentioning the OH in their
Figure 4: Example discussion with the OH’s initial post (left), a successful comment (top right), and an unsuccessful comment (bottom right). The OH’s post is colored based on attention weights (the higher attention the brighter). Sentences with college and SAT sections (reading, writing, math) get more attention than sentences with other subjects (algebra, geometry). The successful comment addresses parts with high attention, whereas the unsuccessful comment addresses parts with low attention.
arguments. For instance, first-person pronouns (e.g., i and me) get high weights, whereas secondperson pronouns (e.g., you are and then you) get low weights. Second, different kinds of politeness seem to play roles. For example, markers of negative politeness (can and can be, as opposed to should and no) and negative face-threatening markers (thanks), are associated with receiving a . Third, asking a question to the OH (e.g., why, do you, and are you) is negatively associated with changing the OH’s view.
We presented the Attentive Interaction Model, which predicts an opinion holder (OH)’s change in view through argumentation by detecting vulnerable regions in the OH’s reasoning and modeling the interaction between the reasoning and a challenger’s argument. According to the evaluation on discussions from the Change My View forum, sentences identified by our model to be vulnerable were addressed more by successful challengers than by unsuccessful ones. The model also effectively captured interaction information so that both vulnerability and interaction information increased accuracy in predicting an OH’s change in view.
One key limitation of our model is that making a prediction based only on one comment is not ideal because we miss context information that connects successive comments. As a discussion between a challenger and the OH proceeds, the topic may digress from the initial post. In this case, detecting vulnerable regions and encoding interactions for the initial post may become irrelevant. We leave the question of how to transfer contextual information from the overall discussion as future work.
Our work is a step toward understanding how to model argumentative interactions that are aimed to enrich an interlocutor’s perspective. Understanding the process of productive argumentation would benefit both the field of computational argumentation and social applications, including cooperative work and collaborative learning.
This research was funded by the Kwanjeong Educational Foundation and NSF grants IIS-1546393, ACI-1443068, and DGE-1745016. The authors thank Amazon for their contribution of computation time through the AWS Educate program.
Khalid Al Khatib, Henning Wachsmuth, Matthias Ha- gen, and Benno Stein. 2017. Patterns of Argumentation Strategies across Topics. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, pages 1362–1368.
Aristotle and George Alexander Kennedy. 2007. On Rhetoric. A Theory of Civic Discourse. Oxford University Press, USA.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. JMLR ’03: The Journal of Machine Learning Research 3.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bah- danau, and Yoshua Bengio. 2014. On the Properties of Neural Machine Translation: Encoder– Decoder Approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. Association for Computational Linguistics, Doha, Qatar, pages 103–111.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo i c Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, pages 681–691.
Tao Ding and Shimei Pan. 2016. Personalized Empha- sis Framing for Persuasive Message Generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1432–1441.
Ivan Habernal and Iryna Gurevych. 2016a. What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in Web argumentation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1214–1223.
Ivan Habernal and Iryna Gurevych. 2016b. Which ar- gument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1589–1599.
Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, New Orleans, LA, USA.
Christopher Hidey, Elena Musi, Alyssa Hwang, Smaranda Muresan, and Kathy McKeown. 2017. Analyzing the Semantic Types of Claims and Premises in an Online Persuasive Forum. In Proceedings of the 4th Workshop on Argument Mining. Association for Computational Linguistics, Copenhagen, Denmark, pages 11–21.
Simon Keizer, Markus Guhe, Heriberto Cuayahuitl, Ioannis Efstathiou, Klaus-Peter Engelbrecht, Mihai Dobre, Alex Lascarides, and Oliver Lemon. 2017. Evaluating Persuasion Strategies and Deep Reinforcement Learning methods for Negotiation Dialogue agents. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. Association for Computational Linguistics, Valencia, Spain, pages 480–484.
Stephanie Lukin, Pranav Anand, Marilyn Walker, and Steve Whittaker. 2017. Argument Strength is in the Eye of the Beholder: Audience Effects in Persuasion. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Association for Computational Linguistics, Valencia, Spain, pages 742–753.
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In ACL (System Demonstrations). pages 55–60.
Vlad Niculae, Joonsuk Park, and Claire Cardie. 2017. Argument Mining with Structured SVMs and RNNs. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, pages 985– 995.
Peter Potash and Anna Rumshisky. 2017. Towards De- bate Automation: a Recurrent Model for Predicting Debate Winners. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, pages 2455–2465.
Vinodkumar Prabhakaran, Ajita John, and Dor e e D Seligmann. 2013. Who Had the Upper Hand? Ranking Participants of Interactions Based on Their Relative Power. In Proceedings of the Sixth International Joint Conference on Natural Language Processing. Asian Federation of Natural Language Processing, Nagoya, Japan, pages 365–373.
Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions. In Proceedings of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, pages 613–624.
Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, and Benno Stein. 2017. Computational Argumentation Quality Assessment in Natural Language. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Association for Computational Linguistics, Valencia, Spain, pages 176–187.
Douglas Walton. 2008. Informal Logic. A Pragmatic Approach. Cambridge University Press.
Lu Wang, Nick Beauchamp, Sarah Shugars, and Kechen Qin. 2017. Winning on the Merits: The Joint Effects of Content and Style on Debate Outcomes. Transactions of Association of Computational Linguistics 5:219–232.
Zhongyu Wei, Yang Liu, and Yi Li. 2016. Is This Post Persuasive? Ranking Argumentative Comments in Online Forum. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Berlin, Germany, pages 195–200.
Justine Zhang, Ravi Kumar, Sujith Ravi, and Cris- tian Danescu-Niculescu-Mizil. 2016. Conversational Flow in Oxford-style Debates. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pages 136–141.
A.1 Topics in the Data
Topics are extracted using LatentDirichletAllocation in scikit-learn v0.19.1, with the following setting:
• n components: 20
• max iter: 200
• learning method: online
• learning offset: 50
A.2 AIM
We implemented our model in PyTorch 0.3.0.
A.3 Baseline
We use LogisticRegression in scikit-learn v0.19.1, with the default settings.
A.4 TFIDF Features
TFIDF is extracted using TfidfVectorizer in scikit-learn v0.19.1, with the default setting.
In the CMV forum, DeltaBot replies to an OH’s comment with the confirmation of a with the user name to which the OH replied. For most OH replies, the (non-)existence of a
cates whether a comment to which the OH replied changed the OH’s view. However, an OH’s view is continually influenced as they participate in argumentation, and thus a
given to a comment may not necessarily be attributed to the comment itself. One example is when a comment does not receive a
when the OH reads it for the first time, but the OH comes back and gives it a
after they interact with other comments. In such cases, we may want to give a credit to the comment that actually led the OH to reconsider a previous comment and change the view.
Hence, we use the following labeling that considers the order in which OHs read comments. We treat the (non-)existence of a in an OH comment as a label for the last comment that the OH read. We reconstruct the order in which the OH reads comments as follows. We assume that when the OH writes a comment, he/she has read all prior comments in the path to that comment.
Based on this assumption, we linearize (i.e., flatten) the original tree structure of the initial post and all subsequent comments into a linear sequence S. Starting with empty S, for each of the OH’s comments in chronological order, its ancestor comments that are yet to be in S and the comment itself are appended to S. And for each of the OH’s comments, its preceding comment in S is labeled with if the OH’s comment has a
and 0 otherwise.
This ensures that the label of a comment to which the OH replied is the (non-)existence of a in the OH’s first reply. If an OH reply is not the first reply to a certain comment (as in the scenario mentioned above), or a comment to which the OH replied is missing, the (non-)existence of a
in that reply is assigned to the comment that we assume the OH read last, which is located right before the OH’s comment in the restructured sequence.
Figure 5: Successful examples of vulnerable region detection.
Figure 6: Unsuccessful examples of vulnerable region detection.
Table 3: Top n-grams with the most positive/negative weights for logistic regression.
The topic similarity between a pair of sentences is computed as the consine similarity between the topic distributions of the sentences.
ran LDA on the entire data with 100 topics, taking each post/comment as a document. We treat the top 100 words for each topic as topic words.
The second step is to compute the topic distribution of each sentence. We simply counted the frequency of occurrences of topic words for each topic, and normalized the frequencies across topics.
Lastly, we computed the cosine similarity between the topic distributions of a pair of sentences.
The n-grams that contribute most to prediction for logistic regression are shown in table 3.