Question answering (QA) has been a long-standing research problem in natural language processing (NLP) since 1960s. In this paper, we focus on a sub-type of QA tasks: answering stock related questions. Our goal is to automatically generate convincing natural language answers to stock related questions, just like the answers given by the professional stock analysts. Table 1 shows two representative StockQA examples. Although fundamentally a QA problem, StockQA is quite different from those well studied in previous research due to the following two characteristics. First, StockQA is different from traditional QA tasks in obtaining answers. In previous QA tasks, the answers are typically selected from a knowledge base (e.g., entities) (Be- rant et al. 2013; Yin et al. 2016) or question-answer pairs
Table 1: Two representative StockQA examples.
collected from web forums (e.g., sentences) (Bian et al. 2008; Cong et al. 2008). In contrast, due to the dynamic nature of stock related QA, we generate answers based on the question and the knowledge of the referred stock. In our task, the answers to a given question may vary according to the time when the questions are asked, as well as the stock referred to. Accordingly, the retrieval-based approaches cannot be applied in this task.
Second, number understanding and generation is critical in StockQA for generating reasonable answers. In previous knowledge-base QA tasks (Berant et al. 2013; Bordes et al. 2015), numerical values are either not involved or simply treated as attribute values of entities (just like other types of attribute values). For example, in answering questions about the elevation of a mountain, the numerical answer is generally selected from a knowledge base like other entities (e.g., mountain names). In StockQA, however, reasonable answers often contain numbers, which are generated by analyzing relations between numbers in question and stock numerical knowledge (e.g., the numeric value of cost price in question and close price in knowledge base), as well as text words in question and numbers in stock knowledge (e.g., the words “resistance level” in question and “high price” in knowledge base). In addition, numbers in answers generally refer to an estimated price of support level or resistance level, which cannot be generated with an arithmetic operation (e.g., subtraction or rounding) on numbers in the knowledge base, as done in (Murakami et al. 2017).
In this work, we treat the StockQA task as a natural language generation problem, and address it with a memory-augmented encoder-decoder model. The encoder summarizes the question consisting of words and numbers, and the memory stores stock numerical knowledge including open price, close price, trading volume, and other stock indicators. The decoder generates answers by attending the representations from both the encoder and memory. In response to the challenge of number processing, we propose different mechanisms for number composition and decomposition by exploiting their inside character information. Among them, a hybrid word-character model achieves the best performance by introducing two additional networks to consult character components whenever the model encounters a number.
Contributions. We make the following contributions:
1. We introduce a novel StockQA task and define the corresponding evaluation metrics.
2. We collect a large-scale real-world dataset containing 180K instances, which will be released to offer public resources for this task.
3. We present an end-to-end framework for the task, which combines advanced sub-networks for sentence generation and number processing. Experiments demonstrate the potential of the framework through both qualitative and quantitative evaluation.
4. We reveal that benefiting from the informative retrieved answers, a simple hybrid retrieval-generation model can significantly improves the diversity and informativeness of the generated answers.
Problem Formulation. The challenge of StockQA lies in the dynamic nature of stock related QA, in which the QA pairs may vary according to the time when the questions are asked, as well as the stock referred to. Hence, the system requires the ability to formulate different answers to the exact same question. We model the dynamical property as a stock knowledge base, which describes the referred stock with numeric features being fetched at the time when the question is asked. Specifically, the knowledge base KB consists of a set of stock features, which are in the form of <feature, value> pairs with the values being float numbers, as shown in Table 2. The features could be prices, volumes, or some widely used indicators. In this paper, we do not consider the non-numeric features, such as the market news. We leave the exploration of such features as a future work. The goal of StockQA is to generate a natural language answer A that could answer the question Q, given the corresponding stock knowledge base KB. Most of the questions in StockQA are “what” and “how” questions, such as asking for stock trend prediction (i.e., what-type) and action recommendation (i.e., how-type). To generate reasonable answers, it is important to capture the relations between the indicators in question and the stock features in knowledge base, as well as the relative value of numbers. For example, to answer the question “what is the support level?”, the model should first capture the pattern that “support level” generally corresponds to “low price” of a stock. Then, the generated answer is expected to contain a number, whose value is close to the low price of the stock stored in the knowledge base.
Table 2: Stock numerical knowledge base.
Table 3: Heuristics to combine the metrics. “Rel”: relevance, “Pre”: precision, “Flu”: fluency, “Inf”: informativeness.
As seen, the number to be generated cannot be calculated by an exact arithmetic operation (i.e., subtraction), as done in (Murakami et al. 2017). Instead, it requires an estimation operator to compare the value of numbers, and thus generate a number in a reasonable scale.
Evaluation. Given that the generated answer is a natural language sentence instead of a single entity, we cannot measure its quality with precision and recall, which are commonly used in factoid QA tasks. As we aim at generating answers to convince the enquirers, we use two widely-used automatic metrics in language generation tasks: diversity and informativeness. Diversity (Li et al. 2016a) measures the ratios of distinct n-grams of the answers, while informativeness measures the number of unique n-grams of the answers.
In addition, we use four more manual evaluation metrics to measure the generated answer from four different perspectives: relevance, fluency, precision, and informativeness. Among them, “precision” and “informativeness” are two specific metrics for StockQA, which set it apart from common knowledge base QA task. Precision measures whether the generated answer has factual errors. For example, if a user asks for an estimation of support level, while the answer offers a number with value being higher than the high price, we treat it as “factual error”. A reasonable and convincing answer generally contains sufficient information, including the interpretation of the recent trend, the prediction of future trend, and the action recommendations according to the user’s request and the analysis. There are two ratings for fluency (i.e., “yes” or “no”) and precision (i.e.,
Table 4: Dataset statistics.
“has factual errors” or “no factual errors”), and three ratings (i.e., “poor”, “normal”, or “good”) for both relevance and informativeness. Table 3 shows the heuristic rules of mapping the metrics to a ranking score ranging from 0 to 1 with five uniform scales. As seen, relevance is a zero-tolerance requirement for the generated answers. Above that, precision and fluency would serve as another threshold to filter out low-quality answers. Finally, informativeness is used as an indicator to identify high-quality answers. For example, generic answers, which are quite common in dialogue and QA tasks, are labelled as “poor informativeness”.
Dataset. We collected the question-answer text pairs from Chinese online stock forums.2 On these forums, users could post their questions about the stocks for free or at very low price. Factoid questions are not encouraged on these sites, as it could be easily found on search engines or financial websites. To get a relatively clean dataset, we only keep the QA pairs that contain one single involved stock. We also delete a QA pair if its answer contains a different stock other than the one mentioned in the question. Table 4 lists the statistics about the data we built. As seen, 32.3% of questions and 39.9% of answers contain numbers, which indicate number processing is a critical component of StockQA.
Knowledge Base Question Answering. The task of knowledge base question answering has a long history, which refers to constructing answers by querying a knowledge base (Bian et al. 2008; Cong et al. 2008; Berant et al. 2013; Yao, Berant, and Van Durme 2014; Bordes et al. 2015; Xu et al. 2016; Yin et al. 2016). Knowledge base contains facts expressed in various forms, such as factual sentences (Weston, Chopra, and Bordes 2015; Sukhbaatar et al. 2015), logical form (Berant et al. 2013), and relation triples (Yin et al. 2016; Miller et al. 2016). In this work, we focus on numerical knowledge base, which consists of float numbers in different scales, and thus requires a specific component on number processing. In addition, most previous work aim at retrieving answers – generally entities, from the knowledge base (Bordes et al. 2015; Weston, Chopra, and Bordes 2015; Sukhbaatar et al. 2015; Miller et al. 2016). In contrast, we generate answers in the form of natural language sentences based on the information from the knowledge base.
Data to Text Generation. In recent years, there has been a growing interest in generating text to describe structured data, such as weather forecasts(Goldberg, Driedger, and Kit- tredge 1994; Reiter et al. 2005; Belz 2007; Angeli, Liang, and Klein 2010) and image captioning(Vinyals et al. 2015). One traditional method is using hand-crafted rules(Gold- berg, Driedger, and Kittredge 1994; Dale, Geldof, and Prost 2003; Reiter et al. 2005). More recently, automatically generating text from series or structured data is widely studied by exploiting neural network based models (Vinyals et al. 2015). We take a further step on this direction by generating interpretations based on structured data as well as text (i.e., user questions).
Large-scale numerical data is commonly available in various industries, such as pharmacy, games, finance, and telecommunications. Interpreting such data is in high demand, while manual interpretation would be prohibitively expensive, both time-wise and financially. Several researchers turned to automatic generation of descriptions on numerical data with neural network models (Mei, Bansal, and Walter 2016; Murakami et al. 2017). Our work is closely related to stock market comment generation (Murakami et al. 2017), which describes the change of only the market stock prices that are generally in the same scale. In contrast, our task is more difficult since (1) the numbers (e.g., prices of different stocks) are not in the same scale, which poses difficulty to capturing relation between them; and (2) the numbers in the knowledge base cannot be used to construct the answer with copy or arithmetic operations.
Architecture As shown in Figure 1, we employ a memory-augmented encoder-decoder architecture for the StockQA task, which consists of three components:
• encoder that summarizes the question into a sequence of vector representations;
• memory that stores stock knowledge-base as an array of representation pairs;
• decoder that generates the answer word-by-word based on the representations from both the encoder and memory.
Encoder. Suppose that represents a question to answer. The encoder is a bi-directional Recurrent Neural Network (RNN) (Schuster and Paliwal 1997) to encode the question q into a sequence of hidden states
, where
is the annotation of
from the bi-directional RNN.
Memory. The memory is essentially a key-value memory network (Miller et al. 2016), which is an array of slots in the form of (key, value) pairs. The key layer K = is used for calculating the attention distribution over the memory slots, and the value layer V =
is used for encoding stock knowledge representation. Each key-value pair (
) of the stock knowledge corresponds to a distinct stock feature
,
Figure 1: Architecture of the proposed approach. At each decoding step, the vector that represents the current focus of question and the decoder state
that summarizes the previously generated words server as a query to retrieve the memory, and a value vector
that embeds the related stock knowledge is returned for generating the target word.
where and
are the name and value of the l-th stock feature respectively. In this work, we use word embedding as the key embedding matrix:
, and we discuss choice of embedding numeric feature values later.
Decoder. The decoder is another RNN to generate the answer word by word. At each step t, the decoder first selects part of the question to focus on:
where is alignment probability computed from a comparison of the previous decoder state
and the question annotation
using a question attention model (Bahdanau, Cho, and Bengio 2015). The decoder RNN state
is updated as
where is an activation function, and
is the word embedding of the previously generated word
. Given that
represents the current focus of the question and
encodes the partially generated answer, we use both of them to retrieve related records stored in the memory:
where is alignment probability calculated with another memory attention model. The probability of generating the
t-th word is computed by
where first linearly transforms its input and then applies a softmax function.
Number Encoding and Decoding StockQA is an open-vocabulary problem, since the vocabularies of numbers keep growing as the number of stocks grows and stock price changes over time. Nearly all neural generative models have used quite restricted vocabularies, crudely treating all other words the same with an “<UNK>” symbol, called Out-of-Vocabulary (OOV) words. In this task, numbers in both question-answer pair and stock knowledge base are low-frequent or even unseen in the training corpus and thus typically treated as OOV words. The root of the OOV problem lies in the complete inability of word-level neural models to sensibly assign nonzero probability to previously unseen words (i.e., numbers in this task).
Inspired by recent successes of smaller granularity on alleviating the OOV problem (Luong and Manning 2016; Sennrich, Haddow, and Birch 2016; Lee, Cho, and Hofmann 2017), we decompose numbers into characters and construct the meaning of numbers (in both question and knowledge base) compositionally from characters, as well as generate numbers (in answer) character by character. In this work, we propose two types of word-character models, which differ at how the character model is integrated into the word-level encoder-decoder framework.
Sequential Word-Character Model. A simple way is to decompose numbers into smaller granularities, such as
Figure 2: A hybrid word-character models for number generation, in which numbers are highlighted in blue color. For simplicity, we skip the memory component.
characters (Bahdanau et al. 2016; Lee, Cho, and Hofmann 2017) or subwords (Luong, Socher, and Manning 2013; Sennrich, Haddow, and Birch 2016). We split numbers at both sides into characters, and directly feed the characters to the encoder and decoder RNNs. Ideally, the encoder RNN is able to construct the meaning of numbers compositionally from characters, and the decoder RNN is able to generate numbers character by character. We use an independent character RNN to encode numbers in knowledge base and use the last hidden states as their representations.
A potential problem of this strategy is that encoder and decoder states need to embed information for (1) question answering and (2) number composition and decomposition. We hypothesize that such overloaded use of encoder and decoder states makes training the model difficult. Inspired by recent work (Rockt¨aschel, Welbl, and Riedel 2017), we employ a hybrid scheme (Luong and Manning 2016) to explicitly separate the two types of functions.
Hybrid Word-Character Model. The hybrid model introduces two additional networks to consult character components whenever the model encounters a number, as shown in Figure 2. Specifically, the character-level encoder builds representations for numbers on-the-fly from character units, while the character-level decoder generates numbers character by character based on the corresponding word-level decoder state. For example, the character encoder updates its state over ‘9’, ‘.’, ‘8’, and ‘ ’ (the boundary tag). The final hidden state will be used as the representation for the current number, which is fed to the word-level encoder RNN.
To enable character-level generation for numbers, we replace all numbers with a special symbol “<NUM>”. When the word-level model generates “<NUM>”, we use another separate RNN to generate the character sequence given the current word-level state as well as the weighted memory value vector. We expect the value vector retrieved from the memory can guide the character-level decoder to generate reasonable numbers.3 We train the hybrid model by consulting this character-level decoder to recover the correct surface form of numbers. Accordingly, the training objective is , where
and
are the likelihoods of the word-level and character-level models, respectively.
Setup
Data Processing. Since the QA pairs are in Chinese, we need to segment the sentences into words. We used two strategies: (1) standard word segmentation which is trained on Chinese Treebank – a domain highly different from the stock data; and (2) Byte Pair Encoding (BPE) (Sennrich, Haddow, and Birch 2016): we split the question and answer sentences by numbers and punctions, and subsequently apply the BPE algorithm to these fragments. The intuition behind is that there exist a large number of terminologies and collocations in the answers, which can not be captured by the standard word segmenter. We expect that the BPE algorithm can automatically mine them based on their frequent occurrences. The word segmentation splits the corpus into 2.02M question tokens and 2.49M answer tokens, while BPE splits into 1.49M question tokens and 2.10M answer tokens. We limited the source and target vocabulary size to 5000 for both strategies, which cover 98.9% and 99.2% words for word segmentation, and all tokens for BPE.
Evaluation. We randomly extracted 1000 sentence pairs as the validation set, and another 500 sentence pairs as the final test set. We used an automatic metric – 2-gram BLEU (Pa- pineni et al. 2002), to select the best trained model on the validation set, and used human evaluation on the test set to report convincing results. Specifically, we asked two annotators to label the answers using the pre-defined metrics.
Model. We used a bidirectional LSTM layer for encoding and one LSTM layer for decoding. The encoder LSTM layer consists of 128 hidden units, and the decoder LSTM layer consists of 256 hidden units. The dimensionality of word embeddings, memory key layer and value layer is 128. When character model is applicable, the dimensionality of character embeddings, encoder layer and decoder layer is 32, 64, and 128, respectively; the interpolation weight . We used Adam (Kingma and Ba 2015) with a learning rate of 0.01 and the batch size is 128.
Generative Model
Manual Evaluation Table 5 shows the manual evaluation scores on the test set. We first compare the segmentation strategies with sequential word-character model for number processing (Rows 1 vs. 2). As seen, the BPE algorithm outperforms the standard word segmenter by dividing the sentence into larger granularities, which eases the understanding of relatively “shorter” questions and thereby reduces the number of irrelevant and imprecise answers (i.e., ).
Table 5: Manual evaluation of answers generated by the proposed generative models and humans. A score of “0” denotes low-quality answers that are not relevant to the questions, while “1” denotes high-quality answers perform well in all the metrics.
Figure 3: Visualization of attention matrices. y-axis is the generated answer (i.e., “11.45 .”) and x-axes are (a) question (“Expert, what is the support level of #STOCK#?”) and (b) numeric stock knowledge.
Table 6: Performance on number generation. We divide the generated answers into two parts by whether they contain numbers (“w/ Number”) or not (“w/o Number”).
Clearly, the hybrid word-character model significantly outperforms its sequential counterpart (Rows 2 vs. 3), especially on the portions of good answers (i.e., ). We attribute this to the additional character-level neural networks, which separates the processing of numbers from encoder and decoder states. We expect that this explicit design would lead to a better generation of numbers, which we will validate later through experiments.
To make sure that the evaluation metrics are reasonable and convincing, we also ask the annotators to evaluate the answers generated by humans. As seen, 84.2% of human answers are labelled as high-quality (i.e., ), proving that the evaluation metrics are reasonable. On the other hand, there are still 15.8% of the human answers is poor in at least one metric (i.e.,
), which poses difficulty to the model training by being noisy instances. The performance gap between answers generated from the proposed model and humans (0.62 vs. 0.84), indicates large room for further improvement.
Table 7: Portions and scores of generated answers with different error types. Portion numbers are absolute percentage over all instances in the test set.
Contribution Analysis In this experiment, we investigate whether the improvement of hybrid model comes from better performance on number generation. To this end, we divide the generated answers into two parts by whether they refer to number generation. As shown in Table 8, the proposed models achieve similar performances on generating answers that do not contain numbers, although there are still considerable differences on the portions the proposed variants account for. Concerning number generation, the hybrid model outperforms its sequential counterpart in terms of both quantity and quality, which validates our belief.
Figure 3 shows an example on number generation. Given a question on support level that generally refers to the low price of a stock, the word-level decoder first selects the corresponding question context with a question attention, and subsequently retrieve the information about low price from the memory. Based on both context vectors, the character-level decoder is able to generate a float number “11.45”, which is close to the low price “11.4” as expected.
Table 8: Performance of hybrid retrieval-generation model. “Generative” denotes the best generative model in the above section.
Error Analysis From Table 5, we found that 48.8% of answers generated by our best model (Row 3) still perform poor in at least one of the four metrics (i.e., ). Table 7 lists the statistics on detailed error types. Note that the sum of portions on each error type is not be equal to 48.8%, since an answer may have more than one type of errors (e.g., poor in both relevance and informativeness). The majority of error answers comes from poor informativeness, which is mainly caused by producing generic answers – a common problem in dialogue and question answering tasks with neural network models (Sordoni et al. 2015; Vinyals and Le 2015; Serban et al. 2016; Li et al. 2016b).
Hybrid Retrieval-Generation Model We expect that the generic answer problem can be alleviated by imitating similar answers retrieved from the training corpus, which are high quality samples: they are natural sentences that exhibit no bias towards shortness or vagueness. An straightforward implementation of such idea is a hybrid retrieval-generation model, which uses an additional set of encoder and attention model to encode and select part of the retrieved answer for generating each target word.
Formally, let be the retrieved answer, the probability of generating the
th target word
is rewritten as
where is the context vector calculated by an separate attention model over the representations of retrieved answer.
We retrieved answers from the training answers based on the similarities of both the question and stock knowledge. We use Jaccard distance to compute the similarity between two question sentences. For stock similarity, we first compute the tendency of a stock by the ratios of open prices out of MovAvg5, MovAvg10, and MovAvg20. The ratios of current stock knowledge and the retrieved stock are then used to compute the similarity.
Manual Evaluation. Table 8 lists the manual evaluation scores of the hybrid retrieval-generation model. As seen, the retrieval answers share the same overall score with the generation model. However, the retrieval model performs better at the answers without numbers, which is opposite to the generative model. This is intuitive, since the numbers in the retrieved answers often conflicts with the values in the stock knowledge, thus has factual error. As expected, the hybrid retrieval-generation model combines advantages of both models, and improves the overall performance.
Table 9: Evaluation of diversity at different granularities, ranging from 1-gram to 4-gram. Higher number denotes higher diversity.
Table 10: Evaluation of informativeness at different granularities. Higher number denotes higher informativeness. “Manual” denotes manual evaluation score.
Diversity Evaluation. We analyze the diversity of our model comparing to the three baselines quantitatively and qualitatively. If the diversity for one model is better, the n-grams of the generated sentences will be significantly larger than other models. Starting from this intuition, we count the number of unique n-grams appears in each answer and normalize it with the total number of n-grams. The result is shown in Table 9. As the retrieved answers are searched from the training data, the diversity of the retrieved answers are significantly better than other models. The generation model tends to generate generic responses, which reconfirm our previous findings. By utilizing retrieved answers, the diversity of the hybrid model increases significantly, which shows the benefits of external information source.
Informativeness Evaluation. We analyze the informativeness of different models based on the size of the unique n grams, as listed in Table 10. We find that the generative model tends to produce short answers, and the hybrid model greatly alleviates this problem by considering information of retrieved answers.
We also list the results of manual evaluation on informativeness, which measures whether the answer provides useful information to humans. The three ratings for informativeness (“poor”, “normal”, “good”) corresponds to the scores (1, 2, 3). As seen, although the retrieval model offers long answers in natural sentences, the embedded information may not be always useful for humans. The hybrid model can combine advantages of both models, and improve the informativeness of the generated answers.
We present a pioneering work on the generative stock related question answering (StockQA) task. One key challenge of this task lies in understanding and generating numbers in different scales, which is the focus of this work. We have exploited several mechanisms to learn to compose and decompose numbers. Experimental results show that a hybrid word-character model can produce reasonable answers with separate components to explicitly process numbers.
Although the proposed generative model is able to generate acceptable answers with reasonable numbers, it still suffers from generic answer problem. Our study shows that a hybrid retrieval-generation model can greatly alleviates this problem, and significantly improves the diversity of informativeness of the generated answers.
[Angeli, Liang, and Klein 2010] Angeli, G.; Liang, P.; and Klein, D. 2010. A simple domain-independent probabilistic approach to generation. In EMNLP.
[Bahdanau et al. 2016] Bahdanau, D.; Chorowski, J.; Serdyuk, D.; Brakel, P.; and Bengio, Y. 2016. End-to-end attention-based large vocabulary speech recognition. In ICASSP.
[Bahdanau, Cho, and Bengio 2015] Bahdanau, D.; Cho, K.; and Bengio, Y. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
[Belz 2007] Belz, A. 2007. Probabilistic generation of weather forecast texts. In NAACL.
[Berant et al. 2013] Berant, J.; Chou, A.; Frostig, R.; and Liang, P. 2013. Semantic parsing on freebase from question-answer pairs. In EMNLP.
[Bian et al. 2008] Bian, J.; Liu, Y.; Agichtein, E.; and Zha, H. 2008. Finding the right facts in the crowd: factoid question answering over social media. In WWW.
[Bordes et al. 2015] Bordes, A.; Usunier, N.; Chopra, S.; and Weston, J. 2015. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075.
[Cong et al. 2008] Cong, G.; Wang, L.; Lin, C.-Y.; Song, Y.- I.; and Sun, Y. 2008. Finding question-answer pairs from online forums. In SIGIR.
[Dale, Geldof, and Prost 2003] Dale, R.; Geldof, S.; and Prost, J.-P. 2003. Coral: Using natural language generation for navigational assistance. In ACSC.
[Goldberg, Driedger, and Kittredge 1994] Goldberg, E.; Driedger, N.; and Kittredge, R. I. 1994. Using naturallanguage processing to produce weather forecasts. IEEE Expert 9(2):45–53.
[Kingma and Ba 2015] Kingma, D. P., and Ba, J. 2015. Adam: A method for stochastic optimization. In ICLR.
[Lee, Cho, and Hofmann 2017] Lee, J.; Cho, K.; and Hof- mann, T. 2017. Fully character-level neural machine translation without explicit segmentation. TACL.
[Li et al. 2016a] Li, J.; Galley, M.; Brockett, C.; Gao, J.; and Dolan, B. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 110–119. San Diego, California: Association for Computational Linguistics.
[Li et al. 2016b] Li, J.; Galley, M.; Brockett, C.; Jianfeng, G.; ; and Dolan, B. 2016b. A diversity-promoting objective function for neural conversation models. In NAACL.
[Luong and Manning 2016] Luong, M.-T., and Manning, C. D. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In ACL.
[Luong, Socher, and Manning 2013] Luong, T.; Socher, R.; and Manning, C. 2013. Better word representations with recursive neural networks for morphology. In CoNLL.
[Mei, Bansal, and Walter 2016] Mei, H.; Bansal, M.; and Walter, M. R. 2016. What to talk about and how? selective generation using LSTMs with coarse-to-fine alignment. In NAACL.
[Miller et al. 2016] Miller, A.; Fisch, A.; Dodge, J.; Karimi, A.-H.; Bordes, A.; and Weston, J. 2016. Key-value memory networks for directly reading documents. In EMNLP.
[Murakami et al. 2017] Murakami, S.; Watanabe, A.; Miyazawa, A.; Goshima, K.; Yanase, T.; Takamura, H.; and Miyao, Y. 2017. Learning to generate market comments from stock prices. In ACL.
[Papineni et al. 2002] Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL.
[Reiter et al. 2005] Reiter, E.; Sripada, S.; Hunter, J.; Yu, J.; and Davy, I. 2005. Choosing words in computer-generated weather forecasts. Artificial Intelligence 167(1-2):137–169.
[Rockt¨aschel, Welbl, and Riedel 2017] Rockt¨aschel, T.; Welbl, J.; and Riedel, S. 2017. Frustratingly short attention spans in neural language modeling. In ICLR 2017.
[Schuster and Paliwal 1997] Schuster, M., and Paliwal, K. K. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45(11):2673–2681.
[Sennrich, Haddow, and Birch 2016] Sennrich, R.; Haddow, B.; and Birch, A. 2016. Neural Machine Translation of Rare Words with Subword Units. In ACL.
[Serban et al. 2016] Serban, I. V.; Sordoni, A.; Bengio, Y.; Courville, A.; and Pineau, J. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI.
[Sordoni et al. 2015] Sordoni, A.; Bengio, Y.; Vahabi, H.; Li- oma, C.; Simonsen, J. G.; and Nie, J. 2015. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In CIKM.
[Sukhbaatar et al. 2015] Sukhbaatar, S.; Szlam, A.; Weston, J.; and Fergus, R. 2015. End-to-end memory networks. In NIPS.
[Vinyals and Le 2015] Vinyals, O., and Le, Q. 2015. A neu- ral conversational model. In ICML, Deep Learning Workshop.
[Vinyals et al. 2015] Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2015. Show and tell: A neural image caption generator. In CVPR.
[Weston, Chopra, and Bordes 2015] Weston, J.; Chopra, S.; and Bordes, A. 2015. Memory networks. In ICLR.
[Xu et al. 2016] Xu, K.; Reddy, S.; Feng, Y.; Huang, S.; and Zhao, D. 2016. Question answering on freebase via relation extraction and textual evidence. In ACL.
[Yao, Berant, and Van Durme 2014] Yao, X.; Berant, J.; and Van Durme, B. 2014. Freebase QA: Information extraction or semantic parsing. In ACL.
[Yin et al. 2016] Yin, J.; Jiang, X.; Lu, Z.; Shang, L.; Li, H.; and Li, X. 2016. Neural generative question answering. In IJCAI.