The last 5 years have seen proof that neural network-based word embedding models provide term representations that are a useful information source for a variety of tasks in natural language processing. In information retrieval (IR), “traditional” models remain a high baseline to beat, particularly when considering efficiency in addition to effectiveness [6]. Combining the word embedding models with the traditional IR models is therefore very attractive and several papers have attempted to improve the baseline by adding in, in a more or less ad-hoc fashion, word-embedding information. Onal et al. [10] summarized the various developments of the last half-decade in the field of neural IR and group the methods in two categories: aggregate and learn. The first one, also known as compositional distributional semantics, starts from term representations and uses some function to combine them into a document representation (a simple example is a weighted sum). The second method uses the word embedding as a first layer of another neural network to output a document representation.
The advantage of the first type of methods is that they often distill down to a linear combination (perhaps via a kernel), from which an explanation about the representation of the document is easier to induce than from the neural network layers built on top of a word embedding. Recently, the issue of explainability in IR and recommendation is generating a renewed interest [15].
In this sense, Zhang et al. [14] introduced a new model for combining high-dimensional vectors, using a mixture model of von Mises-Fisher (VMF) instead of Gaussian distributions previously suggested by Clinchant and Perronnin [3]. This is an attractive hypothesis because the Gaussian Mixture Model (GMM) works on Euclidean distance, while the mixture of von Mises-Fisher (moVMF) model works on cosine distances—the typical distance function in IR.
In the following sections, we set up to replicate the experiments described by Zhang et al. [14]. They are grouped in three sets: classification, clustering, and information retrieval, and compare “standard” embedding methods with the novel moVMF representation.
In general, we follow the experimental setup of the original paper and, for lack of space, we do not repeat here many details, if they are clearly explained there.
2.1 Datasets
All experiments are conducted on publicly available datasets and are briefly described here below.
Classification. Two subsets of the movie review dataset: (i) the subjectivity dataset (subj) [11]; and (ii) the sentence polarity dataset (sent) [12].
Clustering. The 20 Newsgroups datasetwas used in the original paper, but the concrete version was not specified. We selected the “bydate” version, because it is, according to its creators, the most commonly used in the literature. It is also the version directly load-able in scikit-learn, making it therefore more likely that the authors had used this version.
Retrieval. The TREC Robust04 collection [13].
2.2 Models
The methods used to generate vectors for terms and documents are:
TF-IDF. The basic term frequency - inverse document frequency method [5]. Implemented in the scikit-learn library.
LSI. Latent Semantic Indexing [4].
LDA. Latent Dirichlet Allocation [2].
cBoW. Word2vec [9] in the Continuous Bag-of-Word (cBow) architecture.
PV-DBOW/DM. Paragraph vector (PV) is a document embedding algorithm that builds on Word2vec. We use here both its implementations: Distributed Bag-of-Words (PV-DBOW) and Distributed Memory (PV-DM) [7].
http://qwone.com/˜jason/20Newsgroups/ https://scikit-learn.org/stable/modules/generated/sklearn.datasets.
The LSI, LDA, cBoW, and PV implementations are available in the gensim library.
Fisher Kernel (FK). The FK framework offers the option to aggregate word embeddings to obtain fixed-length representations of documents. We use Fisher vectors (FV) based on (i) a Gaussian mixture model (FV-GMM) and (ii) a mixture of von Mises-Fisher distributions (FV-moVMF) [1].
We first fit (i) a GMM and (ii) a moVMF model on previously learnt continuous word embeddings. The fixed-length representation of a document X containing T words —expressed as , where is the word vector representation of word —is then given by = [], where K is the number of mixture components. The vectors , having the dimension (d) of the word vectors , are explicitly given by [3,14]:
where are the mixture weights, ) = ) is the soft assignment of to (i) Gaussian and (ii) VMF distribution i, and = diag(), with the covariance matrix of Gaussian i. In (i), refers to the mean vector; in (ii) it indicates the mean direction and is the concentration parameter.
We implement the FK-based algorithms by ourselves, with the help of the scikit-learn library for fitting a mixture of Gaussian models and of the Spherecluster packagefor fitting a mixture of von Mises-Fisher distributions to our data. The implementation details of each algorithm are described in what follows.
Each of the following experiments is conceptually divided in three phases. First, text processing (e.g. tokenisation); second, creating a fixed-length vector representation for every document; finally, the third phase is determined by the goal to be achieved, i.e. classification, clustering, and retrieval.
For the first phase the same pre-processing is applied to all datasets. In the original paper, this phase was only briefly described as tokenisation and stop-word removal. It is not given what tokeniser, linguistic filters (stemming, lemmatisation, etc.), or stop word list were used. Knowing that the gensim library was used, we took all standard parameters (see provided code). Gensim however does not come with a pre-defined stopword list, and therefore, based on our own experience, we used the one provided in the NLTK libraryfor English.
For the second phase, transforming terms and documents to vectors, Zhang et al. [14] specify that all trained models are 50 dimensional. We have additionally experimented with dimensionality 20 (used by Clinchant and Perronnin [3] for clustering) and 100, as we hypothesized that 50 might be too low. The TF-IDF model is 5000 dimensional (i.e. only the top 5000 terms based on their tf-idf value are used), while the Fischer-Kernel models are 15 dimensional, where d = {20, 50, 100}, as just explained. In what follows, d refers to the dimensionality of LSI, LDA, cBow, and PV models.
The cBoW and PV models are trained using a default window size of 5, keeping both low and high-frequency terms, again following the setup of the original experiment. The LDA model is trained using a chunk size of 1000 documents and for a number of iterations over the corpus ranging from 20 to 100. For the FK methods, both fitting procedures (GMM and moVMF) are independently initialised 10 times and the best fitting model is kept.
For the third phase, parameters are explained in the following sections.
3.1 Classification
Logistic regression is used for classification in Zhang et al., and therefore also used here. The results of our experiments, for d = 50 and 100-dimensional feature vectors, are summarised in Table 1. For all the methods, we perform a parameter scan of the (inverse) regularisation strength of the logistic regression classifier, as shown in Fig. 1(a) and (b). Additionally, the learning algorithms are trained for a different number of epochs and the resulting classification accuracy assessed, cf. Fig. 1(c) and (d).
Table 1: Results of classification experiments on the subj and sent datasets. Shown are the mean accuracy and standard deviation, under 10-fold cross-validation, for optimally chosen hyperparameters (i.e. top values in Figure 1).
Fig. 1(a) indicates that cBow, FV-GMM, FV-moVMF, and the simple TFIDF, when properly tuned, exhibit a very similar accuracy on subj—the given confidence intervals do not indeed allow us to identify a single, best model. Surprisingly, TF-IDF outperforms all the others on the sent dataset [Fig. 1(b)]. Increasing the dimensionality of the feature vectors, from d = 50 to 100, has the effect of reducing the gap between TF-IDF and the rest of the models on the sent dataset (see Table 1).
Fig. 1: Results of classification experiments, for 50-dimensional feature vectors, on the subj dataset [top panels, (a) and (c)] and sent dataset [bottom panels, (b) and (d)]. LSI and LDA achieve low accuracy (see Table 1) and are omitted here for visibility. The left panels [(a) and (b)] show the effect of (inverse) regularisation of the logistic regression classifier on the accuracy, while the right panels [(c) and (d)] display the effect of training for the learning algorithms. The two symbols on the right axis in panels (a) and (b) indicate the best (FV-moVMF) results reported in [14].
3.2 Clustering
For clustering experiments, the obtained feature vectors are passed to the kmeans algorithm. The results of our experiments, measured in terms of Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI), are summarised in Table 2. We used both d = 20 and 50-dimensional feature vectors. Note that the evaluation of the clustering algorithms is based on the knowledge of the ground truth class assignments, available in the 20 Newsgroups dataset.
As opposed to classification, clustering experiments show a generous imbalance in performance and firmly speak in favour of PV-DBOW. Interestingly, TF-IDF, FV-GMM, and FV-moVMF, all providing high-dimensional document representations, have a low clustering effectiveness.
3.3 Document Retrieval
For these experiments, we extracted from every document of the test collection all the raw text, and preprocessed it as described in the beginning of this section. The documents were indexed and retrieved for BM25 with the Lucene 8.2 search
Table 2: Results of clustering experiments (mean performance and standard deviation over 10 runs) in terms of Adjusted Rand Index (ARI) and Normalised Mutual Information (NMI).
engine. We experimented with three topic processing ways: (1) title only, (2) description only, and (3) title and description. The third way produces the best results and closest to the ones reported by Zhang et. al [14], and hence are the only ones reported here.
An important aspect of BM25 is the fact that the variation of its parameters kand b could bring significant improvement in performance, as reported by Lipani et. al [8]. Therefore, we performed a parameter scan for [0, 3] and [0, 1] with a 0.05 step size for both parameters. For every TREC topic, the scores of the top 1000 documents retrieved from BM25 were normalised to [0,1] with the min-max normalisation method, and were used in calculating the scores of the documents for the combined models [14].
The original results, those of our replication experiments with standard (= 1.2 and b = 0.75) and best BM25 parameter values—measured in terms of Mean Average Precision (MAP) and Precision at 20 (P@20)—are outlined in Table 3.
Table 3: Results of retrieval experiments with 95% confidence intervals.
We replicated previously reported experiments that presented evidence that a new mixture model, based on von Mises-Fisher distributions, outperformed a series of other models in three tasks (classification, clustering, and retrieval— when combined with standard retrieval models).
Since the source code was not released in the original paper, important implementation and formulation details were omitted, and the authors never replied to our request for information, a significant effort has been devoted to reverse engineer the experiments. In general, for none of the tasks were we able to confirm the conclusions of the previous experiments: we do not have enough evidence to conclude that FV-moVMF outperforms the other methods. The situation is rather different when considering the effectiveness of these document representations for clustering purposes: we find indeed that the FV-moVMF significantly underperforms, contradicting previous conclusions. In the case of retrieval, although Zhang et. al’s proposed method (FV-moVMF) indeed boosts BM25, it does not outperform most of the other models it was compared to.
Acknowledgments Authors are partially supported by the H2020 Safe-DEED project (GA 825225).
1. A. Banerjee, I. S. Dhillon, J. Ghosh, and S. Sra. Clustering on the unit hypersphere using von mises-fisher distributions. J. Mach. Learn. Res., 6:1345–1382, December 2005.
2. D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993–1022, March 2003.
3. St´ephane Clinchant and Florent Perronnin. Aggregating continuous word embed- dings for information retrieval. In Proceedings of the workshop on continuous vector space models and their compositionality, pages 100–109, 2013.
4. S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–407, 1990.
5. Z. S. Harris. Distributional structure. Word, 10(2-3):146–162, 1954.
6. S. Hofst¨atter and A. Hanbury. Let’s measure run time! extending the IR replicability infrastructure to include performance aspects. In Proceedings of the Open-Source IR Replicability Challenge co-located with 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, OSIRRC@SIGIR 2019, Paris, France, July 25, 2019., pages 12–16, 2019.
7. Quoc Le and Tomas Mikolov. Distributed representations of sentences and docu- ments. In International conference on machine learning, pages 1188–1196, 2014.
8. Aldo Lipani, Mihai Lupu, Allan Hanbury, and Akiko Aizawa. Verboseness fission for bm25 document length normalization. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval, pages 385–388. ACM, 2015.
9. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space, 2013.
10. Kezban Dilek Onal, Ye Zhang, Ismail Sengor Altingovde, Md Mustafizur Rahman, Pinar Karagoz, Alex Braylan, Brandon Dang, Heng-Lu Chang, Henna Kim, Quinten McNamara, Aaron Angert, Edward Banner, Vivek Khetan, Tyler McDonnell, An Thanh Nguyen, Dan Xu, Byron C. Wallace, Maarten de Rijke, and Matthew Lease. Neural information retrieval: at the end of the early years. Information Retrieval Journal, 21(2):111–182, Jun 2018.
11. B. Pang and L. Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271–278, 2004.
12. B. Pang and L. Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 115–
124. Association for Computational Linguistics, 2005.
13. Ellen M. Voorhees. The trec robust retrieval track. SIGIR Forum, 39(1):11–20, June 2005.
14. R. Zhang, J. Guo, Y. Lan, J. Xu, and X. Cheng. Aggregating neural word embed- dings for document representation. In Gabriella Pasi, Benjamin Piwowarski, Leif Azzopardi, and Allan Hanbury, editors, Advances in Information Retrieval, pages 303–315. Springer International Publishing, 2018.
15. Yongfeng Zhang, Yi Zhang, Min Zhang, and Chirag Shah. EARS 2019: The 2nd international workshop on explainable recommendation and search. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019., pages 1438–1440, 2019.