Let’s describe the three images in Figure 1a. Most of you will speak sentences varying vastly from image to image. In fact, the ability of using diverse language to describe the colorful visual world is a gift to humans, but a formidable challenge to machines. Although recent advances in visual representation learning [12, 35, 11] and language modeling [13, 40] demonstrate the impressive power of modeling the diversity in their respective modalities, it is still far from
Figure 1: The motivation of the proposed learning to Collocate Neural Modules (CNM) for image captioning: neural module collocation imitates the inductive bias — sentence pattern, which regularizes the diverse training effectively.
being resolved to establish a robust cross-modal connection between them. Indeed, image captioning is not the only model that can easily exploit the dataset bias to captioning even without looking at the image, almost all existing models for vision-language tasks such as visual Q&A [18, 8, 37] have been spotted mode collapse to certain dataset idiosyncrasies, failed to reproduce the diversity of our world — the more complex the task is, the more severe the collapse will be, such as image paragraph generation [22] and visual dialog [5]. For example, in MS-COCO [27] training set, as the co-occurrence chance of “man” and “standing” is 11% large, a state-of-the-art captioner [2] is very likely to genera-
Figure 2: By comparing our CNM with a non-module baseline (an upgraded version of Up-Down [2]), we have three interesting findings in tackling the dataset bias: (a) more accurate grammar. % denotes the frequency of a certain pattern in MS-COCO, (b) more descriptive attributes, and (c) more accurate object interactions. The ratio ./. denotes the percentage of co-occurrence, e.g“sheep”. We can see that CNM outperforms the baseline even with highly biased training samples.
tion “man standing”, regardless of their actual relationships such as “milking”, which is 0.023% rare. We will discuss more biased examples in Figure 2 later.
Alas, unlike a visual concept in ImageNet which has 650 training images on average [6], a specific sentence in MSCOCO has only one single image [27], which is extremely scarce in the conventional view of supervised training. However, it is more than enough for us humans — anyone with normal vision (analogous to pre-trained CNN encoder) and language skills (analogous to pre-trained language decoder) does NOT need any training samples to perform captioning. Therefore, even though substantial progress has been made in the past 5 years since Show&Tell [42], there is still a crucial step missing between vision and language in modern image captioners [2, 30, 31]. To see this, given a sentence pattern in Figure 1b, your descriptions for the three images in Figure 1a should be much more constrained. In fact, studies in cognitive science [10, 38] show that do us humans not speak an entire sentence word by word from scratch; instead, we compose a pattern first, then fill in the pattern with concepts, and we repeat this process until the whole sentence is finished. Thus, structuring such patterns is what our human “captioning system” practices every day, and should machines do so. Fortunately, as we expected, for the sentence pattern in Figure 1b, besides those three captions, we have thousands more in MS-COCO.
In this paper, we propose learning to Collocate Neural Modules (CNM) to fill the missing gap in image captioning, where the module collocation imitates the sentence pattern in language generation. As shown in Figure 1c, CNM first uses the FUNCTION module for generating function word “a”, and then chooses the ATTRIBUTE module to describe the adjectives like “black” of the “cat”, which will be generated by the OBJECT module for nouns, followed by RELA- TION module for verbs or relationships like “sits in”. Therefore, the key of CNM is to learn a dynamic structure that is an inductive bias being faithful to language collocations.
Though using neural module networks is not new in vision-language tasks such as VQA [3], where the question is parsed into a module structure like COLOR(FIND(‘chair’)) for “What color is the chair?”; for image captioning, the case is more challenging as only partially observed sentences are available during captioning, and the module structure by parsing is no longer applicable. To this end, we develop the following techniques for effective and robust CNM training. 1) Inspired by the policy network design in partially observed environment reinforcement learning [7], at each generation time step, the output of the four modules will be the fused according to their soft attention, which is based on the current generation context. 2) We adopt multi-step reasoning, i.e., stacking neural modules [14]. These two methods stabilize the CNM training greatly. 3) To further introduce expert knowledge, we impose a linguistic loss for the module soft attention, which should be faithful to part-of-speech collocations, e.g., AT- TRIBUTE module should generate words that are ADJ.
Before we delve into the technical details in Section 3, we would like to showcase the power of CNM in tackling the dataset bias in Figure 2. Compared to a strong non-module baseline [2], the observed benefits of CNM include: 1) more accurate grammar like less ‘a/an’ error and ‘singular/plural’ error (Figure 2a), thanks to the joint reasoning of FUNCTION and OBJECT module, 2) more descriptive attributes (Figure 2b) due to ATTRIBUTE module, and 3) more accurate interactions (Figure 2c) due to RELATION module. Moreover, we find that when only 1 training sentence of each image is provided, our CNM will suffer less performance deterioration compared with the strong baseline. Extensive discussions and human evaluations are offered in Section 4.2, where we validate the effectiveness of CNM on the challenging MS-COCO image captioning benchmark. Overall, we achieve 127.9 CIDEr-D score on Karpathy split and a single model 126.0 c40 on the official server.
Our contributions are summarized as follows:
• Our CNM is the first module networks for image captioning. This enriches the spectrum of using neural modules for vision-language tasks.
• We develop several techniques for effective module collocation training in partially observed sentences.
• Experiment results show that significant improvement can be made by using neural modules. CNM is a generic framework that supports potential improvement like more principled module and controller designs.
Image Captioning. Most early image captioners are template-based models that they first structure sentence patterns and then fill the words into these fixed patterns [24, 25, 32]. However, since the functions used for generating templates and for generating words are not jointly trained, the performances are limited. Compared with them, modern image captioners which achieve superior performances are attention based encoder-decoder methods [43, 42, 36, 28, 30, 2, 45, 31]. However, unlike the template based models, most of the encoder-decoder based models generate word one by one without structure. Our CNM makes full use of the advantages of both template and encoder-decoder based image captioners which can generate captions by structuring patterns and end-to-end training. In particular, from the perspective of module network, several recent works can be reduced to a special case of our CNM. For example, Up-Down [2] only adopts OBJECT module, and NBT [31] only uses OBJECT and FUNCTION modules while they treat all the non-object words as function words.
Neural Module Networks. Recently, the idea of decomposing the network into neural modules is popular in some vision-language tasks such as VQA [3, 15], visual grounding [29, 46], and visual reasoning [37]. In these tasks, highquality module layout can be obtained by parsing the provided sentences like questions in VQA. Yet in image captioning, only partially observed sentences are available and the module structure by parsing is not applicable anymore. For addressing such a challenge, we propose to dynamically collocate neural modules on-the-fly during sentence generation.
Figure 3 shows the encoder-decoder structure of our learning to Collocate Neural Modules (CNM) model. The encoder contains a CNN and four neural modules to generate features for language decoding (cf. Section 3.1). Our decoder has a module controller that softly fuses these features into a single feature for further language decoding by the followed RNN (cf. Section 3.2.1). Note that a linguistic loss is imposed for making the module controller more faithful to part-of-speech collocations (cf. Section 3.2.3). Besides the language generation, the RNN would also output the accumulated context of the partially observed sentence as the
Figure 3: The encoder-decoder pipeline of our learning to Collocate Neural Modules (CNM) image captioner. The dash lines from RNN to FUNCTION module and the module controller mean that both of these sub-networks require the contextual knowledge of partially observed sentences.
input to FUNCTION module and controller for linguistic information, which is helpful for these grammar-related modules. For multi-step reasoning, the entire decoder of CNM will repeat this soft fusion and language decoding M times (cf. Section 3.2.2). The residual connections are also implemented for directly transferring knowledge from lower layers to higher ones.
3.1. Neural Modules
Four neural modules are designed for predicting the orthogonal knowledge from the image, e.g., OBJECT module focuses on the object categories while ATTRIBUTE module focuses on the visual attributes. In this way, the caption generation can be disentangled from dataset bias at the wordlevel, i.e., the words are generated from the visual knowledge from each module, not merely from the language context which is more likely overfitted to dataset bias. For example, the more accurate description “bird-perch-tree” will be reduced to “bird-fly” without using RELATION module, due to the high co-occurrence of “bird” and “fly” in the dataset. Now, we detail each of the modules.
OBJECT Module. It is designed to transform the CNN features to a feature set containing the knowledge on object categories, i.e., the feature set
facilitates the prediction of nouns like “person” or “dog”. The input of this module is
, which is an
feature set of N RoI features extracted by a ResNet-101 Faster R-CNN [35]. This ResNet is pre-trained on object detection task by using the object annotations of VG dataset [23]. Formally, this module can be formulated as:
where is the
output feature set. ATTRIBUTE Module. It is designed to transform the CNN features to a feature set
on attribute knowledge, for generating adjectives like “black” and “dirty”. The input of this module is an
feature set extracted by a ResNet-101 Faster R-CNN, and the network used here is pre-trained on attribute classification task by using the attribute annotations of VG dataset. Formally, this module can be written as:
where is the
feature set output from this module. RELATION Module. It transforms the CNN features to a feature set
representing potential interactions between two objects. This transferred feature set
would help to generate verbs like “ride”, prepositions like “on”, or quanti-fiers like “two”. This module is built based on the multi-head self-attention mechanism [40], which automatically seeks the interactions among the input features. Here, we use
in Eq. (1) as the input because these kinds of features are widely applied as the input for successful relationship detection [48, 47]. This module is formulated as:
where MultiHead() means the multi-head self-attention mechanism, MLP(
) is a feed-forward network containing two fully connected layers with a ReLU activation layer in between [40], and
is the
feature set output from this module. Specifically, we use the following steps to compute the multi-head self-attention. We first use scaled dot-product to compute k self-attention head matrices as:
where are all
trainable matrices,
is the dimension of each head vector, and k is the number of head matrices. Then these k heads are concatenated and linearly projected to the final feature set M:
where is a
trainable matrix, M is the
feature set.
FUNCTION Module. It is designed to produce a single feature for generating function words like “a” or “and”. The input of this module is a
dimensional context vector c provided by the RNN, as the dashed line drawn in Figure 3. We use c as the input because it contains rich language context knowledge of the partially generated captions, and such knowledge is suitable for generating function words, like “a” or “and”, which require few visual
Figure 4: The detailed structure of our module controller. This controller will generate four soft weights by an LSTM for softly fusing attended features of four modules into a single fused feature .
knowledge. This module is formulated as:
where is the
dimensional output feature.
3.2. Controller
Figure 4 shows the detailed design of the module controller, which contains three attention networks, and one LSTM for soft weights generation. The output of this controller is a single fused feature vector which would be used for the next step reasoning by the followed RNN as in Figure 3. Next, we describe our module controller.
3.2.1 Soft Fusion
Yet, it is still an open question on how to define a complete set of neural modules for visual reasoning [46, 3]. However, we believe that a combination of simple neural modules can approximate to accomplish a variety of complex tasks [14]. Before the soft fusion, three additive attention networks are used to respectively transform feature sets output from three visual modules into three more informative features:
where , and
are the
dimensional transformed features of
, and
produced by three visual modules (cf. Section 3.1), respectively; h is the
dimensional query vector produced by an LSTM (specified in Section 3.3); and the three attention networks own the same structure as that in [2] while the parameters are not shared.
After getting the three transformed features, , and
from Eq. (7) and the output
from FUNCTION module, the controller generates four soft weights for them. The process of generating soft weights is formulated as:
where the input x is the concatenation of three visual embedding vectors and the context vector accumulated in the RNN used in Eq.(6); is a fourdimensional soft attention vector; and the output vector
will be fed into the RNN for the subsequent language decoding.
We use x for generating soft weights because both visual clues () and the language context knowledge c of partially generated captions are all indispensable for achieving satisfied module collocation. Also, since the layouts of modules at a new time step are highly related to the previous ones, an LSTM is applied here to accumulate such knowledge for generating new soft weights.
3.2.2 Multi-Step Reasoning
Different from many sentence-provided visual tasks like VQA where approximately perfect module layout can be parsed by the fully observed sentences, our module layout is still noisy because only partially observed sentences are available. To robustify the visual reasoning, we repeat the soft fusion and language decoding M times as in [40, 34, 20]. In this way, the generated captions are usually more relevant to the images by observing more visual clues. For example, as the experiment results shown in Section 4.2, when multi-step reasoning is implemented, more accurate quantifiers are generated because the visual patterns of the objects with the same category can be accumulated. In addition, residual connections (cf. Figure 3) are used for directly transferring knowledge from lower layers to higher ones when such knowledge is already sufficient for word generation.
3.2.3 Linguistic Loss
For ensuring each module to learn the orthogonal and nontrival knowledge from the image, e.g., OBJECT module focuses more on object categories instead of visual attributes, even it owns the same structure as ATTRIBUTE module. We design a linguistic loss which is imposed on the module controller for regularizing the training by making the controller faithful to human expert knowledge on part-of-speech collocation.
We build this loss by extracting the words’ lexical categories (e.g., adjectives, nouns, or verbs) from ground-truth captions by the Part-Of-Speech Tagger tool [39]. According to these lexical categories, we assign each word a 4-dimensional one hot vector , indicating which module should be chosen for generating this word. In particular, we assign OBJECT module to nouns (NN like “bus”), ATTRIBUTE module to adjectives (ADJ like “green”), RE- LATION module to verbs (VB like “drive”), prepositions (PREP like “on”) and quantifiers (CD like “three”), and FUNCTION module to the other words (CC like “and”).
By providing these expert-guided module layout , the cross-entropy value between
and soft weights w in Eq.(8) is imposed to train the module controller:
Note that this linguistic loss is imposed on all the M module controllers in the language decoder (cf. Section 3.2.2).
3.3. Training and Inference
By assembling the neural modules, module controller, ResNet-101 [12] as CNN, and the top-down LSTM [2] as RNN, our CNM image captioner can be trained end-to-end. More specifically, at time step t, the query vector h in Eq. (7) is the output of the first LSTM of the top-down structure at the same time step, and the context vector c in Eq. (6) and Eq. (8) is the output of the second LSTM of the top-down structure at time step .
Given a ground-truth caption with its ex-tracted part-of-speech tags
, we can end-to-end train our CNM by minimizing the linguistic loss proposed in Eq. (9) and the language loss between the generated captions and the ground-truth captions. Suppose that the probability of word s predicted by the language decoder of our CNM model is P(s), we can define the language loss
as the cross-entropy loss:
or the negative reinforcement learning (RL) based reward [36]:
where r is a sentence-level metric for the sampled sentence and the ground-truth
, e.g., the CIDEr-D [41] metric. Given the linguistic loss and language loss, the total loss is:
where is a trade-off weight. When inference in language generation, we adopt the beam search strategy [36] with a beam size of 5.
4.1. Datasets, Settings, and Metrics
MS-COCO [27]. This dataset provides one official split: 82,783, 40,504 and 40,775 images for training, validation and test respectively. The 3rd-party Karpathy split [19] was also used for the off-line test, which has 113,287, 5,000, 5,000 images for training, validation and test respectively. Visual Genome [23] (VG). We followed Up-Down [2] to use object and attribute annotations provided by this dataset to pre-train CNN. We filtered this noisy dataset by keeping the labels which appear more than 2, 000 times in the training set. After filtering, 305 objects and 103 attributes remain. Importantly, since some images co-exist in both VG and COCO, we also filtered out the annotations of VG which also appear in COCO test set. Settings. The captions of COCO were addressed by the following steps: the texts were first tokenized on white spaces; all the letters were changed to lowercase; the words were removed if they appear less than 5 times; each caption was trimmed to a maximum of 16 words. At last, the vocabulary included totally 10, 369 words.
In Eq. (1), and
were set to 2,048 and 1,000 respectively; and in Eq. (6),
was set to 1,000. The number of head vectors k in Eq. (5) was 8. At training time, Adam optimizer [21] was used and the learning rate was initialized to
and was decayed by 0.8 for every 5 epochs. The cross-entropy loss Eq. (10) and the RL-based loss Eq. (11) were in turn used to train our CNM 35 epochs and 100 epochs respectively. The batch size was set to 100. In our experiments, we found that the performance is non-sensitive to
in Eq. (12). By default, we set the trade-off weight
and
when the cross-entropy loss and RL-based loss were used as language loss, respectively. Metrics. Five standard metrics were applied for evaluating the performances of the proposed method: CIDEr-D [41], BLEU [33], METEOR[4], ROUGE [26], and SPICE [1].
4.2. Ablative Studies
We conducted extensive ablations for CNM, including architecture and fewer training sentences.
Architecture. We will investigate the effectiveness of designed modules, soft module fusion, linguistic loss, and deeper decoder structure in terms of proposing research questions (Q) and empirical answers (A).
Q1: Will each module generate more accurate module-specific words, e.g., will OBJECT module generate more accurate nouns? We deployed a single visual module as the encoder and the top-down attention LSTM [2] as the decoder. When OBJECT, ATTRIBUTE, and RELATION modules were used, the baselines are denoted as Module/O, Module/A, and Module/R, respectively. In particular, baseline Module/O is the upgraded version of Up-Down [2]. Q2: Will the qualities of the generated captions be improved when the modules are fused? We designed three strategies for fusing modules by using three kinds of fusion weights. Specifically, when we set all the fusion weights as 1, the baseline is called Col/1; when soft fusion weights were used, the baseline is called Col/S; and when GumbelSoftmax layer [16] was used for hard selection, the baseline is called Col/H.
Q3: Will the expert knowledge of part-of-speech collocations provided by the linguistic loss benefit the model? We added the linguistic loss to baselines Col/H and Col/S to get baselines Col/S+L and Col/H+L, respectively. Noteworthy, linguistic loss can not be used to Col/1 since we do not need module controller here.
Q4: Will better captions be generated when a deeper language decoder is implemented? We stacked the language decoder of baseline Col/S+L M times to get baseline CNM#M. Also, we designed Module/O#M by stacking M times of the top-down LSTM of baseline Module/O to check whether the performances can be improved when only the deeper decoder is used.
Evaluation Metrics. For comprehensively validating the effectiveness of our CNM, we not only computed five standard metrics (cf. Section 4.1), but also conducted human evaluation and calculated the recalls of five part-of-speech words. Specifically, we invited 20 workers for human evaluation. We exhibited 100 images sampled from the test set for each worker and asked them to pairwisely compare the captions generated from three models: Module/O, Col/S+L, and CNM#3. The captions are compared from two aspects: 1) the fluency and descriptiveness of the generated captions (the top three pie charts in Figure 5); 2) the relevance of the generated captions to images (the bottom three pie charts in Figure 5). For calculating the recalls of five part-of-speech words, we counted the ratio of the words in predicted captions to the words in ground-truth captions. Such results are reported in Table 2.
A1. From Table 2, we can observe that each single module prefers to generate more accurate module-specific words, e.g., the recall of nouns generated by Module/O is much higher than Module/A. Such observation validates that each module can indeed learn the knowledge of the corresponding module-specific words.
A2. As shown in Table 1, when modules are fused, the performances can be improved. Also, by comparing Col/1, Col/S, and Col/H, we can find that Col/S achieves the highest performance. This is reasonable since compared with Col/1, Col/S can make word generation ground to the spe-cific module. Compared with Col/H, Col/S can exploit more knowledge from all the modules when the modules are not correctly collocated.
A3. As shown in Table 1 and 2, we can find that the performances of Col/S+L are better than Col/S. Such observations
Table 1: The performances of various methods on Karpathy split. The metrics: B@N, M, R, C, and S denote BLEU@N, METEOR, ROUGE-L, CIDEr-D, and SPICE, respectively.
Figure 5: The pie charts each comparing the two methods in human evaluation.
Table 2: The recalls (%) of five part-of-speech words.
validate that the expert supervision can indeed benefit the caption generation. In addition, from the results shown in Figure 5, we can find that when soft module fusion and linguistic loss are deployed, the generated captions has higher qualities evaluated by humans.
A4. By inspecting the standard evaluation scores in Table 1,
Table 3: The CIDEr-D loss (CIDEr-D) of using fewer train- ing sentences.
Figure 6: The visualizations of the caption generation process of two methods: CNM#3 and Module/O. Different colours refer to different modules, i.e., red for OBJECT module, purple for RELATION module, and black for FUNCTION module. For simplicity, we only visualize the module layout generated by the last module controller of the deeper decoder.
the recalls of words in Table 2, and the human evaluations in Figure 5, we can find that when a deeper decoder is used, e.g., CNM#3 vs. CNM#1, the qualities of the generated captions can be improved. Also, by comparing Module/O#3 with CNM#3, we can find that only using a deeper decoder is not enough for generating high qualities captions.
Fewer Training Samples. To test the robustness of our CNM in the situation where only fewer training sentences are available (cf. Section 1), we randomly assigned X sentences among all the annotated captions to one image for training models CNM#3 and Module-O to get baselines CNM&X and Module-O&X. The results are reported in Table 3, where the values mean the losses of CIDEr-D compared with the model trained by all sentences, and the values in the bracket are the CIDEr-D scores.
Results and Analysis. From Table 3, we can find that both two models will be damaged if fewer training sentences are provided. Interestingly, we can observe that our CNM can halve the performance loss compared to Module/O. Such observations suggest that our CNM is more robust when fewer training samples are provided, compared with the traditional attention-based method.
4.3. Comparisons with State-of-The-Arts
Comparing Methods. Though various captioning models are developed in recent years, for fair comparisons, we only compared our CNM with some encoder-decoder methods due to their superior performances. Specifically, we compared our method with SCST [36], StackCap [9], Up-Down [2], LSTM-A [45], NBT [31], CAVP [28], RFNet [17], and SGAE [44]. Among these methods, Up-
Table 4: The performances of various methods on MS- COCO Karpathy split trained by cross-entropy loss.
Table 5: The performances of various methods on the online MS-COCO test server.
Down and NBT are specific cases of our CNM where only OBJECT modules are deployed. All of StackCap, CAVP, and RFNet use wider encoders or deeper decoders, while they do not design different modules. In addition, we also equipped our CNM a dictionary preserving language bias as in SGAE [44], and this model is denoted as CNM+SGAE. Results. Table 4 and 1 show the performances of various methods trained by cross-entropy loss and RL-based loss, respectively. We can see that our single model CNM+SGAE in Table 1 achieves a new state-of-the-art CIDEr-D score. Specifically, by deploying four compact modules, soft module fusion strategy, and linguistic loss, our CNM can obviously outperform the models, e.g., StackCap, CAVP, and RFNet, which also use deeper decoders or wider encoders. When the dictionary preserving language bias is learned as in SGAE, even the query embeddings do not contain high-level semantic knowledge created by graph convolution network as SGAE, our CNM+SGAE also achieve better performances than SGAE. From the results of the online test reported in Table 5, we can find that our single model has competitive performances and can achieve the highest CIDEr-D c40 score. In addition, Figure 6 shows the visualizations of the captioning process of our CNM and Module/O (the upgraded version of Up-Down). From this figure, we can observe that our CNM can generate more relevant description “bird perch” and less overfitted to dataset bias of high co-occurrence word combination “bird fly”.
4.4. Limitations and Potentials
Though we design three techniques, e.g., soft module fusion, linguistic loss, and multi-step reasoning for robustifying the module collocation, improper module collocations
Figure 7: The limitations of our CNM model.
still exist since the sentence patterns are structured dynamically without a global “oracle”. As a result, inaccurate description will be generated by given the improper module collocations. For example, as shown in Figure 7a top, at time step 4, RELATION module is chosen inaccurately and the verb “standing” is generated, while two cows have different actions; in Figure 7a bottom, at time step 3, it is more suitable to generate the noun “toilet”, but FUNCTION module is chosen and inaccurate description “white and black cat” is generated. For tackling this limitation, more advanced techniques like Reinforcement Learning could be exploited for guiding the module collocations.
Another limitation of our CNM is insufficient commonsense reasoning. Specifically, many adjectives which require commonsense reasoning can hardly be generated by our model, e.g., “rural”, “rusty”, or “narrow” are all commonsense adjectives. Figure 7b gives two examples, where the words “rusted” and “rural” cannot be generated. One possible solution is to design a REASON module where a memory network preserving the commonsense knowledge is exploited and then the context knowledge can be used as queries for reasoning. The model CNM+SGAE is one preliminary experiment designed for resolving such limitation. From Table 1, we can see that the performance indeed improves. This may shed some light on using more sophisticated modules and commonsense reasoning strategies.
We proposed to imitate the humans inductive bias — sentences are composed by structuring patterns first — for image captioning. In particular, we presented a novel modular network method: learning to Collocate Neural Modules (CNM), which can generate captions by filling the contents into collocated modules. In this way, the caption generation is expected to be disentangled from dataset bias. We validated our CNM by extensive ablations and comparisons with state-of-the-art models on MS-COCO. In addition, we discussed the model limitations and thus the corresponding potentials are our future work.
[1] P. Anderson, B. Fernando, M. Johnson, and S. Gould. Spice: Semantic propositional image caption evaluation. In European Conference on Computer Vision, pages 382–398. Springer, 2016.
[2] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, number 5, page 6, 2018.
[3] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 39–48, 2016.
[4] S. Banerjee and A. Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
[5] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, D. Parikh, and D. Batra. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 326–335, 2017.
[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei- Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
[7] J. Foerster, I. A. Assael, N. de Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pages 2137–2145, 2016.
[8] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913, 2017.
[9] J. Gu, J. Cai, G. Wang, and T. Chen. Stack-captioning: Coarse-to-fine learning for image captioning. AAAI, 2017.
[10] J. Hale, C. Dyer, A. Kuncoro, and J. R. Brennan. Finding syntax in human encephalography with beam search. arXiv preprint arXiv:1806.04127, 2018.
[11] K. He, G. Gkioxari, P. Doll´ar, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
[12] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
[13] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
[14] R. Hu, J. Andreas, T. Darrell, and K. Saenko. Explainable neural computation via stack neural module networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 53–69, 2018.
[15] R. Hu, J. Andreas, M. Rohrbach, T. Darrell, and K. Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE In-
ternational Conference on Computer Vision, pages 804–813, 2017.
[16] E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.
[17] W. Jiang, L. Ma, Y.-G. Jiang, W. Liu, and T. Zhang. Re- current fusion network for image captioning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 499–515, 2018.
[18] J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901– 2910, 2017.
[19] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015.
[20] J.-H. Kim, J. Jun, and B.-T. Zhang. Bilinear attention net- works. In Advances in Neural Information Processing Systems, pages 1571–1581, 2018.
[21] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[22] J. Krause, J. Johnson, R. Krishna, and L. Fei-Fei. A hierar- chical approach for generating descriptive image paragraphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 317–325, 2017.
[23] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2017.
[24] G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. Baby talk: Understanding and generating image descriptions. In Proceedings of the 24th CVPR. Citeseer, 2011.
[25] P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi. Collective generation of natural image descriptions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 359–368. Association for Computational Linguistics, 2012.
[26] C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.
[27] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- manan, P. Doll´ar, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
[28] D. Liu, Z.-J. Zha, H. Zhang, Y. Zhang, and F. Wu. Context- aware visual policy network for sequence-level image captioning. arXiv preprint arXiv:1808.05864, 2018.
[29] D. Liu, H. Zhang, Z.-J. Zha, and F. Wu. Explainability by parsing: Neural module tree networks for natural language visual grounding. arXiv preprint arXiv:1812.03299, 2018.
[30] J. Lu, C. Xiong, D. Parikh, and R. Socher. Knowing when to look: Adaptive attention via a visual sentinel for image
captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 6, page 2, 2017.
[31] J. Lu, J. Yang, D. Batra, and D. Parikh. Neural baby talk. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7219–7228, 2018.
[32] M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daum´e III. Midge: Generating image descriptions from computer vision detections. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 747–756. Association for Computational Linguistics, 2012.
[33] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics, 2002.
[34] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.
[35] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015.
[36] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel. Self-critical sequence training for image captioning. In CVPR, volume 1, page 3, 2017.
[37] J. Shi, H. Zhang, and J. Li. Explainable and explicit visual reasoning over scene graphs. arXiv preprint arXiv:1812.01855, 2018.
[38] L. R. Slevc. Saying what’s on your mind: Working mem- ory effects on sentence production. Journal of experimental psychology: Learning, memory, and cognition, 37(6):1503, 2011.
[39] K. Toutanova and C. D. Manning. Enriching the knowledge sources used in a maximum entropy part-of-speech tagger. In Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics-Volume 13, pages 63–70. Association for Computational Linguistics, 2000.
[40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008, 2017.
[41] R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
[42] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In CVPR, 2015.
[43] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudi- nov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048–2057, 2015.
[44] X. Yang, K. Tang, H. Zhang, and J. Cai. Autoencoding scene graphs for image captioning. arXiv preprint arXiv:1812.02378, 2018.
[45] T. Yao, Y. Pan, Y. Li, Z. Qiu, and T. Mei. Boosting image captioning with attributes. In IEEE International Conference on Computer Vision, ICCV, pages 22–29, 2017.
[46] L. Yu, Z. Lin, X. Shen, J. Yang, X. Lu, M. Bansal, and T. L. Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1307–1315, 2018.
[47] R. Zellers, M. Yatskar, S. Thomson, and Y. Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5831–5840, 2018.
[48] H. Zhang, Z. Kyaw, S.-F. Chang, and T.-S. Chua. Visual translation embedding network for visual relation detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5532–5540, 2017.
This supplementary document will further detail the following aspects in the main paper: A. Network Architecture, B. Details of Human Evaluations, C. More Qualitative Examples.
Here, we introduce the detailed network architectures of all the components in our model, which includes four neural modules, a module controller, and decoders.
6.1. Neural Modules
In Section 3.1 of the main paper, we show how to use four neural modules to generate the orthogonal knowledge from the image. The detail structures of these four modules are respectively listed in the following tables: 1) OBJECT module in Table 6, 2) ATTRIBUTE module in Table 7, 3) RELATION module in Table 8, and 4) FUNCTION module in Table 9. In particular, the input vector c of FUNCTION module in Table 9 (1) is the output of an LSTM in the language decoder, and we will specify this context vector in Section 6.3.
6.2. Module Controller
In Section 3.2.1 of the main paper, we discuss how to use module controller to softly fuse four vectors generated by attention networks and FUNCTION module. The common structure of three attention networks used in Eq.(7) and the detail process of soft fusion in Eq.(8) are demonstrated in Table 10 and 11, respectively. Specifically, the hidden vector h in Table 10 (2) and the context vector c in Table 11 (1) are the outputs of two different LSTMs in the language decoder, and both of them will be specified in Section 6.3.
Table 6: The details of OBJECT module.
Table 7: The details of ATTRIBUTE module.
6.3. Language Decoder
As discussed in Section 3.2 of the main paper, the whole language decoder is built by stacking M single language decoders with a common structure while the parameters are different. We set the top-down LSTM [2] as our single language decoder and its architecture is shown in Table 12. Specifically, for the m-th decoder, the input in Table 12 (1) is the output of the
-th decoder. When m = 1, this input is word embedding vector
, where
is a trainable embedding matrix and
is the one-hot vector of the word generated at time step
. In Table 12 (2), the output of the second LSTM
at time step
is used as the context vector c in Table 9 (1) and Table 11 (1), and the output of the first LSTM
in Table 12 (11) is used as the hidden vector h in Table 10 (2). After getting the output of the M-th language decoder
, a fully connected layer and softmax activation are used for producing the word distribution P(s) (cf. Section 3.3 of the main paper).
In the experiment (cf. Section 4.2 and Figure 5 of the main paper), we conducted human evaluation for better evaluating the qualities of the captions generated by different methods. In humane evaluation, the invited workers were required to compare the captions from two perspectives: 1) the fluency, e.g., less grammar error, and descriptiveness, e.g., more human-like descriptions, of the generated captions, and 2) the relevance of the generated captions to images. Figure 8 shows one example of the interface of our human evaluation.
Figure 9 exhibits three visualizations for explaining how RELATION module generates relation specific words. For example, in the middle figure, at the third time step, RELA- TION module focuses more on the “paw” part (red box) of one bird, and meantime the knowledge about “bird” (yellow box) and “tree” (blue box) is also incorporated to the “paw” part of the bird by multi-head self-attention technique (cf. Eq.(4) of the main paper). By exhaustively considering these visual clues, a more accurate action “perch” is generated.
Figure 10 shows more comparisons between captions generated by CNM and Module/O. We can find that compared with Module/O, our CNM prefers to use some more accurate words to describe the appeared objects, attributes, and relations. For example, in Figure 10 (a), the attribute “busy” can be assigned to “street”, and in Figure 10 (c), the action “feed” can be correctly generated.
Table 8: The details of RELATION module.
Table 9: The details of FUNCTION module.
Table 10: The details of the common structure of three attention networks.
Table 11: The details of soft fusion.
Table 12: The details of the single language decoder.
Figure 8: The evaluation interface for comparing captions generated by different models.
Figure 9: Three visualizations show how RELATION module generates relation specific words like quantifiers and verbs. The red box in each image is the attended image region (with the largest soft weight) when RELATION module generates a relation specific word. The thickness of lines connecting different boxes is determined by the soft attention weights computed by self-attention technique in Eq.(4). The thicker the line connecting two boxes is, the larger the soft weight between two bounding boxes is.
Figure 10: The visualizations of the caption generation process of two methods: CNM#3 and Module/O. For CNM, different colours refer to different modules, i.e., blue for ATTRIBUTE module, red for OBJECT module, purple for RELATION module, and black for FUNCTION module. For simplicity, we only visualize the module layout generated by the last module controller of the deeper decoder and only the image region with the largest soft weight is shown. For Module/O, only image region with the largest soft weight is visualized with black boundary.