REFFERING Expression Comprehension (REC), also refersto Phase Grounding (Locating) [2], [22], [26] or Visual Grounding [36], [38], is a text-based object detection task. It aims to locating the target region in an image based on a natural language query, e.g., “man in coat and pants”. Recently, REC has drawn great attention in the computer vision community [7], [8], [11], [19], [26], [37], [38], and its advancement has been supported by a series of benchmark datasets [9], [12], [23] and methods [2], [7], [8], [22], [33], [35].
Most existing REC methods follow a multi-stage pipeline [7], [8], [16], [17], [18], [33], [35], [36], [38]. As shown in Fig.1- (b), the core steps include: 1) generating regional proposals by using the region proposal network (RPN) [25] 1; 2) extracting visual&textual features with convolutional neural networks (CNN) and language encoders such as LSTM [6], respectively; 3) and then ranking the matching degree of each region-query pair to determine the target regions. To further enhance the efficiency of the language-vision matching, some recent models, e.g., CMN [7] and MAttNet [33], also implement additional modules for obtaining features of locations or relationships.
Although the complex multi-stage process can guarantee a relatively high performance of REC, it is also computation-ally expensive, e.g., about 3.0 FPS (frame per second) [7], [33], [35], [38], which poses a huge obstacle to a lot of practical applications such as video surveillance and text-to-image retrieval. In addition, these multi-stage approaches rely on pre-trained object
Fig. 1. (a) Examples of Referring Expression Comprehension. (b) Proce- dures of existing methods (left) and the proposed baseline model (right).
detectors, e.g., Faster RCNN [25], to obtain candidate regions, which essentially limits them to a fixed set of object categories that the detector was trained on [27].
Differing from prior work, we argue that fast and efficient REC can be achieved by a one-stage inference, formulating the task as a bounding box regression problem instead of the conventional region-query ranking problem. We tackle the real-time requirement by enforcing the model to be trained in an end-to-end fashion and directly output the target bounding box. To validate this intuition, we first introduce a baseline model in Sec.3, which embeds a language module directly to a pre-trained object detection model, e.g., YOLO3 [24], as illustrated in Fig.1-(b). Another appealing property of one-stage REC is that when the language-vision alignment is well modeled, the model can theoretically perform any fine-grained detection for unseen categories without additional training examples.
In this paper, we also identify two main issues that limit the
upper bound of the one-stage REC, which are (1) The diversity of referring expressions: The expression content in existing datasets [9], [18] varies vastly, which includes various concepts like locations, appearances, fine-grained object categories and actions etc. In this case, the language-vision alignment requires visual features at different semantic levels, which can not be satisfied only using the last convolution layers. (2) The complexity of expressions: Some expressions may contain multiple conditions to indicate the referent, as shown in Fig.1-(a), which requires the model to perform global comparisons and inferences over the entire image.
To this end, we propose a Real-time Global Inference Network, termed RealGIN, with two innovative designs to address the above issues. The framework is shown in Fig.2. To address the diversity of expressions and enhance the discriminability of visual features, RealGIN adopts a novel Adaptive Feature Selection (AFS) scheme to adaptively fuses features at different semantic levels based on the expression content. To release the complexity of expressions, we propose an innovative module called Global Attentive ReAsoNing unit (GARAN), which uses the textual feature as a pivot to collect and diffuse context information, and performs global reasoning towards the conditions in expressions. Meanwhile, we also use the ground truth bounding box to achieve differentiable attentions in the GARAN unit, which subsequently boosts the model performance.
The proposed RealGIN is evaluated on three widelyused benchmarks, i.e., RefCoCo [9], RefCoCo+ [9] and RefCoCog [18], respectively. Very competitive performances against the most advance multi-stage SoA methods, e.g., MAttNet [33], are obtained. In some specific tasks, RealGIN even obtains distinct improvements, e.g., +10% on Test A of RefCOCO. Most importantly, RealGIN achieves real-time processing (26 FPS), which is about 10 times faster than the current SoA methods. Conclusively, the main contributions of this paper are:
• We propose a real-time and one-stage REC model to address the computation inefficiency and the generalization ability of REC, termed Real-time Global Inference network (RealGIN).
• We address the issue of the expression diversity with an AFS unit, which enhances the descriptive power of RealGIN by adaptively fusing features of different semantic levels according to the textual content.
• We address the issue of expression complexity with the GARAN unit, which performs a global reasoning over the entire image to handle the linguistic conditions in expressions. Notably, GARAN unit can also be generalized to most multi-modal tasks.
Referring expression comprehension (REC) is a task of grounding (locating) target regions in an image based on the given naturallanguage expression. Existing methods regard REC as a task of selecting the best region from a set of proposals/objects based on the given expression [2], [7], [8], [22], [33], [35]. To this end, the region proposal network (RPN) [25] is typically adopted to generate object proposals from the given image, and CNN is used to extract the corresponding visual features. In some settings, object proposals provided by the datasets [9], [17] are used. In this work, we focus on the automatic generation of object bounding boxes.
In terms of the methodology used, existing studies can be divided into two broad categories. The first is to learn a joint embedding network for two modalities and compute P (r|o) of each region-query pair, where r denotes the expression and denotes the object [2], [16], [26], [30], [37]. Rohrbach et al. [26] proposed an LSTM network to encode the expression to obtain the textual feature, which is further used as reference information to perform visual attention [31] on region features. The obtained attention scores are further used to indicate the relevance between expression and region. Wang et al. [30] proposed a deep Canonical Correlation Analysis (CCA) [5] network to learn the joint embedding space to rank each region-query pair. Based on the model proposed in [26], Liu et al. [16] added two attribute classifiers to enhance the descriptive ability of visual features, and the predicted attributes are further used as additional features for ranking each region-expression pair.
The other is to model the task as P (o|r), which looks for the object o given the expression r that maximizes the probability [8], [17], [18], [19], [34]. This type of methods typically take visual region features account into the language modeling and use the last hidden state of language network, e.g., LSTM, to predict the relevance of each region-query pair. There are also some works that combine the above two strategies and regards the comprehension and generation of referring expressions as a joint task [17], [18], [35]. Our model is different from the above methods by casting REC as a multi-modal bounding box regression task.
A challenging problem in REC is the modeling of context information based on the linguistic conditions in the expression, which is the key to distinguish the referent from other objects [7], [33], [37], [38]. The simplest method is to use the holistic image information as additional features for ranking region-expression pairs [38]. Some methods compare the visual features from neighbor regions [19], [34], [35], and use multiple instance learning (MIL) [3] to optimize the model. In recent work, Zhang et al. [37] uses a variational Bayesian method to model the referent and context region pairs, Hu et al. [7] and Yu et al. [33] proposed modularized components to obtain features of locations and relationships in addition to the original visual region ones. However, these region-wise comparisons subsequently exacerbate the computation cost of REC. Our work adopts a global reasoning unit to model such context information, which can be optimized by an end-to-end manner.
We first present a baseline model to preliminarily validate the feasibility of one-stage REC. Unlike prior works of selecting the best matching proposal from candidates [7], [8], [18], [33], our baseline model predicts the bounding box directly based on the expression.
As illustrated in Fig.1, the baseline model consists of two main modules: the visual backbone and the language encoder. The visual backbone can be any CNN network, and we use the convolutional feature maps before the last pooling layer as the visual representations, denoted by , where s is the scale and m is the channel number. The language encoder is a bi-GRU network, where the last hidden states of the forward and backward GRUs are added as the textual feature, denoted by
.
Afterwards, we project these two types of features into the same semantic space and then fuse with each regional feature
Fig. 2. The framework of the proposed Real-time Global Inference Network (RealGIN). are feature maps of different scales, which are dynamically fused by AFS to obtain
DownScaleConv.” denotes the convolutional operations used for projecting
same dimension as
denotes the textual feature of the expression. Based on
, the GARAN unit is used to perform global reasoning on
to get the new feature map
, upon which the multimodal matrix
is obtained by fusing it with
in
to obtain a multi-modal matrix
, formulated as:
where and
are the weight matrices, and
denotes any activation function.
is the feature vector of
. Based on
, we add a linear convolution layer to predict the target bounding boxes.
In terms of the bounding box prediction, we refer to the setting of Yolo3 [24], an recent one-stage detection model. Specifically, the baseline model will predict N bounding boxes on each anchor (cell) of the feature map. Each bounding box has the width and height priors, denoted by , and we predict its 4 coordinates, denoted by
. Given the offset of the anchor from the top left corner of the image, denoted by
, the predicted bounding box b corresponds to:
Here, denotes the Sigmoid function. In addition to the 4 coordinates, we predict a confidence score p for the identification of the target object in the predicted box, which is formulated as:
where is the prediction logit. So, the regression layer will output a matrix with a dimension of
. The loss function of the baseline model is then formulated as:
where is the regression loss of bounding box,
is the binary prediction loss of the confidence score, and
is a binarized ground truth with an IOU threshold of 0.5 [24]. In
, we use the binary cross-entropy to measure the regression loss of
and
. For
and
, we adopt the smooth l 1-loss [25] which is found to be more stable than the
loss used in [24] in our experiments.
During the test, the model will output the bounding box with the highest confidence2. Note that, the settings of the above bounding box prediction and loss function can be modified according to any other one-stage detection model.
Based on the baseline model, we further propose a real-time global inference network, denoted as RealGIN, to address the two main issues of the one-stage referring expression comprehension, i.e., the diversity and the complexity of expressions, as discussed in Sec.1. RealGIN has the same visual backbone, language encoder, prediction layer and loss function as the baseline model, and its main differences are two innovative designs embedded into the end-to-end modeling, i.e., adaptive feature selection scheme and global attentive reasoning.
4.1 Adaptive Feature Selection
The main target of Adaptive Feature Selection (AFS) is to adaptively fuse visual feature maps at different semantic levels to enhance the model discriminability, thereby handling the varying content in expressions. The illustration of AFS is shown in Fig.2.
Specifically, given k visual feature maps from different scales, we first use convolution layers to project them to the same resolution and depth by setting the kernel size, stride and channel number. After that, we perform a weighted sum on these maps, and the final visual feature maps can be formulated as:
Fig. 3. (a) The structure of the proposed GARAN unit. (b) The illustration of GARAN unit with multi-head attentions.
Here, are the weights predicted by the textual feature
.
With this setting, we are able to fuse visual information at different semantic levels to represent the varying expressions, which is found to be effective in modeling expressions about object appearances.
4.2 Global Attentive Reasoning
The key to address the complexity of referring expressions is to model context information well based on the linguistic conditions involved. To this end, a one-stage REC model should be able to perceive expression-related information over the entire image, and complement the intrinsic deficiency of convolutional feature maps, i.e., the limited receptive field of anchors in a feature map [14]. We achieve the above goal by proposing a Global Attentive ReAsoNing unit (GARAN).
GARAN is an attention-based component that greatly differs from existing attention models [28], [31], [32], which typically use the attention feature as the visual feature [28], [31] or to refine the textual feature [32]. In contrary, GARAN uses the textual feature as a pivot to collect expression-related information over the whole image, and then selectively diffuse this information to all anchors, as illustrated in Fig.3-(a).
Concretely, given the textual feature and the visual feature map
, we fuse two types of features to predict the two attention maps
and
over anchors. Here,
is used to obtain the attention feature, while
is used in the information diffusion. Each value
in
is calculated by:
Here, are the projection weight matrices, respectively.
is the scalar product of two projected vectors, indicating the relevance between features of two modalities. Values in
are obtained in the same way as
, and the only difference is that we use Sigmoid function to obtain the attention weights. Then, we perform the weighted sum of
to obtain the attention feature
, which can be formulated as:
Afterwards, based on , we convert
to a feature matrix with the same dimension of
, denoted by
. The feature
Fig. 4. Two methods for converting the ground truth bounding box to the attention label. The first one is to directly calculate the overlapping degrees, e.g., IoU, between cells and the target. The second one used in this paper is to place the copied box on each anchor (cell) and then calculate the overlapping degrees between the copied one and the target one.
vector of
is the diluted attention feature obtained by
. Lastly, we combine
with
using element- wise addition and obtain the new feature maps
with another convolution layer.
GARAN with multiple attentions. Towards a better context modelling, we draw on the multi-attention design from [29] to enhance the attention ability of GARAN, as shown in Fig.3-(b).
Specifically, we decompose into k splits from the last dimension, and obtains a set of feature maps, denoted by
. Afterwards, we perform the above attention process on each feature map and then recombine all generated attention matrices to obtain a complete one, denoted by
.
With such a design, RealGin will perform attentions on a set of truncated visual features processed by different convolution filters, which can perceive more accurate and sufficient context information for bounding box regression.
Differential Attentions. Due to the lack of corresponding annotations, previous attention-based models cannot directly optimize the attention layer [31], [32]. In REC, we can use the ground truth bounding box as the supervision information to achieve differential attentions.
A natural solution for converting the bounding box to the label information is to calculate the overlapping degree, e.g., IoU, between each grid cell in the feature map and the given box, as shown in the top example of Fig.4. However, such a method will lead to a result that many anchors have the same attention score, which can not accurately reflects the importances of visual regions towards the final task.
To this end, we propose an innovative method to calculate the ground truth scores of attentions. As shown in the bottom example of Fig.4, we place the bounding box with the same shape of the target one on each anchor. Then, the attention score of an anchor is decided by the IoU value between its placed box and the target one. With this setting, there is at most one anchor with a score of 1.0. Besides, for those anchors that are distant from the center point of ground truth box, their attention scores will decrease according to the distances, which is closer to the behavior of human cognition [31] and more suitable for our task.
Given the ground truth score of an anchor , we compute the sigmoid binary cross entropy with the attention logit
defined in Eq.6. Then the attention loss is formulated as:
where is the hyper parameter.
We further evaluate the proposed baseline model and RealGIN on five benchmark datasets, i.e., RecCOCO [9], RefCOCO+ [9], RefCOCOg [18], ReferIt [9] and Flicker 30k entities [23], and compare them to a set of SoA methods. Besides, we also validate the generalization ability of RealGIN on the Zero-Shot Grounding (ZSG) dataset [27].
5.1 Datasets and Evaluation Metric
RefCOCO (UNC RefExp) [9] has 142,210 referring expressions for 50,000 bounding boxes in 19,994 images from MSCOCO [15], which is split into train, validation, Test A and Test B with a number of 120,624, 10,834, 5,657 and 5,095 samples, respectively. The expressions are collected using an interactive game interface [9], and they are typically short sentences with a average length of 3.5 words. The categories of bounding boxes in Test A are people while the ones in Test B are objects.
RefCOCO+ [9] has 141,564 expressions for 49,856 boxes in 19,992 images from MS-COCO. It is also divided into splits of train (120,191), val (10,758), Test A (5726) and Test B (4,889). Compared to RefCOCO, its expressions include more appearances (attributes) than absolute locations [9] to describe the target box. Similar to RefCOCO, expressions of Test A in RefCOCO+ are about people while the ones in Test B are objects.
RefCOCOg (Google RefExp) [18] has 95,010 expressions for 49,822 boxes in 25,799 images from MS-COCO. The split is 85,474 and 9,536 samples for training and validation. Since the test set is not released, we use the UNC partitions [33], [34] of the val split for validation and testing. Compared to the above two datasets, expressions in RefCOCOg are collected in a noninteractive way, and the lengths are longer (8.4 words on average), of which content includes both appearances and locations of the referent.
ReferIt [9] contains 130,525 expressions referring to 96,654 distinct objects in 19,894 images of real world scenes. All the expressions are collected by the two player game. These expressions cover most objects present in the IAPR corpus.
Flickr 30k Entities [23] is built based on the Flickr30k dataset, which augments about 158k captions from Flickr30k with 244k coreference chains, linking mentions of the same entities across different captions for the same image, and associating them with 276k bounding boxes.
Metric. Following the setting in prior works [9], [33], [34], we use the precision as the evaluation metric. Specifically, when the overlapping degree, i.e., IoU, between the predicted bounding box and the ground truth one is larger than 0.5, then the prediction is correct.
TABLE 1 Ablation studies of the baseline model and RealGIN on the val set of three datasets, where k denotes the number of attentions and denotes pretraining on ImageNet. *is the final setting of RealGIN.
TABLE 2 Comparisons between other context modeling scheme and the proposed GARAN on RefCOCO.
5.2 Experimental Settings
For the proposed baseline model and RealGIN, we use the Darknet from Yolo3 [24] as the visual backbone on RefCOCO, RefCOCO+ and RefCOCOg datasets. Following the setting in [33], the backbone is pretrained on the MS-COCO dataset while removing the images appeared in the validation and test sets of RefCOCO, RefCOCO+ and RefCOCOg. Considering that some previous methods [36], [39] use backbones pretrained on the complete MS-COCO dataset, we also report the experiment results using the pre-trained Darknet by Yolo3 [24] for fair comparisons. For the baseline model, we use the last convolution feature map of DarkNet as the visual features with a shape of . For RealGIN, the outputs of Layer26 and Layer43 of DarkNet are used addition-ally with a dimension of
and
, respectively. The kernal size and channel number of convolution layers for projecting Layer92 and Layer152 are set to be
and 1,024, and the stride is 4 and 2, respectively. Words in the expressions are initialized with GLOVE embeddings [20]. The dimension of the forward GRU and the backbward GRU is set to be 1,024. In terms of multi-modal fusion, the dimension d in Eq.1 is 1,024. For the GARAN unit in RealGIN, the number of attentions k is 4, and the attention dimension is 256. The filter size and channel number of the convolution layer after the GARAN unit and before the prediction layer are set to be
and 1,024, respectively. The activation function for convolutional layers in the baseline model and RealGIN is LeakyReLU [24], and the Batch Normalization is added after each convolutional layer. The hyper parameter
is 0.05. During the training, the parameters of visual backbone are fixed. The batch size is 64, and the learning rate is set to be 0.001, which is halved for every 10 epochs. The max epoch is set to be 60, and the early stop is applied when the loss does not decay during 5 epochs. The optimizer is Adam [10].
TABLE 3 Comparisons with the state-of-the-arts on RefCOCO, RefCOCO+ and RefCOCOg under the setting of automatic bounding box generation.
The Google RefCOCOg val split. ∗ We test SoA methods that release source codes on the same hardware, i.e., GTX1080ti. Different DL libraries used will slightly affect the speed.
backbone pretrained with the complete MS-COCO set.
TABLE 4 Comparisons with the state-of-the-arts on Refrit and Filckr30k under the setting of automatic bounding box generation.
∗ Reimplemented Results A multi-scale version of RealGIN using ZSGNet as the base model.
5.3 Experimental Results
5.3.1 Quantitative Analysis
We first evaluate the effectiveness of each component in our model, of which results are shown in Tab.1. The first block in Tab.1 presents different settings of our baseline model: “visual only” denotes using the Yolo3 [24] as the REC model and predicting the target bounding box by selecting the one with the highest objectness score. From these results, we observe that pretraining on object detection datasets is important to the model, which can provide good detection capabilities. However, the result of “visual only” also implies the importance of the language-vision modeling. The second block shows the efficacy of RealGIN’s designs. As shown in Tab.1, each design in RealGIN significantly improves the model performance. In RefCOO+, where expressions are about object appearances, the improvement by AFS is up to 10.4%, strongly suggesting that our design greatly enhances the visual features. In RefCOCOg with long expressions, GARAN with att loss can also bring up to 4.1% increase, which also confirms its advantage in the context reasoning. Meanwhile, the benefit of the attention loss is also significant.
In terms of the context modeling, we also compare the proposed GARAN unit to the alternatives. Here, “Baseline+Att” denotes using the attention feature to update the textual feature by following the setting in [31], [32]. “+Loss” means the use of attention loss proposed in Sec.4.2. “+Transformer” denotes using the multi-attention layer [29] to model the context information, and k denotes the number of attention heads. As shown in Tab.2, under the same settings, GARAN is better than the alternatives, and its advance becomes more significant when combined with the attention loss, which greatly confirms the merit of the proposed GARAN.
In terms of the model performance and processing speed, we compare the proposed baseline model and RealGIN with the SoA methods on RefCOCO, RefCOCO+ and RefCOCOg, the results of which are shown in Tab.3. From Tab.3, we can first observe that our baseline model is already better than most SoA methods, validating the feasibility of one-stage REC. Its performance on the location-related dataset, i.e., RefCOCO, also shows its intrinsic advantages in spatial information modeling. To explain, prior multi-stage methods typically perform language-vision matching between feature vectors of the region and the expression, resulting in the lost of spatial information. In contrast, in one-stage inference, the model directly predicts the bounding box from the feature matrix, where the regional structure and spatial information are well preserved. Another important observation is that RealGIN has achieved very close performance to the most advanced SoA method, i.e., MAttNet, on all three datasets. However, we also observe RealGIN’s shortcoming in referring large instances like People on Test A of RefCOCO+ compared to MAttNet. The main reason is that the region features used in MAttNet merit at describing more complete object information than that of CNN feature maps used in our scheme [1]. Nevertheless, while retaining high performance, RealGIN significantly outperforms all SoA
Fig. 5. Visualizations of the baseline model and RealGIN. The green boxes are the ground truths while the red ones are the predictions. Values of v under images are AFS scores of feature maps of three scales.
methods in terms of the computation efficiency. As shown in Tab.3, the processing speed of SoA methods is only 1.3 to 2.1 FPS, while the one of RealGIN is 26.3 FPS, which is about 10 to 20 times faster than the existing methods. In addition to the superior accuracy, this speed advantage also strongly confirms the contribution of this paper. We further compare our method with SOTA on another two datasets, i.e., Flickr30k and ReferIT, as shown in Tab.4. The sizes of targets in these two datasets varies vastly, so we extend RealGIN to a multi-scale version by using the ong-stage model ZSGNet [27] as our base model. On these two datasets, our performance gains are more significant, greatly outperforming existing methods. Compared to ZSGNet, which also use the one-stage design, our merits validate the proposed designs again, i.e., AFS and GARAN.
5.3.2 Qualitative Analysis
We further visualize the predictions of the baseline and RealGIN to gain insights into both models, as shown in Fig.5. Fig.5-(a) shows the comparisons between the baseline and RealGIN. As shown in these examples, RealGIN can more confidently locate the target objects than the baseline. Meanwhile, it also significantly merits in modeling expressions with attributes or multiple reference conditions, e.g., Exp.(2), (3), which subsequently validates the design of AFS and GARAN. Fig.5-(b) shows visualized attentions and grounding of RealGIN. It can be seen that RealGIN is capable of referring short expressions with high confidence in images with complex visual scenes, e.g., Exp. (2)-(4). Such a result gives a high probability of practical application. Meanwhile, it can also handle long expressions with complex linguistic conditions, showing a high reasoning ability in one-stage REC, e.g., Exp.(1), (4)-(6). Also, RealGIN is able to learn some interesting knowledge never appearing in pretraining, e.g., “horse ass” in Exp.(8), confirming its ability of modeling unseen categories. Besides, we have two findings from these results. First, the AFS scores also reflect the different properties of three datasets, e.g., the appearancerelated RefCOCO+ has more requirements of low-level features. Second, the attention results also suggest that GARAN can capture all expression-related information towards the phase grounding. These two findings further confirm the effectiveness of AFS and GARAN.
Fig.5-(c) gives some failure cases of RealGIN. From these cases, we observe that in addition to the label noise and ambiguities, the main factor leading to failure prediction is the insufficient knowledge about visual concepts appearing in expressions. Some visual concepts are very abstract and appear very few times in the dataset, causing the model to fail to master them well, such as sun shining or the number 2. Meanwhile, the backbone of RealGIN is pretrained with very few object categories, which also limits its recognition performance. With more REC training examples and the pretraining with more categories, we believe that the performance of RealGIN can be further improved.
In this paper, we address the computation inefficiency of Referring Expression Comprehension (REC). All existing methods of REC follow a multi-stage pipeline, which is computation-ally expensive. In contrast, we argue that real-time REC can be achieved by an one-stage method. To validate this motivation, we first presented a simple baseline that embeds a language module to a one-stage object detection model, which still outperforms most prior works. To address two main issues of one-stage REC, i.e.,, the diversity and complexity of expression, we further proposed the RealGIN network with two innovative designs, i.e., AFS and GARAN. AFS adaptively fuses visual features at different semantic levels to represent the varying content in expressions, which greatly enhances the model discriminability. GARAN uses the textual feature as a pivot to collect and diffuse query-aware information to all anchors, providing sufficient context for modeling linguistic conditions in expression. On three benchmark datasets, RealGIN achieves very competitive performance against the most advanced REC method: MAttNet. Most importantly, its processing speed is about 10 times faster than existing multi-stage methods.
[1] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, volume 3, page 6, 2018.
[2] K. Chen, R. Kovvuri, and R. Nevatia. Query-guided regression network with context policy for phrase grounding. international conference on computer vision, pages 824–832, 2017.
[3] T. G. Dietterich, R. H. Lathrop, and T. Lozanoperez. Solving the multiple instance problem with axis-parallel rectangles. Artificial Intelligence, 89(1):31–71, 1997.
[4] A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847, 2016.
[5] D. R. Hardoon, S. Szedmak, and J. Shawetaylor. Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16(12):2639–2664, 2004.
[6] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
[7] R. Hu, M. Rohrbach, J. Andreas, T. Darrell, and K. Saenko. Modeling relationships in referential expressions with compositional modular networks. computer vision and pattern recognition, pages 4418–4427, 2017.
[8] R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, and T. Darrell. Natural language object retrieval. computer vision and pattern recognition, pages 4555–4564, 2016.
[9] S. Kazemzadeh, V. Ordonez, M. Matten, and T. L. Berg. Referitgame: Referring to objects in photographs of natural scenes. pages 787–798, 2014.
[10] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[11] R. Krishna, I. Chami, M. S. Bernstein, and L. Feifei. Referring relationships. computer vision and pattern recognition, 2018.
[12] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. arXiv preprint arXiv:1602.07332, 2016.
[13] J. Li, Y. Wei, X. Liang, F. Zhao, J. Li, T. Xu, and J. Feng. Deep attribute- preserving metric learning for natural language object retrieval. pages 181–189, 2017.
[14] T.-Y. Lin, P. Doll´ar, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In CVPR, volume 1, page 4, 2017.
[15] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ar, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
[16] J. Liu, L. Wang, and M. Yang. Referring expression generation and comprehension via attributes. International Conference on Computer Vision, pages 4866–4874, 2017.
[17] R. Luo and G. Shakhnarovich. Comprehension-guided referring expres- sions. computer vision and pattern recognition, pages 3125–3134, 2017.
[18] J. Mao, J. Huang, A. Toshev, O. M. Camburu, A. L. Yuille, and K. P. Murphy. Generation and comprehension of unambiguous object descriptions. computer vision and pattern recognition, pages 11–20, 2016.
[19] V. K. Nagaraja, V. I. Morariu, and L. S. Davis. Modeling context between objects for referring expression understanding. european conference on computer vision, pages 792–807, 2016.
[20] J. Pennington, R. Socher, and C. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
[21] B. A. Plummer, P. Kordas, M. H. Kiapour, S. Zheng, R. Piramuthu, and S. Lazebnik. Conditional image-text embedding networks. arXiv: Computer Vision and Pattern Recognition, 2017.
[22] B. A. Plummer, A. Mallya, C. M. Cervantes, J. Hockenmaier, and S. Lazebnik. Phrase localization and visual relationship detection with comprehensive image-language cues. international conference on computer vision, pages 1946–1955, 2017.
[23] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hocken- maier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. international conference on computer vision, pages 2641–2649, 2015.
[24] J. Redmon and A. Farhadi. Yolov3: An incremental improvement. arXiv: Computer Vision and Pattern Recognition, 2018.
[25] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6):1137–1149, 2017.
[26] A. Rohrbach, M. Rohrbach, R. Hu, T. Darrell, and B. Schiele. Grounding of textual phrases in images by reconstruction. european conference on computer vision, pages 817–834, 2016.
[27] A. Sadhu, K. Chen, and R. Nevatia. Zero-shot grounding of objects from natural language queries. arXiv: Computer Vision and Pattern Recognition, 2019.
[28] D. Teney, P. Anderson, X. He, and A. V. Den Hengel. Tips and tricks for visual question answering: Learnings from the 2017 challenge. computer vision and pattern recognition, 2018.
[29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. neural information processing systems, pages 5998–6008, 2017.
[30] L. Wang, Y. Li, and S. Lazebnik. Learning deep structure-preserving image-text embeddings. computer vision and pattern recognition, pages 5005–5013, 2016.
[31] K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, volume 14, pages 77–81, 2015.
[32] Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 21–29, 2016.
[33] L. Yu, Z. Lin, X. Shen, J. Yang, X. Lu, M. Bansal, and T. L. Berg. Mat- tnet: Modular attention network for referring expression comprehension. computer vision and pattern recognition, pages 1307–1315, 2018.
[34] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context
in referring expressions. european conference on computer vision, pages 69–85, 2016.
[35] L. Yu, H. Tan, M. Bansal, and T. L. Berg. A joint speaker-listener-reinforcer model for referring expressions. computer vision and pattern recognition, pages 3521–3529, 2017.
[36] Z. Yu, J. Yu, C. Xiang, Z. Zhao, Q. Tian, and D. Tao. Rethinking diversified and discriminative proposal generation for visual grounding. international joint conference on artificial intelligence, pages 1114– 1120, 2018.
[37] H. Zhang, Y. Niu, and S. Chang. Grounding referring expressions in images by variational context. computer vision and pattern recognition, pages 4158–4166, 2018.
[38] Y. Zhang, L. Yuan, Y. Guo, Z. He, I. Huang, and H. Lee. Discriminative bimodal networks for visual localization and detection with natural language queries. computer vision and pattern recognition, pages 1090– 1099, 2017.
[39] B. Zhuang, Q. Wu, C. Shen, I. D. Reid, and A. V. Den Hengel. Parallel attention: A unified framework for visual object discovery through dialogs and queries. computer vision and pattern recognition, pages 4252–4261, 2018.
Fig. 6. Predictions and attentions of RealGIN.