SEGMENTING objects of interest in an image is a fun-damental problem in computer vision field. Researchers have formulated various high-level computer vision tasks related to segmenting objects in images, such as semantic segmentation [1], [2], instance segmentation [3], salient object segmentation [4], [5]. However, these tasks have some limitations. For example, semantic segmentation and instance segmentation assume a pre-defined set of object categories (e.g., cat, person, bus, etc). Salient object segmentation does not have the restriction on the pre-defined object categories and is based on the human visual cognition system to distinguish the most salient objects in the scene from background. But in some complicated cases, the salient objects are ambiguous and may not be unique for different viewers [6]. In recent years, referring image segmentation has attracted the attentions of many researchers. In referring image segmentation, the object of interest to be segmented is specified by a free-form referring expressions in natural language. Fig. 1
c 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. L. Ye and Y. Wang are with the Department of Computer Science, University of Manitoba, Winnipeg, MB R3T 2N2, Canada (e-mail: {yel3, ywang}@cs.umanitoba.ca). Z. Liu is with Shanghai Institute for Advanced Communication and Data Science, and School of Communication and Information Engineering, Shanghai University, Shanghai 200444, China (e-mail: liuzhisjtu@163.com).
Fig. 1. Illustration of the referring image segmentation task. Given an input image and a referring expression, the goal is to generate the segmentation mask for the referred object in the image. The referring expression may use diverse descriptions to identify the referred object, such as object names (e.g. “man”, “guy”), attributes (e.g. “back to camera”, “yellow”) and spatial relationships (e.g. “on left”). The first row shows three query expressions and the last row indicates the corresponding segmentation mask in the image.
illustrates the example of referring image segmentation. With an input image, the segmentation mask can be referred using diverse descriptions for the same object (the first two examples from the left) by its attribute “back to camera” or spatial relationship“on left” to differ the two same categorical objects “man” or “guy”. The prediction can also be identified by the attribute of the shirt “yellow” related to another object in the image. Referring image segmentation is a challenging problem which requires a combinational comprehension of both linguistic and visual information, and enables many real-world applications including interactive image editing, intelligence visual search and human-robot interaction.
There are several existing works in this area. Some of them [7]–[9] represent the whole referring expression and visual features separately. For example, the referring expression is encoded as a hidden vector using recurrent neural network (RNN) or long short-term memory (LSTM) model [10], while the input image is represented using convolutional neural network (CNN) features. The textual feature vector is then combined with visual features at each spatial location followed by deconvolution [7], recurrent refinement [8] and key-word context [9] for producing the final segmentation mask. The limitation of these approaches is that visual and textual features are extracted in an independent way. It may not be able to capture the detailed multimodal information often useful for the referring image segmentation task.
A different line of previous work [11], [12] processes each word in the referring expression in a sequential order. These methods can potentially capture the detailed information of words in the referring expression with visual context by a sequential interaction [11] or synthesis module [12]. However, these methods treat every word equally in their models. This may cause difficulty for long referring expressions that likely contain unimportant words.
To address the limitations of previous work, we propose a dual convolutional LSTM network for referring image segmentation. The convolutional LSTM (ConvLSTM) is originally proposed in [13] to replace fully connected layers with convolutional layers in order to capture spatial-temporal information of a sequence of images. We elaborately modify the original ConvLSTM to fit the multimodal data in the referring segmentation problem by the proposed encoder-decoder framework. Our network models the input image as the spatial information, and formulates the language expression and multi-level features as the temporal sequence for the encoder and the decoder, respectively. Specifically, the encoder network (E-ConvLSTM) is used to capture multimodal feature interactions. It first adopts the feature maps of the input image as the spatial information. For each spatial location of the feature map, we capture the interaction at this location over every word in the referring expression at each recurrent time step of E-ConvLSTM to gradually localize the referred object. In addition, the proposed approach introduces wordlevel attentions embedded into the cell state of E-ConvLSTM to guide the interaction process towards more important words (e.g. words corresponding to the object of interest), instead of treating each word equally in the recurrent step [11], [12]. The decoder network (D-ConvLSTM) formulates attentive multimodal features encoded by E-ConvLSTM at different levels as the sequence and iteratively refines these features to take full advantage of correlations from multi-level features. We further introduce spatial attentions for multi-level features to better focus on the semantics by the high-level features and fine details by the low-level features for a precise segmentation mask.
In summary, the main contributions of this paper lie in the following four aspects:
1) We propose a dual convolutional LSTM network to exploit an encoder-decoder framework for multimodal feature encoder and multi-level segment decoder in the spatial regions for referring image segmentation.
2) The multimodal feature encoder embeds word attention into the cell (memory) state of E-ConvLSTM to adaptively encode the multimodal interaction towards more important words and localize the referred objects.
3) The multi-level segment decoder (D-ConvLSTM) progressively decodes the features with spatial attention at multiple levels to refine more precise referring image segmentation results.
4) Our proposed dual convolutional LSTM network is evaluated on four public available datasets thoroughly and achieves the state-of-the-art performance.
The rest of this paper is organized as follows. We first introduce some related works in Sec. II and the basic ConvLSTM unit as background in Sec. III. Then we present an encoder-decoder framework consisting of the proposed E-ConvLSTM in Sec. IV and D-ConvLSTM in Sec. V in detail. Experimental setup and extensive experimental results for performance evaluation are given in Sec. VI and Sec. VII, respectively. Finally, we make conclusions in Sec. VIII.
In this section, we first review some relevant object seg- mentation tasks. Then a set of vision and language problems is introduced for studies in the intersection of multimodal information processing. Last, the recent studies tightly related to our work about referring image segmentation are presented. Object Segmentation: Object segmentation or extraction from an image can be done rapidly and freely by human but is a challenge problem in computer vision. A major difficulty lies in that computers have to be aware of the object of interest ahead of executing segmentation operation. One straightforward way is to roughly specify an object interactively by human, e.g., drawing a bounding-box [14]. Then a segmentation approach can refine the specific object with well-defined boundaries according to visual cues of the candidate region and background (inside and outside the bounding-box). The desired object can also be identified automatically inspired by human visual system which distinguish visually salient objects from background [4]–[6], [15]. In the recent data exploding era, convolutional neural network [16] drives significant segmentation performance boost in semantic segmentation [1], [2] where all pixels are labeled with pre-defined object categories, and instance segmentation [3] where additional instance labels are available. This makes visual understanding foundation for referring image segmentation in this paper. Combining Vision with Language: There has been a lot of previous works on combining vision with language for different tasks, such as image captioning [17] and visual question answering (vqa) [18]. These models incorporate attention mechanism to explore relevant visual features on the corresponding spatial regions, the localized regions are not precise enough since the goal of these works is to generate a sentence or a bounding box. Visual grounding [19], [20] requires an exact bounding-box as the output according to a referring expression. A joint multimodal embedding is used in reconstruction of the text phrase [19] and modular networks of subject, location and relationship [20]. However, all these grounding methods rely on proposals generated by off-the-shelf object detectors.
For tasks like referring image segmentation, we need to effectively represent the multimodal interaction between the linguistic and visual information and keep detailed spatial information in order to generate a segmentation mask. A straightforward way to combine visual and linguistic features is to use simple concatenation as [7], [8], key-word context [9] or cross-modal self-attention mechanism [21] for multimodal features. Another line of work [11], [12], which is more related to our work, exploits the sequential nature of language and differ from how human solves this problem [22]. However, these two methods consider each word equally in the interaction. Instead, we exploit word attentions over the input expression and embed them into cell states of ConvLSTM to guide the effective multimodal feature interaction for the encoder network.
Referring Image Segmentation: Referring image segmentation is first introduced by [7] to segment the object-of-interest referred by an expression. They concatenate visual, linguistic and spatial features on the spatial feature maps and use a deconvolutional layer to recover a high-resolution segmentation mask. In [11], a recurrent multimodal interaction model is proposed to gradually ground the referred objects onto the image according to the progressive meaning of the language input. Each word is gradually combined with visual features in a sequential order. The work in [12] adopts a similar method as [11] but generates dynamic filters in the synthesis module to incorporate linguistic information. It further uses an incremental module with bilinear upsampling and convolution over feature maps for fine details. However, these works equally encode every word in the referring expression at each step of a recurrent model. It may be difficult to effectively capture the important words in a long referring sentence. In this paper, we propose a multimodal feature encoder to embed word attentions into the cell state of ConvLSTM. The multimodal interaction of visual and linguistic features can be learned adaptively towards more important words in the expression to identify the referred object.
Key-word-aware context is proposed in [9] to combine textual feature with image regions to model their relationships, which aligns key words to different image regions. In order to obtain a more precise mask for segmentation, more elaborated refinement approaches are proposed by [8] and [21]. Specifi-cally, multi-scale features are progressively refined to improve the segmentation mask from a roughly localized mask in [8]. In [21], the cross-modal self-attention network is proposed to capture the long-range dependencies between linguistic and visual contexts and a gated multi-level fusion is then used to extract a precise segmentation mask. Both methods show the effectiveness of the iterative refinement for removing irrelevant regions and producing more precise segmentation masks in the end. Though significant improvements have been achieved by these refinement methods, the importance and relation between different levels of features are not fully exploited. Different from these methods which simply integrate multi-level visual features for refinement, we deploy a decoder to utilize encoded multimodal information and introduce spatial attentions into multi-level features to focus on more specific features in the refinement. It selectively concentrates on the main body of the referred object and boundary details according to the high-level and low-level features, respectively. Long Short-Term Memory Network: Long short-term memory (LSTM) network [10] has been widely adopted for sequential data (e.g., language [23], audio [24] and video [25]). LSTM can effectively capture long-range dependencies in sequential data. LSTM contains fully connected layers in both the input-to-state and state-to-state transitions with four gates including an input gate, a memory gate, a forget gate and an output gate. It can be flexibly used in both the encoder and the decoder framework for sequential inputs and outputs [23]. LSTM can be used to combine word features with visual context for multimodal interaction [11], [17]. Instead of propagating information in one direction as in standard LSTM, bidirectional LSTM is proposed to learn
hidden states from both forward and backward directions at the same time. The forward and backward passes can learn comprehensive information through the sequential input [26], [27]. Convolutional LSTM (ConvLSTM) replaces the fully connected layers of LSTM with convolutional layers to make the sequential learning possible in spatial-temporal domains. It performs better in handling spatio-temporal correlations for a set of images [13]. In addition, several other variants of LSTM are designed to stack multilayer LSTM for skeleton-based action recognition [28], formulate AutoEncoder topology [29] or combine fully convolutional neural networks with LSTM for vehicle counting [30].
A general model for referring image segmentation requires spatial relationships of an image and sequential dependency of words. Convolutional networks and LSTM [10] are two powerful feature representation approaches for an image and words, respectively. A convolutional network extracts hierarchical spatial semantic features from an image. LSTM models long-range dependency for sequential words. Convolutional LSTM (ConvLSTM) [13] is an extension of the vanilla LSTM for capturing spatio-temporal relationship of data. It replaces fully connected layers in LSTM with convolutional layers for input-to-state and state-to-state transitions so that spatial correlation can also be built within the sequential inference process. In order to simultaneously capture spatial and sequential information, we introduce a dual convolutional LSTM framework for multimodal feature encoder and multi-level segment decoder between an image and words. The proposed network captures the interaction between linguistic and visual context needed for localizing the referred object. It also maintains the spatial information needed for producing a precise segmentation mask.
is the input at time step t in ConvLSTM. The complete ConvLSTM operation can be summarized as follows:
where W represents the ConvLSTM parameters, is the sigmoid function and
is the element-wise product. Here we use
to denote the input gate, forget gate, output gate and memory gate, respectively.
is the hidden state and
is the cell state at each time step t. At each time t, ConvLSTM takes the input
and previous hidden state
to generate
and
at current time.
The essential design of ConvLSTM is to use convolution layer for spatial correlations and uses recurrence over time for sequential dependency. At each time t, the input gate controls how much information from
is exposed to the cell state. The forget gate
controls how much information from the past should be forgotten. This results in an updated cell state
. The output gate
is then used to propagate effective information to the hidden state
at time t as the output. Therefore, the cell state
is considered as memory state to control the information flow update. In other words, it determines how much the current and previous states influence the current hidden state
.
Fig. 2. The multimodal feature encoder with E-ConvLSTM. We embed word attentions into the ConvLSTM cell to adaptively encode the multimodal interaction towards more important words.
Fig. 3. Examples of word attentions. From left to right at each row: the original image, word attentions, visualization of the final feature map. Darker red color means higher attention weights.
The goal of the encoder network of our model is to generate multimodal features that capture both detailed linguistic and visual information. The overall architecture of the encoder network is shown in Fig. 2. First, we use an attention mechanism to learn to focus on certain important words in the referring expression (Sec. IV-A). Then word vectors are reweighed by word attentions and tiled as the same size as the visual feature map. Each word feature is concatenated with the visual feature and the spatial feature for a word-specific multimodal feature. These multimodal features capture the complex interactions between the linguistic information from the referring expression and the visual information from the image. Finally, multimodal interaction (Sec. IV-B) is guided by word attentions to adaptively towards more important words in the referring expression.
A. Word Attention
Instead of representing the entire referring expression as a hidden vector [7], [8] using LSTM, we propose to keep track of the vector representation of each word in the expression. We also learn the relative importance (i.e. word attention) of each word and reweight the vector representation of each word with its corresponding attention score. This allows our model to focus on important words in the expression and use it to adaptively encode multimodal information.
Let us denote a referring expression as where
is the l-th word and L is the number of words. Each word
is represented as a one-hot vector. We project each word into a vector representation
using a word embedding layer, then use a bidirectional LSTM to produce a hidden vector for each word as follows:
where and
denote the forward and backward directions in bidirectional LSTM, respectively.
is used as the vector representation for the l-th word, which concatenates the hidden states of bidirectional LSTM at every time step (word) l with consideration of both previous and future relations of words.
We then apply two linear layers on the word feature and normalize the output to calculate an attention weight
that indicates the relative importance of the word:
where and
are the model parameters of these two linear layers. We then use the attention weight of each word to re-weight its vector representation as follows:
The final vector representation is considered as the generated attentive word feature that takes into account of the relative importance of the l-th word. It conveys discriminative information in the given referring expression. Fig. 3 shows examples of word attentions. The word attentions encourage the network to focus on the more important words so as to identify the referred object (“keyboard” in the top example) or discriminate the similar objects (“black cow” in the bottom example) with the relative location (“front”) and different attribute (“facing to the left”). These word attentions will also contribute differently on the multimodal interaction in the next section.
B. Word Attentive Multimodal Interaction
We propose a word attentive multimodal interaction model that captures the multimodal information, including the linguistic information in the referring expression and the visual/spatial information in the input image. Multimodal Feature Generation: We use a pretrained CNN network to extract a visual feature map for an input image. Let the feature map size be , where W and H are the width and height of the feature map, and
is the channel dimension. We then generate an 8-dimensional vector representing the spatial information at each spatial location in the feature map as [11]. Specifically, we use first three dimensions to encode the relative horizontal coordinates and another three dimensions to encode the relative vertical coordinates. The last two dimensions correspond to the relative sizes (width and height) of the image. The spatial feature map has a dimension of
. All values of this spatial feature map are normalized to the range of
, i.e., the values of the upper left corner and the lower right corner of the spatial feature is
and (1, 1), respectively. This spatial feature is then appended to the visual feature map at each spatial location. Thus the dimension of the final visual feature map is
. This feature map captures both visual and spatial information of an image.
Then the multimodal feature can be generated as follows. For the l-th word, we append its vector representation to each spatial cell of the feature map V . This results in a word-specific multimodal feature
. We repeat this process for all words
. In the end, we obtain L word-specific multimodal feature maps
. Multimodal Interaction: Given the feature maps
obtained for each word separately, we want to combine these feature maps and capture their interactions along each word over time for detailed multimodal comprehension. Different from the previous work [11] that treats each word equally, our multimodal interaction model can pay more attention to important words by taking into account of word attentions. For the word-specific multimodal feature maps
,we modify the standard ConvLSTM (see Sec. III) in order to take into account of the word attentions as follows:
where
The output of the hidden state at each time step summa- rizes the semantic information for all previous seen words before l. So the hidden state at the final step
contains information of all words in the referring expression. The cell state
can take advantage of the attention weight
of the corresponding word. As shown in Eq. 11, the attention
of a word (obtained in Eq. 8) is used to modulate the cell memory of the multimodal interaction. If a word has a higher attention weight, it will encourage the cell state to allow more information to flow from the current state. In contrast, a word with lower attention weight will allow less information flow into the cell state. So the cell state will rely more on the historic memory. In the end, the modified ConvLSTM is able to pay more attention to important words with higher attention weights.
Fig. 2 illustrates the whole process of E-ConvLSTM. Given the input query “lemon on left”, the multimodal feature encoder produces word attention for each word in the referring expression. The different word attentions are used to generate attentive word feature
for multimodal features. In addition, the attention of every word is also embedded into the cell memory of each E-ConvLSTM cell to adjust multimodal interaction towards more informative words (e.g., “lemon” and “left”) adaptively.
Referring image segmentation aims to segment the referred object rather than predicting dense labels for the entire image. Directly applying the multimodal features to predict a segmentation mask might lead to an unsatisfactory result because of the distraction of non-referred regions. In addition, multi-level feature representations help to provide fine details for a more precise boundary of the object. To this end, we propose to generate spatial attention to focus on important spatial regions in the multimodal feature maps. The spatial attentions are then applied at the different levels of encoded multimodal features generated by the E-ConvLSTM. We also propose the multi-level segment decoder with another ConvLSTM (D-ConvLSTM) to sequentially refine the different levels of encoded multimodal features for a precise segmentation mask. The hidden state of the last word from Eq. 10 can be used as the input to the decoder network. Note that
is computed using the CNN features at a particular layer. In practice, we can apply the encoder by using the CNN features at several different layers in the network. We use
to denote the hidden state
in the encoder based on different levels of CNN visual features. In this paper, we set s = 3 and
corresponding to {Res5, Res4, Res3} of a DeepLab101 backbone network [2]. Specifically,
can be generated by applying the encoder network in Sec. IV-B at a particular level s for multimodal representation over the referring expression. Spatial Attention: The spatial attention is generated to focus on important spatial regions in the feature map. We use a convolutional layer with a large kernel (
) to capture relatively large regions. The convolution of a large kernel can effectively gather information with a larger receptive field and is robust for localization [31].
where and
are the parameters of the convolution filters. The sigmoid function
summarizes the importance of different regions in the feature map. Then it is applied to each slice of
. Fig. 4 presents the spatial attentions in different levels of features. It can be observed that spatial attentions can help the decoder refines features from the important spatial regions. The spatial attention corresponding to high-level features tends to concentrate on the referred objects,
Fig. 4. Visualization of the spatial attentions corresponding to different levels of encoded features. (a) original image; (b,c,d) the spatial attentions from the high-level to low-level features.
while the spatial attention of the low-level features tends to be more spread-out.
Fig. 5. An illustration of the multi-level segment decoder with D-ConvLSTM. The decoder network iteratively refines the segmentation mask by using the features extracted from different CNN levels. It also uses spatial attentions to focus on image regions that are informative for generating the segmention mask.
Multimodal Feature Refinement: Existing works [8], [12] have shown that refining the segmentation mask over multi-level features can significantly improve the performance with respect to fine boundaries and completed objects. Instead of directly taking visual features from different level layers of the network, we propose to use multimodal features that have incorporated word-attentive interactions and spatial attentions.
As shown in Fig. 5, the decoder network progressively integrates these features from high-level to low-level semantics as follows:
where represents the hidden state of each time step over different level s in the refinement. The hidden state
at the last time step of Eq. 13 is adopted as the output of the decoder network. Finally,
is fed to another convolutional layer to produce a 2-D probability score map
normalized with sigmoid function. The probability score can be trained with a ground truth label map
by a binary cross entropy loss function as:
where is the whole set of pixels in the image and n is n-th pixel in it.
In this section, we introduce the datasets in Sec. VI-A and evaluation metrics in Sec. VI-B, and also describe the implementation details of the proposed approach in Sec. VI-C.
A. Datasets
We evaluate our model on four publicly available datasets including Google-Ref [32], UNC [33], UNC+ [33] and Referit [34]. All experiments are conducted under the same train and test split sets as [11].
The Google-Ref dataset is composed of 104,560 expressions referring, 54,822 objects and 26,711 images. These images and ground truth masks are collected from the MS COCO dataset [35] and referring expressions are annotated from Amazon Mechanical Turk. The referring expressions of the Google-Ref dataset are longer with an average length of 8.43 words compared with the other three datasets and multiple objects with the same category can appear in a single image.
The UNC dataset is also based on the MS COCO dataset which contains 19,994 images with 142,209 referring expressions for 50,000 objects. It is gathered by a two-player game [34] where one player annotates the image region according to the expression by other player interactively. Both location and appearance words can be used to describe the referred objects.
The UNC+ dataset is similar as the UNC dataset with a total of 141,564 expressions for 49,856 objects in19,992 images. The main difference from the UNC dataset is that location information is not allowed to refer to the object of interest. In other words, referring expressions purely rely on the appearance and context descriptions to describe referred objects.
The Referit dataset is built on IAPR TC-12 dataset [36] which includes masks for stuffs, e.g., “sky” and “ground”, in addition to objects. Same as the UNC and UNC+ datasets, referring expressions of Referit is also collected by the two-player game. It has 130,525 expressions referring to 96,654 object masks in 19,894 images in total.
B. Evaluation Metrics
Following the previous work [11], we use Intersection-over-union (IoU) and Precision@X (Prec@X) as the evaluation metrics. IoU measures the intersection area divides by the union area between ground-truth and a predicted segmentation mask averaged over all test data. To be more specific, let S
TABLE I ABLATION STUDY OF THE RELATIVE CONTRIBUTIONS OF DIFFERENT COMPONENTS OF THE PROPOSED NETWORK ON THE UNC VAL SET. THE FIRST THREE ROWS CORRESPOND TO VARIANTS OF OUR MODEL WHERE SOME COMPONENTS ARE REMOVED. THE 4TH ROW IS OUR MODEL. THE LAST ROW IS OUR MODEL WITH DCRF POSTPROCESSING. THE RESULTS SHOW THAT EACH COMPONENT OF OUR MODEL HELP IMPROVING THE PERFORMANCE.
TABLE II COMPARISON OF THE SEGMENTATION PERFORMANCE WITH THE STATE-OF-THE-ART METHODS IN TERM OF IOU. THE TOP FIVE METHODS ARE DIRECTLY EVALUATED BY TAKING THE OUTPUTS OF NETWORKS AND THE BOTTOM THREE RESULTS ARE POST-PROCESSED WITH DCRF.
and be the ground-truth and predicted segmentation masks, respectively. IoU is defined as the ratio between the intersection and the union of these two masks, i.e.
. For a more precise comparison, Prec@X is provided to evaluate detailed contributions in ablation study. It measures the percentage of test images which have higher IoU than a threshold X. We choose the value of X ranging from 0.5 to 0.9 with an interval of 0.1.
C. Implementation Details
We use DeepLab-101 network [2] with pre-trained weights from the Pascal VOC dataset [37] as previous works [8], [9], [11], [21] to extract visual features. The input image is resized and zero-padded to and keep the maximum length of the referring expression as 20, so the spatial dimensions of the visual features from {Res5, Res4, Res3} are the same as W = H = 40 thanks to dilated convolution. The visual features at different levels are transformed to a fixed channel size of
using
convolution. Every word is first embedded to a 1000 dimensional vector and then passed through a bidirectional LSTM with the cell size of 500. After combining both forward and backward hidden outputs, the dimension of the language feature for every word is
. The cell sizes of E-ConvLSTM and D-ConvLSTM are set to be 1000 and 500, respectively. Furthermore, we apply DCRF [38] which is a widely used post-processing operation for precise segmentation masks. In order to minimize the loss function, the network is trained with Adam optimization algorithm [39] with initial learning rate of 0.00025, weight decay of 0.0005. We employ a “poly” strategy [2] to adaptively tune the learning rate with power of 0.9.
Fig. 6. Comparison of segmentation performance of different lengths of referring expressions on Google-Ref.
We perform ablation study in Sec. VII-A to evaluate the relative contributions of various components of the proposed network. We also present quantitative and qualitative results for comparisons with other state of the art methods in Sec. VII-B and Sec. VII-C, respectively.
A. Ablation Study
To verify the effectiveness of each component of our dual convolutional LSTM network, we first conduct ablation experiments on the UNC validation dataset with the following different variants of the proposed method.
Fig. 7. Visual comparison of referring image segmentation results: (a) original image; (b,c,d,e,f) referring image segmentation masks from [11], [8], [21] and our network without or with DCRF; (g) ground-truth segmentation mask. The corresponding referring expressions are shown above these images in each row.
• E-ConvLSTM(w/o word attention): This model does not use multi-level segment decoder to iteratively refinement of the segmentation mask. Instead the segmentation mask is directly predicted from the output of the multimodal feature encoder. This model also does not use word attentions.
• E-ConvLSTM: This is similar to the previous model, but uses word attentions to adaptively encode the multimodal interaction.
• E-ConvLSTM + D-ConvLSTM(w/o spatial attention): This is similar to the proposed model, but without the spatial attentions for the multimodal feature maps.
• E-ConvLSTM + D-ConvLSTM: This is the complete proposed model.
• E-ConvLSTM + D-ConvLSTM + DCRF: This is our proposed model with DCRF post-processing.
The results of the ablation study are shown in Table V. It can be seen that introducing word attentions into the cell state of E-ConvLSTM leads to better multimodal interaction result for multimodal feature encoder. In addition, multimodal feature refinement at multiple levels by the multi-level segment decoder considerably improves segmentation performance compared with encoder-only methods shown in the first two rows of Table V. Spatial attention further improves the performance due to a stronger feature representation that focuses on important spatial regions. DCRF achieves more precise segmentation masks in the final result. In summary, each component of our model contributes to improve the segmentation performance.
Fig. 8. Visualization on how E-ConvLSTM works for multimodal interaction: (a) original image; (b,c,d) intermediate activation by mean-pooling hidden state output after each underline word has been processed by the encoder network; (e) predicted segmentation mask; (f) ground truth segmentation mask.
Fig. 9. Visualization to illustrate the hidden states of D-ConvLSTM after refining with low-level features gradually: (a) original image; (b,c,d) intermediate activation by mean-pooling hidden state outputs; (e) predicted segmentation mask; (f) ground-truth segmentation mask.
Fig. 10. Visualization of the feature representation. These spatial heatmaps show the responses of our network to the diverse referring expressions that are not included in the dataset.
B. Quantitative Results
We quantitatively compare our model (with or without DCRF post-processing) with state-of-the-art referring image segmentation methods including RMI [11], DMN [12], KWA [9], RRN [8] and CMSA [21] in Table V.
The proposed method outperforms all other methods consistently on all datasets with or without DCRF post-processing. Specifically, the performance improvement on Google-Ref is particularly significant. As we mentioned in Sec. VI-A, the Google-Ref dataset is much more challenging since it has
Fig. 11. Comparison of visualization results of our attention model with the attention method used in image captioning. (a) original image; (b) visualized result of the attention model in [17]; (c) visualized result of our E-ConvLSTM; (d) ground truth segmentation mask.
Fig. 12. Comparison of visualization results of our network with or without the spatial feature. (a) original image; (b) visualized result without spatial features; (c) visualized result with spatial features; (d) ground truth segmentation mask.
longer referring expressions and richer descriptions compared with other datasets. This demonstrates the importance of the proposed E-ConvLSTM which adaptively focuses on more important words during the multimodal interaction. It plays an important role in understanding the referring expression and localizing the referred object simultaneously. The other methods may fail to capture the long-range dependency since considering each word equally such as RMI [11] and RRN [8] or without sequentially multimodal interaction such as KWA [9] and CMSA [21]. In addition, the approaches with multi-level refinement e.g., RRN [8], CMSA [21] and our model achieve better results than other compared methods. It demonstrates that multi-level features help segment objects with clear boundaries and our model benefits from the DConvLSTM decoder with spatial attentions.
In order to clearly show the effect of the length of referring expressions for different models, we follow [11] and evaluate the performance separately for four different groups of the referring expression length range at [1-5], [6-7], [8-10] and [11-20]. As illustrated in Fig. 6, our model outperforms other methods in all cases. The performance gain is particularly noticeable with longer referring expressions. This also manifests the advantage of the proposed E-ConvLSTM encoder which embeds word attentions in the ConvLSTM cell to adaptively encode the multimodal interaction in understanding longer and more complicated referring expressions.
C. Qualitative Results
We show some qualitative examples of the segmentation masks generated by our approach and compare with existing state-of-the-art methods in Fig. 7. We can see that our method can successfully handle appearance attributes (the 1st and 2nd examples) where the complete object has internal high contrasts (white car of the 1st example) and low contrasts to the neighboring object (the brown cow and the shady part of nearby cow of 2nd example). The compared methods shown in the column (b), (c), (d) fail to capture the complete objects or suppress the background clearly. In addition, in the 3rd and 4th examples, our method can accurately identify the objects of interest from complicated referring expressions with relative relationships between homogeneous objects (the container of the 3rd example and the portion of building of the 4th example). A complete object with heterogeneous components is well-identified in the 5th example and large sky regions are obtained with clear boundaries in the bottom example, while the other methods may not be able to accurately understand the complex referring expressions and obtain precise referred objects. The DCRF post-processing can further refine the segmentation mask as a whole without noisy background regions (the 2nd and 6th examples) and recover precise boundary details (the 3rd and 5th examples).
We also present some visualization examples in Fig. 8 and Fig. 9 to help understand how E-ConvLSTM localizes the referred objects along multimodal interaction and how D-ConvLSTM refines segmentation results with multi-level features, respectively. The reddish blocks under each word presents their word attentions in the expression. The intermediate visualized results are extracted from the hidden state tensors of E-ConvLSTM and D-ConvLSTM, which are meanpooled along channel dimension with normalization and resized to the original image resolution. We show the visualization examples after seen underlined word for Fig. 8 and after refining features with each level for Fig. 9. It can be observed that E-ConvLSTM can capture the corresponding concepts as far as words have been seen such as “colored”, “umbrella” in the first row and “glass” in the second row of Fig. 8. Once it interacts with more concrete descriptions in the expression, the network pinpoints to the correct referred objects. For the examples in Fig. 9, the high-level features is capable of roughly identifying the object of interest and gradually refine the response with fine boundary from the second column to fourth column. Fig. 10 presents visualization results when our network responses to diverse referring expressions that are not included in the dataset. The purpose of showing these examples is to verify the generalization ability of our network. The visualization results show that our network can effectively handle different concepts appearing in the expressions and accurately identify the referred objects under different semantics.
Attention models have been widely used and proven important for image captioning [17], [40] to relate a part of active region to each generated word in the caption. These attentionbased methods follow a similar encoder-decoder framework which encodes spatial image features and the hidden state of the word feature to produce the visual context, then decodes the context to predict the next word instead of a segmentation mask in referring image segmentation task. Their methods also have the ability to visualize the related regions of the image through word and context information to illustrate which image region the model is focusing on. In order to compare with attention model of image captioning for visual and linguistic features, we adopt the soft attention model [17] in our encoder network and show visualization results in Fig. 11. It can be observed that the highlighted regions by the attention model of image captioning shown in Fig. 11 (b) are more scattered and do not present a clear boundary of the referred object-of-interest since their method loses spatial information when producing the visual context. These attention maps might be sufficient for the image captioning task, but do not provide precise spatial information needed in the referring image segmentation task. As shown in Fig. 11 (c), our E-ConvLSTM precisely identifies the referred object and suppresses the background regions, since our encoder network can keep the spatial information of the features and embed the word attention in the multimodal interaction by ConvLSTM to adaptively localize the referred objects.
We further provide qualitative examples to illustrate the effect of the spatial features. Fig. 12 presents visualization results generated by our network with and without the spatial features. As shown in Fig. 12 (b), the network without the spatial feature can be confused by the objects with similar appearance, while they can be clearly discriminated by the network with spatial features. For example, the referred hotdog is located at the bottom of the another as shown in the first row and the desired donut is at the relative location compared to other donuts in the second row.
Some examples of failure cases are presented in Fig. 13. In the first example, our network may have failed to identify the exact object due to the language ambiguity where the predicted segmentation mask points to a sparrow different from the ground truth. The second example shows the case where our method correctly understands the referring expression and recognize the attributes of a person, e.g., head, hand and leg, but incorrectly segments the leg on the right which actually belongs to another person on the left. In the third example, our segmentation result is negatively affected by the heavy occlusion to find the bear. In the last example, the failure is caused by some internal gaps within the broccoli that misleads our network to consider they are apart and only segment the left part of the referred broccoli. We are going to explore fine-grained recognition technique to detect these subtle details in the future.
We present a novel dual convolutional LSTM network for the task of referring image segmentation. The E-ConvLSTM
Fig. 13. Some examples of failure cases of our network. (a) original image; (b) predicted segmentation mask; (c) ground truth segmentation mask.
encodes the multimodal feature along word sequence to localize the referred objects and D-ConvLSTM decodes the encoded multimodal information at multiple levels for mask refinement. Extensive experiments on four datasets shows consistent improvement achieved by the proposed network.
[1] E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional networks for semantic segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 4, pp. 640–651, 2017.
[2] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 4, pp. 834–848, 2018.
[3] K. He, G. Gkioxari, P. Doll´ar, and R. Girshick, “Mask r-cnn,” in IEEE International Conference on Computer Vision, 2017, pp. 2961–2969.
[4] M.-M. Cheng, N. J. Mitra, X. Huang, P. H. Torr, and S.-M. Hu, “Global contrast based salient region detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 3, pp. 569–582, 2014.
[5] L. Ye, Z. Liu, L. Li, L. Shen, C. Bai, and Y. Wang, “Salient object segmentation via effective integration of saliency and objectness,” IEEE Transactions on Multimedia, vol. 19, no. 8, pp. 1742–1756, 2017.
[6] Y. Li, X. Hou, C. Koch, J. M. Rehg, and A. L. Yuille, “The secrets of salient object segmentation,” in IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 280–287.
[7] R. Hu, M. Rohrbach, and T. Darrell, “Segmentation from natural language expressions,” in European Conference on Computer Vision. Springer, 2016, pp. 108–124.
[8] R. Li, K. Li, Y.-C. Kuo, M. Shu, X. Qi, X. Shen, and J. Jia, “Referring image segmentation via recurrent refinement networks,” in IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 5745–5753.
[9] H. Shi, H. Li, F. Meng, and Q. Wu, “Key-word-aware network for referring expression image segmentation,” in European Conference on Computer Vision, 2018, pp. 38–54.
[10] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
[11] C. Liu, Z. Lin, X. Shen, J. Yang, X. Lu, and A. L. Yuille, “Recurrent multimodal interaction for referring image segmentation,” in IEEE International Conference on Computer Vision, 2017, pp. 1271–1280.
[12] E. A. Margffoy-Tuay, J. C. P´erez, E. Botero, and P. Arbel´aez, “Dynamic multimodal instance segmentation guided by natural language queries,” in European Conference on Computer Vision, 2018, pp. 630–645.
[13] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” in Advances in Neural Information Processing Systems, 2015, pp. 802–810.
[14] C. Rother, V. Kolmogorov, and A. Blake, “Grabcut: Interactive fore- ground extraction using iterated graph cuts,” in ACM transactions on graphics, vol. 23, no. 3. ACM, 2004, pp. 309–314.
[15] Z. Liu, R. Shi, L. Shen, Y. Xue, K. N. Ngan, and Z. Zhang, “Unsu- pervised salient object segmentation based on kernel density estimation and two-phase graph cut,” IEEE Transactions on Multimedia, vol. 14, no. 4, pp. 1275–1289, 2012.
[16] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, 2012.
[17] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in International Conference on Machine Learning, 2015, pp. 2048–2057.
[18] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh, “Vqa: Visual question answering,” in IEEE International Conference on Computer Vision, 2015, pp. 2425–2433.
[19] A. Rohrbach, M. Rohrbach, R. Hu, T. Darrell, and B. Schiele, “Ground- ing of textual phrases in images by reconstruction,” in European Conference on Computer Vision. Springer, 2016, pp. 817–834.
[20] L. Yu, Z. Lin, X. Shen, J. Yang, X. Lu, M. Bansal, and T. L. Berg, “Mattnet: Modular attention network for referring expression comprehension,” in IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1307–1315.
[21] L. Ye, M. Rochan, Z. Liu, and Y. Wang, “Cross-modal self-attention network for referring image segmentation,” in IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 10 502–10 511.
[22] G. Underwood, L. Jebbett, and K. Roberts, “Inspecting pictures for in- formation to verify a sentence: Eye movements in general encoding and in focused search,” The Quarterly Journal of Experimental Psychology Section A, vol. 57, no. 1, pp. 165–182, 2004.
[23] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in neural information processing systems, 2014, pp. 3104–3112.
[24] E. Marchi, G. Ferroni, F. Eyben, L. Gabrielli, S. Squartini, and B. Schuller, “Multi-resolution linear prediction based features for audio onset detection with bidirectional lstm neural networks,” in IEEE international conference on acoustics, speech and signal processing, 2014, pp. 2164–2168.
[25] N. Srivastava, E. Mansimov, and R. Salakhudinov, “Unsupervised learn- ing of video representations using lstms,” in International conference on machine learning, 2015, pp. 843–852.
[26] A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional lstm and other neural network architectures,” Neural networks, vol. 18, no. 5-6, pp. 602–610, 2005.
[27] M. Sundermeyer, T. Alkhouli, J. Wuebker, and H. Ney, “Translation modeling with bidirectional recurrent neural networks,” in Conference on Empirical Methods in Natural Language Processing, 2014, pp. 14– 25.
[28] S. Zhang, X. Liu, and J. Xiao, “On geometric features for skeleton-based action recognition using multilayer lstm networks,” in IEEE Winter Conference on Applications of Computer Vision. IEEE, 2017, pp. 148– 157.
[29] A. Gensler, J. Henze, B. Sick, and N. Raabe, “Deep learning for solar power forecastingan approach using autoencoder and lstm neural networks,” in IEEE international conference on systems, man, and cybernetics. IEEE, 2016, pp. 2858–2865.
[30] S. Zhang, G. Wu, J. P. Costeira, and J. M. Moura, “Fcn-rlstm: Deep spatio-temporal neural networks for vehicle counting in city cameras,” in IEEE International Conference on Computer Vision, 2017, pp. 3667– 3676.
[31] C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun, “Large kernel matters– improve semantic segmentation by global convolutional network,” in IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4353–4361.
[32] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, “Generation and comprehension of unambiguous object descriptions,” in
IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 11–20.
[33] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, “Modeling context in referring expressions,” in European Conference on Computer Vision. Springer, 2016, pp. 69–85.
[34] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg, “Referitgame: Referring to objects in photographs of natural scenes,” in Conference on Empirical Methods in Natural Language Processing, 2014, pp. 787– 798.
[35] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European Conference on Computer Vision. Springer, 2014, pp. 740–755.
[36] H. J. Escalante, C. A. Hern´andez, J. A. Gonzalez, A. L´opez-L´opez, M. Montes, E. F. Morales, L. E. Sucar, L. Villase˜nor, and M. Grubinger, “The segmented and annotated iapr tc-12 benchmark,” Computer Vision and Image Understanding, vol. 114, no. 4, pp. 419–428, 2010.
[37] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisser- man, “The pascal visual object classes (voc) challenge,” International Journal of Computer Vision, vol. 88, no. 2, pp. 303–338, 2010.
[38] P. Kr¨ahenb¨uhl and V. Koltun, “Efficient inference in fully connected crfs with gaussian edge potentials,” in Advances in Neural Information Processing Systems, 2011, pp. 109–117.
[39] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations, 2014.
[40] J. Lu, C. Xiong, D. Parikh, and R. Socher, “Knowing when to look: Adaptive attention via a visual sentinel for image captioning,” in IEEE Conference on Computer Vision and Pattern Recognition, 2017.