Grounding natural language in visual contents is a fundamental and vital task in the visual-language understanding field. Visual grounding aims to localize the object described by the given referring expression in an im-
Declarative Sentence: A little boy with a Christmas hat is catching a yellow toy. Interrogative Sentence: What is caught by the squatting boy on the floor?
Figure 1. An example of STVG for multi-form sentences.
age, which has attracted much attention and made great progress [14, 25, 6, 42, 39]. Recently, researchers begin to explore video grounding, including temporal grounding and spatio-temporal grounding. Temporal sentence grounding [8, 11, 48, 37] determines the temporal boundaries of events corresponding to the given sentence, but does not localize the spatio-temporal tube (i.e., a sequence of bounding boxes) of the described object. Further, spatio-temporal grounding is to retrieve the object tubes according to textual descriptions, but existing strategies [50, 1, 38, 4] can only be applied to restricted scenarios, e.g. grounding in a frame of the video [50, 1] or grounding in trimmed videos [38, 4]. Moreover, due to the lack of bounding box annotations, researchers [50, 4] can only adopt a weakly-supervised setting, leading to suboptimal performance.
To break through above restrictions, we propose a novel task, Spatio-Temporal Video Grounding for Multi-Form Sentences (STVG). Concretely, as illustrated in Figure 1, given an untrimmed video and a declarative/interrogative sentence depicting an object, STVG aims to localize the spatio-temporal tube of the queried object.
Compared with previous video grounding [50, 1, 38, 4], STVG has two novel and challenging points. First, we localize spatio-temporal object tubes from untrimmed videos. The objects may exist in a very small segment of the video and be hard to distinguish. And the sentences may only describe a short-term state of the queried object, e.g. the action ”catching a toy” of the boy in Figure 1. So it is crucial to determine the temporal boundaries of object tubes by sufficient cross-modal understanding. Secondly, STVG deals with multi-form sentences, that it, not only grounds the conventional declarative sentences with explicit objects, but also localizes the interrogative sentences with unknown objects, for example, the sentence ”What is caught by the squatting boy on the floor?” in Figure 1. Due to the lack of the explicit characteristics of objects (e.g. the class ”toy” and visual appearance ”yellow”), grounding for interrogative sentences can only depend on relationships between the unknown object and other objects (e.g. the action relation ”caught by the squatting boy” and spatial relation ”on the floor”). Thus, sufficient relationship construction and cross-modal relation reasoning are crucial for the STVG task.
Existing video grounding methods [38, 4] often extract a set of spatio-temporal tubes from trimmed videos and then identify the target tube that matches the sentence. However, this framework may be unsuitable for STVG. On the one hand, the performance of this framework is heavily dependent on the quality of tube candidates. But it is difficult to pre-generate high-quality tubes without textual clues, since the sentences may describe a short-term state of objects in a very small segment, but the existing tube pre-generation framework [38, 4] can only produce the complete object tubes from trimmed videos. On the other hand, these methods only consider single tube modeling and ignore the relationships between objects. However, object relations are vital clues for the STVG task, especially for interrogative sentences that may only offer the interactions of the unknown objects with other objects. Thus, these approaches cannot deal with the complicated grounding of STVG.
To tackle above problems, we propose a novel SpatioTemporal Graph Reasoning Network (STGRN) to capture region relationships with temporal object dynamics and directly localize the spatio-temporal tubes without tube pre-generation. Specifically, we first parse the video into a spatio-temporal region graph. Existing visual graph modeling approaches [41, 19] often build the spatial graph in an image, which cannot utilize the temporal dynamics information in videos to distinguish the subtle differences of object actions, e.g. distinguish ”open the door” and ”close the door”. Different from them, our spatio-temporal region graph not only involves the implicit and explicit spatial subgraphs in each frame, but also includes a temporal dynamic subgraph across frames. The spatial subgraphs can capture the region-level relationships by implicit or explicit semantic interactions, and the temporal subgraph can incorporate the dynamics and transformation of objects across frames to further improve the relation understanding. Next, we fuse the textual clues into this spatio-temporal graph as the guidance, and develop the multi-step cross-modal graph reasoning. The multi-step process can support the multi-order relation modeling like ”a man hug a baby wearing a red hat”. After it, we introduce a spatio-temporal localizer to directly retrieve the spatio-temporal object tubes from the region level. Concretely, we first employ a temporal localizer to
determine the temporal boundaries of the tube, and then apply a spatial localizer with a dynamic selection method to ground the object in each frame and generate a smooth tube. To facilitate this STVG task, we contribute a large-scale video grounding dataset VidSTG by adding the multi-form sentence annotations into video relation dataset VidOR. Our main contributions can be summarized as follows:
• We propose a novel task STVG to explore the spatio-temporal video grounding for multi-form sentences.
• We develop a novel STGRN to tackle this STVG task, which builds a spatio-temporal graph to capture the region relationships with temporal object dynamics, and employs a spatio-temporal localizer to directly retrieve the spatio-temporal tubes without tube pre-generation.
• We contribute a large-scale video grounding dataset VidSTG as the benchmark of the STVG task.
• The extensive experiments show the effectiveness of our proposed STGRN method.
2.1. Temporal Localization via Natural Language
Temporal natural language localization is to detect the video clip depicting the given sentence. Early approaches [11, 8, 12, 23, 24] are mainly based on the sliding window framework, which first samples abundant candidate clips and then ranks them. Recently, people begin to develop holistic and cross-modal methods [46, 2, 3, 35, 48, 37] to solve this problem. Chen, Zhang and Xu et al. [2, 3, 48, 22, 37] build frame-by-word interactions from visual and textual contents to aggregate the matching clues. Zhang et al. [46] deal with the structural and semantic misalignment challenges by an explicitly structured graph. Wang et al. [35] propose a reinforcement learning framework to adaptively observe frame sequences and associate video contents with sentences. Further, Mithun and Lin et al. [26, 21] devise the weakly-supervised temporal localization methods requiring only coarse video-level annotations for training. And besides the natural language query, Zhang et al. [49] attempt to localize the unseen temporal clip according to an image query. Although these methods have achieved promising performance, they still remain in the temporal grounding. We further explore spatio-temporal grounding in this paper.
2.2. Object Grounding in Images/Videos
Visual grounding [14, 25, 36, 43, 27, 44, 6, 51, 13, 42, 47, 39, 40] aims to localize the visual object described by the given referring expression. Early methods [14, 25, 36, 43, 27, 44] often extract object features
Figure 2. The Overall Architecture of the Spatio-Temporal Graph Reasoning Network (STGRN). We first extract the region features and learn the query representation. Next, we apply a spatio-temporal graph encoder to develop the multi-step cross-modal graph reasoning. After it, a spatio-temporal localizer with T spatio-temporal convolution layers directly retrieves the tubes from the region level.
by CNN, model the language expression through RNN and learn the object-language matching. Some recent approaches [42, 13] decompose the expression into multiple components and calculate matching scores of each module. And Deng and Zhuang et al. [6, 51] apply the co-attention mechanism to build cross-modal interactions. Further, Yang et al. [40, 39] explore the relationships between objects to improve the accuracy. As for video grounding, existing works [50, 1, 38, 4, 38, 4] can only be applied to restricted scenarios. Zhou and Balajee et al. [50, 1] only ground natural language in a frame of the video. With a sequence of transcriptions and temporal alignment video clips, Huang and Shi et al. [15, 33] ground nouns or pronouns in specific frames by the weakly-supervised MIL methods. And Chen and Yamaguchi et al. [38, 4] localize spatio-temporal object tubes from trimmed videos, but cannot directly deal with raw video streams. In this paper, we propose a novel STVG task to further explore the spatio-temporal video grounding for multi-form sentences.
Given a video and a sentence
depicting an object, STVG is to retrieve its spatio-temporal tube. Figure 2 illustrates the overall framework.
3.1. Video and Text Encoder
We first apply a pre-trained Faster R-CNN [30] to extract a set of regions for each frame, where a video contains N frames and the t-th frame corresponds to K regions denoted by . A region
is associated with its visual feature
and bounding box vector
, where
are the normalized coordinates of the center point and
are the normalized width and height. Be- sides, we obtain the frame features
, i.e., the visual feature of the entire frame from Faster R-CNN.
For the sentence s with L words, we first input the word embeddings into a bi-directional GRU [5] to learn the word semantic features , where
is the concate- nation of the forward and backward hidden states of step i. As the STVG grounds objects described in the declarative or interrogative sentences, we need to extract the query representation from language context. First, we select the entity feature
from
, which represents the queried object, for example,
for ”boy” in Figure 2. Note that for interrogative sentences, the feature of ”who” or ”what” is chosen as the entity feature. Next, an attention method aggregates the textual clues from language context by
where and
are projection matrices,
is the entity- aware feature and
is the query representation.
3.2. Spatio-Temporal Graph Encoder
Our STVG task requires to capture the object relationships and develop the cross-modal understanding, especially for interrogative sentences that may only offer the interaction information of the unknown object with other objects. Thus, we build a spatio-temporal region encoder with T spatio-temporal convolution layers to capture the region relationships with temporal object dynamics and support the multi-step cross-modal graph reasoning.
3.2.1 Graph Construction
We first parse the video into a spatio-temporal region graph, which involves the implicit spatial subgraph , explicit spatial subgraph
in each frame and temporal dynamic subgraph
across frames. Three subgraphs all treat regions as their vertexes V but have different edges. Note that we add the self-loop of each vertex in each subgraph.
Implicit Spatial Graph. We regard the fully-connected region graph in each frame as the implicit spatial graph , where
contains
undirected and unlabeled edges in each frame (including self-loops).
Explicit Spatial Graph. We extract the region triplet to construct the explicit spatial graph, where
and
are the i-th and j-th regions in frame t, and
is the relation predicate between them. Each triplet can be regarded as an edge from i to j. Thus, explicit graph construction can be formulated as a relation classification task [41, 45]. Concretely, given the feature
of re- gion i, feature
of region j and the united feature
of the union bounding box of i and j (also ex- tracted by Faster R-CNN), we first transform three features via different linear leyers and then concatenate them into a classification layer to predict the relations. Similer to existing works [41, 19], we train such a classifier on the Visual Genome dataset [18], where we select top-50 frequent predicates in its training data and add an extra no relation class for non-existent edges. We then predict the relationships between
and
. Eventually, the edges
have 3 directions (including i-to-j, j-to-i and i-to-i of self-loops) and 51 types of labels (top-50 classes plus the self-loop).
Temporal Dynamic Graph. While the spatial graphs model region-region interactions, our temporal graph is to capture the dynamics and transformation of objects across frames. So we expect to connect the regions containing the same object in different frames and then learn more expressive and discriminative object features. For the frame t, we connect its regions with adjacent 2M frames (M for forward frames and M for backward). Too distant frames cannot provide the real-time dynamics. Concretely, we first define the linking score between
from frame t and
from frame k by
where is the cosine similarity of two features,
is the intersection-over-union of two regions, and is the balanced scalar. Here we simultaneously consider the appearance similarity and spatial overlap ratio of two regions. And the temporal distance
of two frames is used to limit the IoU score, that is, for distant frames, the linking score is mainly determined by feature similarity rather than the spatial overlap. Next, for the region
, we select the region
with the maximal linking score from frame k to build an edge, and get 2M + 1 edges for each region (including the self-loop). The unlabeled temporal edges
have 3 directions: forward, backward and self-loop.
3.2.2 Multi-Step Cross-Modal Graph Reasoning
After graph construction, we incorporate the textual clues into this graph and develop the multi-step cross-modal graph reasoning by T spatio-temporal convolution layers.
Cross-Modal Fusion. To capture the relationships associated with the sentence, we first use a cross-modal fusion that dynamically injects textual evidences into the spatio-temporal graph. Concretely, we first utilize an attention mechanism to aggregate the words features for each region. For a region , we calculate the attention weights over word features
, denoted by
where ,
are projection matrices,
is the bias and
is the row vector. And
is the region-aware tex- tual feature for each region i in frame t.
Next, we build the textual gate that takes language information as the guidance to weaken the text-irrelevant regions, given by
where is the sigmoid function,
is the element-wise multiplication,
means the textual gate for re- gion
. And we then concatenate the filtered region feature and textual feature to obtain the cross-modal region features
. Next, we develop T spatio-temporal convo- lution layers for the multi-step graph reasoning.
Spatial Graph Convolution. In each layer, we first develop the spatial graph convolution to capture visual relationships among regions in each frame. Concretely, with cross-modal region features , we first adopt the implicit graph convolution on
that is undirected and unlabeled, given by
where are the regions connected with
in
. The implicit graph convolution can be regarded as a variant of self-attention and we compute the coefficient
by combining the visual features and region locations.
Simultaneously, we develop the explicit graph convolution. Different from the original undirected GCN [16, 34], we consider the direction and label information of edges on the directed and labeled , given by
where are optional matrices by the direction dir(i, j) of edge
are optional bias by the label of edge (i, j). Here, the edge has three directions (i-to-j, j-to-i, i-to-i) and 51 types.
are the regions connected with
. Moreover, the relation coefficient
can also be cho- sen by the label of edge (i, j). Different sentences describe different relations and their grounding is heavily dependent on the specific relation understanding. Thus, the coefficient of explicit edges can be decided by query presentation
, given by
where corresponds to the coefficients of 51 types of relationships.
Temporal Graph Convolution. We next develop the temporal graph convolution on the directed and unlabeled graph to capture the dynamics and transformation of objects across frames. We consider the forward, backward and self-loop edges for each region
with the cross-modal feature
, denoted by
where and
are matrices and dir(i, j) indicates the direction of edge (i, j) to select the corresponding projection matrix, where the temporal edges have three directions. And
is the semantic coefficient for each neighborhood region.
Next, we combine the outputs of spatial and temporal graph convolutions and obtain the result of the first spatio-temporal convolution layer by
In order to support multi-order relation modeling, we perform the multi-step encoding by the spatio-temporal graph encoder with T spatio-temporal convolution layers and learn final relation-aware region features .
3.3. Spatio-Temporal Localizer
In this section, we devise a spatio-temporal localizer to determine the temporal tube boundaries and spatio-temporal tubes of objects from the region level.
Temporal Localizer. We first introduce the temporal localizer, which estimates a set of candidate clips and adjust their boundaries to obtain the temporal grounding [48]. Specifically, we first aggregate the relation-aware region graph into the frame level by an attention mechanism. With the query representation , the region features of each frame are attended by
where represents the relation-aware feature of frame t. We then concatenate these features with their corresponding global frame features
, and apply another BiGRU to learn final frame features
. Next, we define multi-scale candidate clips at each time step t as
, where
are the start and end boundaries of the i-th clips,
is the temporal length of i-th clip and P is the clip number. After it, we estimate all candidate clips by a linear layer with the sigmoid nonlinearity and simultaneously produce the offsets of their boundaries, given by
where corresponds to confidence scores of P candidates at step t and
are the offsets of P clips.
The temporal localizer has two losses: the alignment loss for the clip selection and a regression loss for boundary adjustments. Concretely, for alignment loss, we first compute the temporal IoU score of each clip with the ground truth. And the alignment loss is denoted by
where we use the temporal IoU score rather than 0/1 score to further distinguish high-score clips. Next, we fine-tune the boundaries of the best clip with highest
, which has the boundaries (s, e) and offsets
. We first compute the offsets of this clip from ground truth boundaries
by
and
and define the regression loss by
where R represents the smooth L1 function. Spatial Localizer. With the temporal grounding, we next localize the target regions in each frame. For the t-th
frame with region features , we directly estimate the matching scores of each region by integrating the query representation
and final frame feature
, denoted by
where is the matching score of region i of frame t. Simi- lar to temporal alignment loss, the spatial loss first compute the spatial IoU score
for each region with the ground truth region, where the frames outside the temporal ground truth are omitted. And the spatial loss is denoted by
(15) where is the set of frames in the temporal ground truth. Eventually, we devise a multi-task loss to train our proposed STGRN in an end-to-end manner, given by
where and
are the hyper-parameters to control the balance of three losses.
3.4. Dynamic Selection Method
During inference, we first retrieve the temporal boundaries of the tube from the temporal localizer and then determine the grounded region for each frame by the spatial localizer. A greedy method directly selects the regions with highest matching scores
. However, such generated tubes may not be very smooth. The bounding boxes between adjacent frames may have too large displacements. Thus, to make the trajectory smoother, we introduce a dynamic selection method. Concretely, we first define the linking score
between regions of successive frames t and t + 1 by
where and
are matching scores of regions
and
, and
is the balanced scalar which is set to 0.2. Next, we generate the final spatio-temporal tube Y with the maximal energy E given by
where are the temporal boundaries and we solve this optimization problem using a Vitervi algorithm [10].
As a novel task, STVG lacks a suitable dataset as the benchmark. Therefore, we contribute a large-scale spatio-temporal video grounding dataset VidSTG by augmenting the sentence annotations on VidOR [32].
Table 1. Dataset Statistics about the Number of Declarative and Interrogative Sentences.
4.1. Dataset Annotation
VidOR [32] is the existing largest object relation dataset, containing 10,000 videos and fine-grained annotations for objects and their relations. Specifically, VidOR annotates 80 categories of objects with dense bounding boxes and annotates 50 categories of relation predicates among objects (8 spatial relations and 42 action relations). Specifically, VidOR denotes a relation as a triplet and each triplet is associated with the temporal boundaries and spatio-temporal tubes of subject and object. Based on VidOR, we can select the suitable triplets, and describe the subject or object with multi-form sentences. Taking VidOR as the basic dataset has many advantages. On the one hand, we can avoid laborintensive annotations for bounding boxes. On the other hand, the relationships in the triplets can be simply incorporated into the annotated sentences.
We first split and clean the VidOR data, and then annotate the rest video-triplet pairs with multi-form sentences. The cleaning process is introduced in the supplementary material. For each video-triplet pair, we choose the subject or object as the queried object, and then describe its appearance, relationships with other objects and visual environments. For interrogative annotations, the appearance of queried objects is ignored. We discard video-triplet pairs that are too hard to give a precise description. And a video-triplet pair may correspond to multiple sentences.
4.2. Dataset Statistics
After annotation, there are 99,943 sentence descriptions about 79 types of queried objects for 44,808 video-triplet pairs, shown in Table 4. The average duration of videos is 28.01s and the average temporal length of object tubes is 9.68s. The average lengths of declarative and interrogative sentences are 11.12 and 8.98, respectively. Further, we provide the distribution of 79 types of queried objects and some annotation examples in the supplementary material.
5.1. Experimental Settings
Implementation Details. In STGRN, we first sample 5 frames per second and downsample the frame number of overlong videos to 200. We then pretrain the Faster R-CNN
Table 2. Performance Evaluation Results on the VidSTG Dataset.
on MSCOCO [20] to extract 20 region proposals for each frame (i.e. K = 20). The region feature dimension is 1,024 and we map it 256 before graph modeling. For sentences, we use a pretrained Glove word2vec [28] to extract 300-d word embeddings. As for the hyper-parameters, we set M to 5,
to 0.8,
to 0.2 and set
to 1.0, 0.001 and 1.0, respectively. The layer number T of the spatio-temporal graph encoder is set to 2. For the temporal localizer, we set P to 8 and define 8 window widths [8, 16, 32, 64, 96, 128, 164, 196]. We set the dimension of almost parameter matrices and bias to 256, including the
,
in the explicit graph convolution,
and
in the temporal localizer and so on. And the BiGRU networks have 128-d hidden states for each direction. During training, we apply an Adam optimizer [7] to minimize the multi-task loss
, where the initial learning rate is set to 0.001 and the batch size is 16.
Evaluation Criteria. We employ the m tIoU, m vIoU and vIoU@R as evaluation criteria [9, 4]. The m tIoU is the average temporal IoU between the selected clips and ground truth clips. And we define as the set of frames contained in the selected or ground truth clips, and
as the set of frames in both selected and ground truth clips. We calculate vIoU by
and
are selected and ground truth regions of frame t. The m vIoU is the average vIoU of samples and vIoU@R is the proportion of samples which vIoU > R.
Baseline. Since no existing strategy can be directly applies to STVG, we extend the existing visual grounding method GroundeR [31] and video grounding approaches STPR [38] and WSSTG [4] as the baselines. Considering these methods all lack temporal grounding, we first apply the temporal sentence localization methods TALL [8] and L-Net [3] to obtain a clip and then retrieve the tubes from the trimmed clip by GroundeR, STPR and WSSTG. The GroundeR is a frame-level approach, which originally grounds natural language in a still image. We apply it for each frame of the clip and generate a tube. The STPR and WSSTG are both tube-level methods and adopt the tube pre-generation framework. Specifically, the original STPR [38] only grounds persons from multiple videos, we extend it to multi-type object grounding in a single clip. The original WSSTG [4] employs a weakly-supervised setting, we extend it by applying a supervised triplet loss [39] to select candidate tubes. So we obtain 6 combined baselines GroundeR+TALL, STPR+TALL and so on. We also provide the temporal ground truth to form 3 baselines. We show more baseline details in the supplementary material.
5.2. Experiment Results
Table 2 shows the overall experiment results of all methods, where STGRN(Greedy) uses the greedy region selection for the tube generation rather than the dynamic method. The Random selects the temporal clip and spatial regions randomly. Tem. Gt means that the temporal ground truth is provided. We can find several interesting points:
• The GroundeR+methods independently ground sentences in every frame and achieve worse performance than STPR+
and WSSTG+
methods, validating the temporal object dynamics across frames are vital for spatio-temporal video grounding.
• The model performance on interrogative sentences is obviously lower than declarative sentences, which shows the interrogative sentences with unknown objects are more difficult to ground.
• For temporal grounding, our STGRN achieves a better performance than the frame-level localization method TALL and L-Net, demonstrating the spatio-temporal
Table 3. Ablation Results on the VidSTG Dataset.
region modeling is effective to determine the temporal boundaries of object tubes.
• For spatio-temporal grounding, our STGRN outperforms all baselines on both declarative and interrogative sentences with or without temporal ground truth, which suggests our cross-modal spatio-temporal graph reasoning can effectively capture the object relationships with temporal dynamics and our spatio-temporal localizer can retrieve the object tubes precisely.
• Our STGRN with the dynamic selection method outperforms the STGRN(Greedy) with the greedy method, showing the dynamic smoothness is benefi-cial to generate high-quality tubes.
5.3. Ablation Study
In this section, we conduct the ablation studies on the spatio-temporal region graph that is the key component of our STGRN. Concretely, the spatio-temporal graph includes the implicit spatial subgraph , explicit spatial subgraph
and temporal dynamic subgraph
. We selectively discard them to generate ablation models and report all ablation results in Table 3, where we do not distinguish the declarative and interrogative sentences. From these results, we can find that the full model outperforms all ablation models, validating each subgraph is helpful for spatio-temporal video grounding. If only a subgraph is applied, the model with
achieves the best performance, demonstrating the explicit modeling is the most important to capture object relationships. And if two subgraphs are used, the model with
and
outperforms other models, which suggests the spatio-temporal modeling play a crucial role in relation understanding and high-quality video grounding.
Moreover, the layer number T is the essential hyperparameter of the spatio-temporal graph. We investigate the effect of T by varying it from 1 to 5. Figure 3 shows the experimental results on the criteria m tIoU and m vIoU for both declarative and interrogative sentences. From the results, we can find our STGRN has the best performance when T is set to 2. The one-layer graph cannot sufficiently capture the object relationships and temporal dynamics. And too many layers may result in region over-smoothing, that is,
Figure 3. Effect of the Number of Spatio-Temporal Graph Convolution Layers.
Figure 4. A Example of the Spatio-Temporal Grounding Result.
each region feature tends to be identical. The performance changes on different criteria and sentence types are basically consistent, demonstrating the stable influence of T.
5.4. Qualitative Analysis
We display a typical example in Figure 4. The sentence describes two parallel actions ”grab the hands” and ”jump off the slide” of the boy in a short-term segment, requiring the accurate spatio-temporal grounding. By intuitive comparison, our STRGN gives a more precise temporal segment and generates a more reasonable spatio-temporal tube than the baseline WSSTG+L-net. Moreover, the attention method in the cross-modal fusion module builds a bridge between visual and textual contents, and we visualize the weights of several key regions over the sentences here. We can see that semantic related region-word pairs have a larger weight, e.g., the region of the boy and the word ”child”.
In this paper, we propose a novel spatio-temporal video grounding task STVG and contribute a large-scale dataset VidSTG. We then design a STGRN to capture region relationships with temporal object dynamics and directly localize the spatio-temporal tubes from the region level.
Acknowledgments This work was supported by Zhejiang Natural Science Foundation LR19F020006 and the National Natural Science Foundation of China under Grant No.61836002, No.U1611461 and No.61751209, Sponsored by China Knowledge Centre for Engineering Sciences and Technology and Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies.
[1] Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Object referring in videos with language and human gaze. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4129–4138, 2018.
[2] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat- Seng Chua. Temporally grounding natural sentence in video. In EMNLP, pages 162–171. ACL, 2018.
[3] Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. Localizing natural language in videos. In AAAI, 2019.
[4] Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan-Yee K Wong. Weakly-supervised spatio-temporally grounding natural sentence in video. 2019.
[5] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS, 2014.
[6] Chaorui Deng, Qi Wu, Qingyao Wu, Fuyuan Hu, Fan Lyu, and Mingkui Tan. Visual grounding via accumulated attention. In CVPR, pages 7746–7755, 2018.
[7] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159, 2011.
[8] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: temporal activity localization via language query. In ICCV, pages 5277–5285. IEEE, 2017.
[9] Jiyang Gao, Zhenheng Yang, Chen Sun, Kan Chen, and Ram Nevatia. Turn tap: Temporal unit regression network for temporal action proposals. 2017.
[10] Georgia Gkioxari and Jitendra Malik. Finding action tubes. In CVPR, pages 759–768, 2015.
[11] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, pages 5803– 5812, 2017.
[12] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with temporal language. In EMNLP, pages 1380–1390. ACL, 2018.
[13] Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko. Modeling relationships in referential expressions with compositional modular networks. In CVPR, pages 1115–1124, 2017.
[14] Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In CVPR, pages 4555–4564, 2016.
[15] De-An Huang, Shyamal Buch, Lucio Dery, Animesh Garg, Li Fei-Fei, and Juan Carlos Niebles. Finding ”it”: Weaklysupervised reference-aware visual grounding in instructional videos. In CVPR, June 2018.
[16] Thomas N Kipf and Max Welling. Semi-supervised classifi- cation with graph convolutional networks. In ICLR, 2016.
[17] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, pages 706–715, 2017.
[18] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2017.
[19] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation- aware graph attention network for visual question answering. In ICCV, 2019.
[20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014.
[21] Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, and Huasheng Liu. Weakly-supervised video moment retrieval via semantic completion network. AAAI, 2020.
[22] Zhijie Lin, Zhou Zhao, Zhu Zhang, Zijian Zhang, and Deng Cai. Moment retrieval via cross-modal interaction networks with query reconstruction. IEEE Transactions on Image Processing, 29:3750–3762, 2020.
[23] Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Bao- quan Chen, and Tat-Seng Chua. Attentive moment retrieval in videos. In SIGIR, pages 15–24. ACM, 2018.
[24] Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan Chen, and Tat-Seng Chua. Cross-modal moment localization in videos. In MM, pages 843–851. ACM, 2018.
[25] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, pages 11–20, 2016.
[26] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy- Chowdhury. Weakly supervised video moment retrieval from text queries. In CVPR, pages 11592–11601, 2019.
[27] Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Mod- eling context between objects for referring expression understanding. In ECCV, pages 792–807. Springer, 2016.
[28] Jeffrey Pennington, Richard Socher, and Christopher Man- ning. Glove: Global vectors for word representation. In EMNLP, pages 1532–1543, 2014.
[29] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Grounding action descriptions in videos. Transactions of the Association of Computational Linguistics, 1:25–36, 2013.
[30] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, pages 91–99, 2015.
[31] Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In ECCV, pages 817–834. Springer, 2016.
[32] Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations in usergenerated videos. In ICMR, pages 279–287. ACM, 2019.
[33] Jing Shi, Jia Xu, Boqing Gong, and Chenliang Xu. Not all frames are equal: Weakly-supervised video grounding with contextual similarity and visual clustering losses. In CVPR.
[34] Petar Veliˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR, 2017.
[35] Weining Wang, Yan Huang, and Liang Wang. Language- driven temporal activity localization: A semantic matching reinforcement learning model. In CVPR, pages 334–343, 2019.
[36] Fanyi Xiao, Leonid Sigal, and Yong Jae Lee. Weaklysupervised visual grounding of phrases with linguistic structures. In CVPR, pages 5945–5954, 2017.
[37] Huijuan Xu, Kun He, L Sigal, S Sclaroff, and K Saenko. Multilevel language and vision integration for text-to-clip retrieval. In AAAI, volume 2, page 7, 2019.
[38] Masataka Yamaguchi, Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Spatio-temporal person retrieval via natural language queries. In ICCV, pages 1453–1462, 2017.
[39] Sibei Yang, Guanbin Li, and Yizhou Yu. Cross-modal re- lationship inference for grounding referring expressions. In CVPR, pages 4145–4154, 2019.
[40] Sibei Yang, Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression comprehension. In ICCV, pages 4644–4653, 2019.
[41] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In ECCV, pages 684–699, 2018.
[42] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In CVPR, pages 1307–1315, 2018.
[43] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, pages 69–85. Springer, 2016.
[44] Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. A joint speaker-listener-reinforcer model for referring expressions. In CVPR, pages 7282–7290, 2017.
[45] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR, pages 5831–5840, 2018.
[46] Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis. Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment. In CVPR, pages 1247–1257, 2019.
[47] Hanwang Zhang, Yulei Niu, and Shih-Fu Chang. Grounding referring expressions in images by variational context. In CVPR, pages 4158–4166, 2018.
[48] Zhu Zhang, Zhijie Lin, Zhou Zhao, and Zhenxin Xiao. Cross-modal interaction networks for query-based moment retrieval in videos. In SIGIR, 2019.
[49] Zhu Zhang, Zhou Zhao, Zhijie Lin, Jingkuan Song, and Deng Cai. Localizing unseen activities in video via image query. In IJCAI, 2019.
[50] Luowei Zhou, Nathan Louis, and Jason J Corso. Weakly- supervised video object grounding from text by loss weighting and object interaction. arXiv preprint arXiv:1805.02834, 2018.
[51] Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, and Anton van den Hengel. Parallel attention: A unified framework for
visual object discovery through dialogs and queries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4252–4261, 2018.
Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences –Supplementary Material–
VidOR contains 7,000, 835 and 2,165 videos for training, validation and testing, respectively. Since box annotations of testing videos are unavailable yet, we omit testing videos, split 10% training videos as our validation data and regard original validation videos as the testing data. Considering a video may contain multiple same triplets that have different temporal and bounding box annotations, we cut these videos into several short videos, where each short video contains a triplet that covers a segment of the short video. We then delete unsuitable video-triplet pairs based on three rules: (1) the video length is less than 3 seconds; (2) the temporal duration of the triplet is less than 0.5 seconds; (3) the triplet duration is less than 2% of the video. Next, because too many triplets are related to spatial relations like ”in front of” and ”next to”, we delete 90% spatial triplets to keep the types of relations balanced.
For each video-triplet pair, we choose the subject or object as the queried object, and then describe its appearance, relationships with other objects and visual environments. We discard video-triplet pairs that are too hard to give a precise description. And a video-triplet pair may correspond to multiple sentences. After annotation, there are 6,924 videos (5,563, 618 and 743 for training, validation and testing sets) and 99,943 sentences for 44,808 video-triplet pairs. we show some typical samples in Figure 5 with declarative and interrogative sentences. We can find that the objects may exist in a very small segment of the video and the sentences may only describe a short-term state of the queried object.
Next, we show the distribution of different types of queried objects as Figure 6. The original VidOR contains 80 types of objects, including 3 types of persons, 28 types of animals and 49 types of other objects. After data cleaning and annotating, sentences in VidSTG describes 79 types of objects by the declarative or interrogative ways, including 3 types of persons, 27 types of animals and 49 types of other objects. A rare type (i.e., stingray) is not contained in VidSTG. From Figure 6, we can find the sentences of person types take up the largest proportion and sentence numbers of other categories are relatively uniform.
Moreover, we compare VidSTG with existing video grounding datasets in Table 4. Previous temporal sentence grounding datasets like DiDeMo [11], Charades-STA [8], TACoS [29] and ActivityCation [17] only provide the temporal annotations for each sentence and lack the spatio-temporal bounding boxes. As for existing video grounding datasets, Persen-sentence [38] is originally used for spatio-temporal person retrieval among trimmed videos and only contains one type of objects (i.e. people), which is too simple for the STVG task. And VID-sentence dataset [4] contains 30 categories but also offer the annotations on trimmed videos. Different from them, our VidSTG simultaneously offers temporal clip and spatio-temporal tube annotations, contains more sentence descriptions, has a richer variety of objects, and further supports multi-form sentences.
Since no existing strategy can be directly applies to STVG, we combine the existing visual grounding method GroundeR [31] and spatio-temporal video grounding approaches STPR [38] and WSSTG [4] with the temporal sentence localization methods TALL [8] and L-Net [3] as the baselines. The TALL and L-Net first provide the temporal clip of the target tube and the extended GroundeR, STPR and WSSTG then retrieve the spatio-temporal tubes of objects.
We first introduce the TALL and L-Net approaches. The TALL applies a sliding window framework that first samples abundant candidate clips and then ranks them by estimating the clip-sentence scores. During estimating, TALL incorporates the context features for the current clip to further improve the localization accuracy. And L-Net develops the evolving frame-by-word interactions for video and query contents, and dynamically aggregates the matching evidence to localize the temporal boundaries of clips according to the textual query.
Next, we illustrate the extended grounding methods GroundeR, STPR and WSSTG based on the retrieved clip. The GroundeR is a frame-level approach, which originally grounds natural language in a still image. We apply it for each frame of the clip to obtain the object region and generate a tube by directly connecting these regions. The drawback of this method is the lack of temporal context modeling of regions. Different from it, original STPR and WSSTG are both tube-level methods and adopt the tube pre-generation framework. This framework first extracts a set of spatio-temporal tubes from trimmed clips and then identifies the target tube. The original STPR [38] only grounds persons from multiple videos, we extend it to multi-type object grounding in a single clip. Specifi-cally, we use the pre-trained Faster R-CNN to detect multi-type object regions to generate the candidate tubes rather than only generate person candidate tubes. And during training, we retrieve the correct tube from a video rather multiple videos, where we do not change the loss func-
Figure 5. Annotation Samples with Declarative or Interrogative Sentence Descriptions.
Figure 6. The Distribution of Different Types of Queried Objects in the Entire VidSTG Dataset.
tion of STPR. The original WSSTG [4] employs a weakly-supervised setting, we extend it to the fully-supervised form. Concretely, we discard the original ranking and diversity losses and employs a classics triplet loss [39] on the matching scores of the candidate tubes and sentence. The STPR and WSSTG both have the drawbacks of the tube pre-generation framework: (1) they are hard to pre-generate high-quality tubes without textual clues; (2) they only consider single tube modeling and ignore the rela1ionships between objects. Finally, we obtain 6 combined baselines GroundeR+TALL, STPR+TALL, WSSTG+TALL, GroundeR+L-Net, STPR+L-Net and WSSTG+L-Net. We also provide the temporal ground truth clip to form 3 baselines GroundeR+Tem.Gt, STPR+Tem.Gt and WSSTG+Tem.Gt.
During training, we first train the TALL and L-Net based
Table 4. Dataset Comparison.
on the sentence-clip matching data and train GroundeR, STPR and WSSTG within the ground truth clip. But while inference, we first use TALL and L-Net to determine the clip boundaries and then employ GroundeR, STPR and WSSTG to localize the final tubes. To guarantee the fair comparison, GroundeR, STPR and WSSTG are built on the same TALL or L-Net models, and we apply the Adam optimizer to train all baselines.
9.1. Directed GCN
To confirm the effect of the directed explicit GCN, we replace it with the original undirected GCN [16] and GAT [34]. In Table 5, our directed GCN has a better performance and the results of undirected GCN and GAT are close to the model without explicit subgraph modeling. The reason is that the undirected GCN and GAT have a similar ability with the implicit GCN and may lead to redundancy modeling.
9.2. Query Modeling
Our input setting is consistent with previous grounding works but we adopt a different strategy for query modeling in STGRN. Previous works model the sentence by RNN as a whole query vector. Different from them, we use the NLTK library to recognize the first noun or interrogative word ”who/what” in the sentence, corresponding to the query object. We then select its feature from RNN outputs and adopt context attention to learn the object-aware query vector
. We conduct an ablation study for query
modeling in Table 6, where we also apply the GRU+Object Rec.+Attention to WSSTG+L-Net. Concretely, the object-aware vector replaces the original sentence vector in final localization for L-Net and is added into visually guided sentence features for WSSTG.