Pedestrian detection through Computer Vision is a building block for a multitude of applications. Recently, there has been an increasing interest in convolutional neural network-based architectures to execute such a task. One of these supervised networks’ critical goals is to generalize the knowledge learned during the training phase to new scenarios with different characteristics. A suitably labeled dataset is essential to achieve this purpose. The main problem is that manually annotating a dataset usually requires a lot of human effort, and it is costly. To this end, we introduce ViPeD (Virtual Pedestrian Dataset), a new synthetically generated set of images collected with the highly photo-realistic graphical engine of the video game GTA V (Grand Theft Auto V), where annotations are automatically acquired. However, when training solely on the synthetic dataset, the model experiences a Synthetic2Real domain shift leading to a performance drop when applied to real-world images. To mitigate this gap, we propose two different domain adaptation techniques suitable for the pedestrian detection task, but possibly applicable to general object detection. Experiments show that the network trained with ViPeD can generalize over unseen real-world scenarios better than the detector trained over real-world data, exploiting the variety of our synthetic dataset. Furthermore, we demonstrate that with our domain adaptation techniques, we can reduce the Synthetic2Real domain shift, making the two domains closer and obtaining a performance improvement when testing the network over the real-world images.
Keywords pedestrian detection domain adaptation synthetic datasets convolutional neural networks deep learning
A key task in many intelligent video surveillance systems is pedestrian detection, as it provides essential information for semantic understanding of video. Accurate detection of individual instances of pedestrians in images plays a vital role in a myriad of applications that can positively impact the quality of human life. They range from video surveillance [1, 2], robotics, automotive [3, 4] and assistive technologies to people with visual disabilities , just to name a few.
Convolutional neural networks-based methods  have recently demonstrated their superiority compared to the approaches relying on hand-crafted features. However, despite the recent advances, the pedestrian detection task remains a challenging active research area in Computer Vision. While there exist some large annotated generic datasets suitable for training these supervised learning networks, such as ImageNet  and MS COCO , in many real-world situations they are not enough. Hence, as a consequence, a model trained using these data usually experiences a drastic drop in performance when applied to another scenario at inference time.
The crux of Convolutional Neural Networks (CNNs) is that, to generalize well at inference time, they require a huge amount of diverse labeled data during the training phase, covering the widest number of different scenarios. Since manually annotating new collections of images is expensive and requires a great human effort, a recently promising approach is to gather data from virtual world environments that mimics as much as possible all the characteristics of the real-world scenarios, and where the annotations can be acquired with a partially automated process. To this end, in this work, we provide ViPeD (Virtual Pedestrian Dataset), a new synthetic dataset generated with the highly photo-realistic graphical engine of the video game GTA V (Grand Theft Auto V) by Rockstar North, that extends the JTA (Joint Track Auto) dataset presented in .
The use of synthetic datasets based on 3D rendering to tackle the annotation problem is not new. Some notable examples are GTA5  and SYNTHIA  for semantic segmentation. However, to the best of our knowledge, ViPeD is the first synthetic dataset suitable for the pedestrian detection task, which is annotated with bounding boxes locating the people’s instances present in the scenes.
While synthetic data collections are very appealing, usually, when training solely on a synthetic dataset, the model does not generalize well to real-world data. This performance gap is due to the fact that the network learns from one domain (named training or source domain) and is then applied on another different domain (test or target domain), and is commonly referred as Domain Shift . In this particular case, the source and the target domains are the synthetic and the real-world ones, respectively. Hence, we call this Domain Shift as Synthetic2Real.
In this paper, we propose two different Domain Adaptation (DA) methods to mitigate this Synthetic2Real Domain Shift, suitable for the pedestrian detection task, but possibly applicable to general object detection. The first one consists of training the model exploiting the synthetic data, and then, in a second step, fine-tuning it using the real-world images. Instead, the second one consists of an end-to-end training procedure in which we employ mixed batches containing both synthetic and real data.
First, we test the generalization capabilities of the detector over unseen scenarios. We show that we can obtain better or comparable results when training exploiting the synthetic data than when using the same model trained using only real-world images, just taking advantage of the variety of ViPeD. Secondly, we experiment with the two proposed domain adaptation techniques to boost the performance over specific real-world scenarios. We demonstrate that we can reduce the Synthetic2Real Domain Shift by bringing the two domains closer together, thus achieving better results.
Summarizing, the main contributions of this work are the followings:
• We introduce and make publicly available ViPeD, a new vast synthetic dataset suitable for the pedestrian detection task, generating the images using photo-realistic video game GTA V (Grand Theft Auto V), that extends the JTA (Joint Track Auto) dataset presented in .
• We present two supervised Domain Adaptation techniques to mitigate the Synthetic2Real Domain Shift existing between the synthetic and the real images.
• We conduct extensive experimentation on various real-world pedestrian detection datasets present in the literature. First, we test the detector’s generalization capabilities, demonstrating that we achieve comparable or better results using synthetic data during the training phase rather than relying solely on the real-world images. Second, we experiment with the two proposed DA solutions to boost the performance over specific real-world scenarios, bringing the synthetic and the real domains closer, achieving better results.
Specifically, in this work, we extend our previous paper . Compared to , we obtain better results, employing a new state-of-the-art detector that exhibits higher performance and introducing a new domain adaptation strategy. Furthermore, we carry out extensive experimentation over additional publicly available datasets, demonstrating the robustness of our approach over different real-world scenarios. The code, the models, and the dataset are made freely available at https://ciampluca.github.io/viped/.
In this section, we review some relevant works about the object and pedestrian detection task. We also analyze some previous studies on DA, focusing on the Synthetic2Real domain shift.
2.1 Pedestrian Detection
Pedestrian detection is highly related to object detection. It deals with locating and recognizing instances of pedestrians’ specific class, usually in images of urban environments, without taking into account group dynamics. We can subdivide approaches for the pedestrian detection task into two main research areas. The first class of detectors is based on handcrafted features, such as ICF (Integral Channel Features) [14, 15, 16, 17, 18]. These methods can usually rely on higher computational efficiency, at the cost of lower accuracy. On the other hand, more recently, Deep Neural Network approaches have been explored. For example, [19, 20, 21, 22] proposed some solutions based on the CNN networks  to detect pedestrians, even accounting for different scales.
Recent advances using CNNs were also possible thanks to the availability of many new datasets. Some of the most used in literature are Caltech , INRIA , MOT17Det , MOT19Det , and CityPersons . In this work, we considered the latter three ones since they describe very heterogeneous video-surveillance scenarios, and they have proved to be enough challenging due to their high variability outlining most of the real-world problematic situations. The Caltech and the INRIA datasets are instead specifically collected for detecting pedestrians in self-driving contexts, a different scenario not considered in this paper.
2.2 Synthetic2Real Domain Adaptation
With the need for huge amounts of labeled data, synthetically-generated datasets have recently gained considerable interest. Some notable examples are GTA5  and SYNTHIA  for semantic segmentation.
However, as already mentioned in the introduction, there is a non-negligible domain gap between the synthetic and the real worlds. Many techniques try to fill this gap, using both supervised and unsupervised approaches. An exhaustive survey about deep learning DA techniques is provided in . For example, authors in  and in  proposed two unsupervised domain adaptation solutions for the counting task and the segmentation task, respectively, taking advantage of the output space. Authors in  created JTA (Joint Track Auto), a synthetic dataset taken from the highly photo-realistic video game GTA V. They demonstrated that it is possible to reach excellent results on tasks such as people tracking and pose estimation when validating on real data. In this work, we extend this dataset, making it suitable for the pedestrian detection task.
Authors in [31, 32] have also focused on learning features from synthetic data for the pedestrian detection task. Still, they did not take into account deep learning approaches, exploring only traditional detection techniques. In , instead, the authors employed a synthetic dataset to train a CNN able to detect objects belonging to different classes in a video. This CNN is responsible only for the classification of the objects, while the detection of them relied on a background subtraction algorithm based on Gaussian Mixture Models (GMMs). This approach’s performance over real-world scenarios was evaluated employing two pedestrian detection datasets, one of which, the 2D MOT 2015 , is an older version of the dataset we used to carry out our experiments. To the best of our knowledge,  and  are the closest works to our. In particular, they also used GTA V as the source for the acquisition of the synthetic data, but they focus their efforts on the vehicle detection task.
In this section, we illustrate the motivations for using synthetic data, pointing out the main benefits and drawbacks. Then, we introduce and describe the construction of ViPeD , our synthetic collection of images exploited for training the pedestrian detector.
3.1 Training with Synthetic Datasets
As already pointed out in Section 1, the main drawback of CNN-based methods is that they hinge on large quantities of annotated data. Since they require ground truth labels for supervised learning, they may not generalize well to unseen images, especially when there is a large domain gap between the training (source) and the test (target) sets, such as different perspectives, illuminations, and object scales. This gap often severely hampers the application of CNN-based solutions to very large scale scenarios since annotating images for all the possible cases is an expensive operation, implying a considerable human-effort.
A possible solution to this problem is to create a vast and suitable dataset by collecting images from virtual world environments that resemble, as closely as possible, all the characteristics of the target real-world scenarios. Here, the main advantage is that the labels of the images can be acquired with a partially automated process, and so the data collection is significantly less costly. Consequently, it is possible to record a considerable amount of images covering a large number of different scenarios.
However, besides these positive aspects, there are some drawbacks to be considered. In particular, synthetic images’ appearance is still significantly different from that observed in real-world images, even using current rendering techniques. Thus, the model trained solely on the synthetic dataset does not generalize to real-world data as one might expect due to the Synthetic2Real Domain Shift.
With the purpose of reducing the described domain shift, domain adaptation techniques can be exploited during the CNN-based networks’ training phase. These methods try to make more similar the two data distributions, i.e., the distribution of the features belonging to the synthetic world and the one belonging to the real-world environment. Thus, it is possible to take advantage of the synthetic dataset’s diversity, mitigating the underlying differences between the two domains.
ViPeD (Virtual Pedestrian Dataset) is a new synthetic dataset generated with the highly photo-realistic graphical engine of the video game GTA V (Grand Theft Auto V) by Rockstar North. It extends JTA (Joint Track Auto) dataset, presented in . The dataset includes a total of about 500K images, extracted from 512 full-HD videos of different urban scenarios. These videos are organized into a training set (256 videos) and a test set (the remaining 256 videos).
While we can reuse the already existing JTA images, we need to generate suitable annotations for the pedestrian detection task. Indeed, the JTA dataset provides only skeletal information useful in the pose estimation and tracking tasks. In our scenario, instead, we are required to annotate each pedestrian with the four coordinates (x, y, w, h) delimiting its minimum enclosing bounding box. Hence, we employed the already available JTA images producing a new set of labels suitable for our task.
Estimating the precise bounding box surrounding each pedestrian instance can be tricky, as we do not have access to the underlying GTA game engine. Other works tried to overcome this problem by using some interesting work-around. For example,  extracted the semantic masks around each object in the scene and separated the instances by exploiting the depth information available through the depth buffer.
Our solution relies instead on the skeletal information already provided by the JTA annotations. Indeed, differently from , we deal with multiple instances of pedestrians in possibly highly crowded scenarios. In these cases, the depth information may be insufficient for distinguishing two different pedestrians, leading to possible severe bounding box estimation errors.
As a first approximation, we exploited the skeleton joints’ positions in screen coordinates, directly available from the JTA annotations, for drawing the minimum bounding box enclosing all the skeleton joints (green boxes in Figure 1b). However, it can be noticed that the bounding boxes produced using this simple procedure are undersized compared to the full-sized pedestrian instance, as the skeleton always lays below the skin surface. We solved this issue by constructing a bigger bounding box (blue boxes in Figure 1b), obtained by estimating an amount of padding through a simple heuristic. In particular, we estimated the height of a pedestrian mesh, denoted as , from the height of its skeleton, through the formula:
where z is the distance of the pedestrian center of mass from the camera, and is a parameter that depends on the camera projection matrix.
Figure 1: (a) Pedestrians in the JTA (Joint Track Auto) dataset with their skeletons. (b) Examples of annotations in the ViPeD (Virtual Pedestrian Dataset) dataset; original bounding boxes are in green, while the sanitized ones are in blue.
The z value for each pedestrian is already included in the JTA annotations, while is unknown since we can not access the camera parameters. Then, we evaluated from Equation 1, estimating for a small representative population of pedestrians. To this end, we isolated 50 random pedestrians from different scenarios, and we manually annotated them with their height in pixels units. At this point, it has been possible to recover the value of from Equation 1 performing a simple linear regression to find the best fit.
The height padding depends basically only on the distance of the pedestrian from the camera. Instead, the width is also linked to the specific pedestrian pose. However, we found that we can ignore these pose-dependent effects while still obtaining an excellent estimate by deriving the pedestrian width assuming no changes in the original bounding box aspect ratio. For this reason, we simply derived from the computed as follows:
where r is the aspect ratio of the bounding box enclosing the skeleton. Some examples of final estimated bounding boxes are shown in blue in Figure 1b.
We then assessed the quality of the produced bounding boxes. In Figure 2, we report a histogram depicting the distribution of the distances of the pedestrians from the camera. We observed that human annotators tend not to annotate pedestrians far than a certain amount from the camera in real-world datasets. We compute this distance limit by finding the minimum bounding box height, in pixels, occurring in human annotations of the MOT17Det  dataset, and seeing at what distance from the camera we reach this bounding box limit size on the JTA annotations. We concluded that human annotators do not include bounding boxes for pedestrians farther than 30–40 meters from the camera. Then, to be consistent with real-world datasets on which we will validate our approach, we cleaned the produced bounding boxes by pruning all the ones enclosing pedestrians farther than 40 meters.
Figure 2: Histogram of distances between pedestrians and cameras.
In Figure 3, we report some examples of images of the ViPeD dataset together with the sanitized bounding boxes.
In this section, we describe the object detector and the domain adaptation strategies we employed in this work. We exploited Faster R-CNN , a widely used state-of-the-art object detector that we briefly review in Section 4.1. We train this CNN using ViPeD, our collection of synthetic images automatically annotated, already outlined in Section 3.2. To mitigate the existing domain shift between these data and the real-world ones, we propose two domain adaptation techniques. The first one, described in Section 4.2, consists of training the detector with the synthetic data and then fine-tuning it exploiting the real-world images. In the second approach, described in Sec. 4.3, we employ instead another supervised technique, called Balanced Gradient Contribution (BGC) [11, 37], where we mix the synthetic and the real-world data during the training phase. Figures 4 and 5 show an overview of the two solutions.
4.1 Faster R-CNN Object Detector
We exploit Faster-RCNN  as object detector architecture. In our previous work , we employed YOLOv3 , another state-of-the-art object detector. Here, our choice fell on Faster R-CNN since it provides better performance. Furthermore, we do not consider pedestrian detection-specific solutions since the two proposed domain adaptation techniques can also be applied to other tasks, accounting for another class of objects different from the pedestrian one.
Faster R-CNN is a two-stage CNN-based algorithm composed of different networks: The backbone, the Region Proposal Network (RPN), and the Evaluation Network (EN). In the first stage, a CNN acts as a backbone, extracting the input image features. Starting from these features space, the RPN is in charge of generating region proposals that might contain objects. Briefly, RPN slices pre-defined region boxes (called anchors) over this space and ranks them,
Figure 3: Examples of images of the ViPeD dataset together with the sanitized bounding boxes.
suggesting the ones most likely containing objects. Once RPN produces the Regions Of Interests (ROIs), they might be of different sizes. Since it is hard to work on features having different sizes, RPN reduces them into the same dimension using the Region of Interest Pooling algorithm. These fixed-size proposals are finally processed by the EN, responsible for classifying and locating the objects inside them. Then, given an input image, the EN network final outputs are class scores and bounding boxes coordinates.
Faster R-CNN is then a versatile and modular network in which it is possible to change the building blocks. Regarding the backbone, our choice fell on the ResNet-50 network, a lighter version of the very popular ResNet-101 network . Indeed, Faster R-CNN with ResNet-50 can produce satisfactory detection results compared to the low computational resources and the time required during the training and test phases.
4.2 Domain Adaptation using Real-World Fine-Tuning
The first proposed DA solution relies on a Transfer Learning (TL) strategy. As pointed out in , DA is a particular TL case that employs labeled data in one or more relevant source domains to execute the task in a target domain. In particular, the crucial point in this methodology consists of fine-tuning a previously trained model with the target-domain data.
We divide our fine-tuning methodology into two different steps.
In the first step, we consider as the baseline the Faster R-CNN detector described in the above section, having a ResNet-50 backbone pre-trained on the COCO dataset , a large collection of images depicting complex everyday scenes of ordinary objects in their natural context, divided into 80 different categories. Since this network is a generic object detector that can distinguish between many different classes of objects, we modify the EN building block to adapt the model to our purposes. In particular, we reduce the last fully connected layers of the detector to recognize and locate object instances belonging only to a specific category, i.e., the pedestrian category. Then, we train this modified Faster R-CNN-based network exploiting our synthetic images of the ViPeD, leaving all the model weights unfrozen during this phase so that the back-propagation algorithm can tune them.
Then, in the second step, we fine-tune this pre-trained model using real-world images as the target domain. So, in the end, the network will have processed both source and target images, memorizing in its weights information from both the domains. Figure 4 shows an overview of this approach. This method, looking at real images in this last step, is particularly useful for boosting the detector’s performance on a specific real-world target scenario.
Figure 4: Overview of the first domain adaptation technique. In a first step, we train the detector using ViPeD, our synthetic collection of images. Then, in a second step, we fine-tune the network using real-world images.
4.3 Domain Adaptation using Balanced Gradient Contribution
The second DA approach is an end-to-end training, so it benefits from not relying on a two-step process like the previous one.
As in the previous solution, we start with the modified Faster R-CNN detector having the ResNet-50 backbone pre-trained on the COCO dataset. This time, we train the network using mixed batches, i.e., we employ batches containing synthetic and real-world images simultaneously, given a fixed mixing ratio. As explained in , the real-world data acts as a regularization term over the synthetic data training loss. In particular, we exploit batches composed of 2/3 of synthetic images and of 1/3 of real-world data. Thus, statistics from both domains are considered throughout the entire procedure, creating a more accurate model for both.
Again, during this phase, we leave all the network weights unfrozen so that the back-propagation algorithm can modify the network parameters accordingly. Consequently, we mitigate the Synthetic2Real Domain Shift straight in a single-step training. Figure 5 shows an overview of this approach.
In the experiments, we employed this technique for boosting the detector’s performance on a precise target scenario, using batches composed of the synthetic data and the real-world images specific for the particular considered scenes. Besides, we exploited this solution also for achieving wide generalization capabilities, considering batches containing synthetic images and generic real-world images containing pedestrians.
In this section, we briefly report some details about the real-world datasets exploited for the experiments. Then, we show and discuss the results concerning the generalization capabilities of our detector trained using ViPeD. Finally, we illustrate the performance of the two domain adaptation techniques over specific real-world scenarios.
5.1 Real-World Datasets
MOT17Det  and MOT19Det  are newly introduced datasets with manual annotations for pedestrian detection that are particularly suitable for surveillance applications. They comprise a collection of challenging images (5,316 and 8,931, respectively) taken from multiple sequences with various crowded scenarios having different viewpoints, weather conditions, and camera motions. The authors provided training and test subsets, but they released only the ground-truth annotations belonging to the former. The performance metrics concerning the test subsets are instead available submitting results to their MOT Challenge website (https://motchallenge.net). The main peculiarity of MOT19Det compared to MOT17Det is the massive crowding of the collected scenarios.
CityPersons dataset  consists of a large and diverse set of stereo video sequences recorded in streets from different cities in Germany and neighboring countries. In particular, the authors provide 5,000 images from 27 cities labeled with bounding boxes and divided across train/validation/test subsets. This dataset is more focused on self-driving applications, and images are collected from a moving car.
Figure 5: Overview of the second domain adaptation technique. We mitigate the Synthetic2Real domain shift in a single-step training procedure, employing mixed batches containing both synthetic and real images at the same time.
COCOPersons dataset is a split of the popular COCO dataset  comprising images collected in general contexts belonging to 80 categories. We filter these images considering only the ones belonging to the persons category. Hence, we obtain a new dataset of about 66,000 images containing at least one pedestrian instance.
We evaluate the detection performances using the standard mean average precision (mAP) metric. In particular, we consider the detection proposals having a score confidence greater than 0.05. Then, we employ the COCO mAP  and the MOT AP metrics , fixing the IoU threshold to 0.5 and varying only the detection confidence threshold.
5.2.1 Testing Generalization Capabilities
To test the generalization capabilities, we train the detector on a source domain, and then we validate it on a different target domain. In particular, we train the model using a dataset, and then we test it on another one. In this way, we guarantee that the two distributions are different and not related.
In particular, we train the modified Faster R-CNN-based detector described in Section 4.1 using ViPeD. This procedure corresponds to the first step of the previously described domain adaptation solution (see 4.2). We evaluate this model testing it on the real-world datasets MOT17Det, MOT19Det and CityPersons, defining three validation subsets containing images not present in the training subset.
To form a solid baseline for this experiment, we train the same detector using every one of the three real-world datasets, and then we test them over the remaining two. We also report a further baseline considering the detector trained only on the real-world general-purpose COCO dataset, considering only the detections belonging to the person category.
We also experiment with the mixed-batch domain-adaptation approach explained in Section 4.3, using the same evaluation protocol as before. We exploit batches composed of 2/3 of ViPeD and by the remaining 1/3 of COCOPersons. We choose the latter as the real-world dataset since it depicts humans in highly heterogeneous scenarios, and it is not biased towards a specific application (e.g., autonomous driving). Again, we evaluate this model testing on all the three remaining real-world datasets.
We report the results in Table 1. Note that we omit results concerning a specific dataset if employed during the training phase, for a fair evaluation of the overall generalization capabilities.
Table 1: Evaluation of the generalization capabilities. The first section of the table reports results obtained training the detector with real-world data, while the latter is related to the model trained over synthetic images. ViPeD + Real refer to the mixed batch experiments with 2/3 ViPeD and 1/3 of COCOPersons. Results are evaluated using the COCO mAP. We report in bold the best results.
In most cases, as we can see, our network performs better than those trained using only the manually annotated real-world datasets, taking advantage of the high variability and size of the ViPeD dataset. In particular, concerning the MOT17Det dataset, all our solutions trained with synthetic data outperform those trained with real ones. We obtain the best results using the mixed-batch approach. Considering the MOT19Det dataset, we achieve the best result in training the detector with ViPeD. CityPersons is the only dataset on which the algorithm maintains higher performances when trained with real-world data. In particular, the highest mAP on CityPersons is obtained when the detector is trained with the MOT17Det dataset. However, the mixed-batch approach achieves, in this case, results comparable with the baselines.
5.2.2 Testing Domain Adaptation Techniques over Specific Real-world Scenarios
To test how the two proposed domain adaptation techniques behave when considering specific target real-world scenarios, we consider the MOT17Det and MOT19Det real-world datasets.
Regarding the fine-tuning DA approach, we consider as training sets those proposed by the authors of [25, 26], and we obtain the evaluation of our results over the test sets submitting them to the Mot Challenge website. For the mixed-batch DA solution, during the training phase, we inject in the same batch 2/3 of synthetic images from the ViPeD dataset and 1/3 of real-world images from the training subsets of the MOTDet17 or the MOT19Det dataset. Again, we validate our results by submitting them to the Mot Challenge website.
Table 2 and Table 3 report the results for the two considered scenarios. We report our results together with the state-of-the-art approaches publicly released in the MOT Challenges (at the time of writing).
Table 2: Evaluation of the two Domain Adaptation (DA) techniques on the MOT17Det dataset. FT-DA (Fine Tuning DA) is the first proposed solution, while MB-DA (Mixed Batch DA) is the second one. Results are evaluated using the MOT mean average precision (mAP).
Table 3: Evaluation of the two DA techniques on the MOT19Det dataset. FT-DA (Fine Tuning DA) is the first proposed solution, while MB-DA (Mixed Batch DA) is the second one. Results are evaluated using the MOT mAP.
As we can see, the two DA approaches can mitigate the Synthetic2Real Domain Shift. In both datasets, we obtain an improvement in performance compared to the results in Table 1. It is also worth noting that we achieve competitive results in both scenarios compared to the state-of-the-art, reaching the first and the second places in the leader boards of the MOT17Det and MOT19Det challenges, respectively.
In this work, we addressed the pedestrian detection task by proposing a CNN-based solution trained using synthetically generated data. The choice of training a CNN using synthetic data is motivated by the fact that the network, to generalize well, requires a considerable amount of manually annotated images representing different scenarios. This procedure usually requires a significant human effort, and it is error-prone.
To this end, we introduced a synthetic dataset named ViPeD , containing a massive collection of images rendered from the highly photo-realistic video game GTA V developed by Rockstar North and a full set of precise bounding boxes annotations around all the visible pedestrians. To the best of our knowledge, it is the first synthetic dataset suitable for the pedestrian detection task.
Furthermore, we proposed two different Domain Adaptation techniques to mitigate the Synthetic2Real Domain Shift, which are suitable for the pedestrian detection task and possibly applicable to more general object detection tasks.
The experiments showed that, in most cases, the detector trained with the synthetic data can generalize better on unseen scenarios than the same algorithm trained using only the manually annotated real-world datasets. Moreover, the two proposed DA approaches can mitigate the underlying differences between the two worlds, obtaining a performance improvement on specific real-world scenarios.
In our opinion, the result of this work opens new perspectives to address the scalability of pedestrian and object detection methods for large physical systems with limited supervisory resources. Using our freely available model trained using ViPeD , future researchers will have at their disposal a detector able to localize instances of people over images belonging to a multitude of different scenarios and, therefore, a system robust to newly added sources of data. On the other hand, they will also have the possibility of further specializing the detector to work over new added real-world scenarios using our two domain adaptation techniques, obtaining an additional performance boost.
 Muhammad Bilal, Asim Khan, Muhammad Umar Karim Khan, and Chong-Min Kyung. A low-complexity pedestrian detection framework for smart video surveillance systems. IEEE Transactions on Circuits and Systems for Video Technology, 27(10):2260–2273, 2016.
 Domonkos Varga and Tamás Szirányi. Robust real-time pedestrian detection in surveillance videos. Journal of Ambient Intelligence and Humanized Computing, 8(1):79–85, 2017.
 Dariu M Gavrila and Stefan Munder. Multi-cue pedestrian detection and tracking from a moving vehicle. International journal of computer vision, 73(1):41–59, 2007.
 Amnon Shashua, Yoram Gdalyahu, and Gaby Hayun. Pedestrian detection for driving assistance systems: Singleframe classification and system level performance. In IEEE Intelligent Vehicles Symposium, 2004, pages 1–6. IEEE, 2004.
 Yingli Tian. Rgb-d sensor-based computer vision assistive technology for visually impaired persons. In Computer Vision and Machine Learning with RGB-D Sensors, pages 173–194. Springer, 2014.
 Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
 J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, June 2009.
 Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
 Matteo Fabbri, Fabio Lanzi, Simone Calderara, Andrea Palazzi, Roberto Vezzani, and Rita Cucchiara. Learning to detect and track visible and occluded body joints in a virtual world. In European Conference on Computer Vision (ECCV), 2018.
 Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision, pages 102–118. Springer, 2016.
 German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
 Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521–1528. IEEE, 2011.
 Giuseppe Amato, Luca Ciampi, Fabrizio Falchi, Claudio Gennaro, and Nicola Messina. Learning pedestrian detection from virtual worlds. In Image Analysis and Processing – ICIAP 2019, pages 302–312, Cham, 2019. Springer International Publishing.
 Rodrigo Benenson, Mohamed Omran, Jan Hosang, and Bernt Schiele. Ten years of pedestrian detection, what have we learned? In Computer Vision - ECCV 2014 Workshops, pages 613–627, Cham, 2015. Springer International Publishing.
 Shanshan Zhang, Christian Bauckhage, and Armin B. Cremers. Informed haar-like features improve pedestrian detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
 S. Zhang, R. Benenson, and B. Schiele. Filtered channel features for pedestrian detection. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1751–1760, June 2015.
 Shanshan Zhang, Rodrigo Benenson, Mohamed Omran, Jan Hosang, and Bernt Schiele. How far are we from solving pedestrian detection? In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
 Woonhyun Nam, Piotr Dollar, and Joon Hee Han. Local decorrelation for improved pedestrian detection. In Advances in Neural Information Processing Systems 27, pages 424–432. Curran Associates, Inc., 2014.
 Y. Tian, P. Luo, X. Wang, and X. Tang. Deep learning strong parts for pedestrian detection. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1904–1912, Dec 2015.
 F. Yang, W. Choi, and Y. Lin. Exploit all the layers: Fast and accurate cnn object detector with scale dependent pooling and cascaded rejection classifiers. In 2016 IEEE CVPR, pages 2129–2137, June 2016.
 Zhaowei Cai, Quanfu Fan, Rogerio S. Feris, and Nuno Vasconcelos. A unified multi-scale deep convolutional neural network for fast object detection. In Computer Vision – ECCV 2016, pages 354–370, Cham, 2016. Springer International Publishing.
 Pierre Sermanet, Koray Kavukcuoglu, Soumith Chintala, and Yann Lecun. Pedestrian detection with unsupervised multi-stage feature learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013.
 P. Dollar, C. Wojek, B. Schiele, and P. Perona. Pedestrian detection: An evaluation of the state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(4):743–761, April 2012.
 N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 886–893 vol. 1, June 2005.
 Anton Milan, Laura Leal-Taixé, Ian Reid, Stefan Roth, and Konrad Schindler. Mot16: A benchmark for multi-object tracking. arXiv preprint arXiv:1603.00831, 2016.
 Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixe. Cvpr19 tracking and detection challenge: How crowded can it get? arXiv preprint arXiv:1906.04567, 2019.
 Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele. Citypersons: A diverse dataset for pedestrian detection. CoRR, abs/1702.05693, 2017.
 Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312:135–153, 2018.
 Luca Ciampi, Carlos Santiago, Joao Paulo Costeira, Claudio Gennaro, and Giuseppe Amato. Unsupervised vehicle counting via multiple camera domain adaptation. arXiv preprint arXiv:2004.09251, 2020.
 Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7472–7481, 2018.
 B. Kaneva, A. Torralba, and W. T. Freeman. Evaluation of image features using a photorealistic virtual world. In 2011 International Conference on Computer Vision, pages 2282–2289, Nov 2011.
 J. Marín, D. Vázquez, D. Gerónimo, and A. M. López. Learning appearance in virtual scenarios for pedestrian detection. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 137–144, June 2010.
 Erik Bochinski, Volker Eiselein, and Tomas Sikora. Training a convolutional neural network for multi-class object detection using solely virtual world data. In Advanced Video and Signal Based Surveillance (AVSS), 2016 13th IEEE International Conference on, pages 278–285. IEEE, 2016.
 Laura Leal-Taixé, Anton Milan, Ian Reid, Stefan Roth, and Konrad Schindler. Motchallenge 2015: Towards a benchmark for multi-target tracking. arXiv preprint arXiv:1504.01942, 2015.
 Matthew Johnson-Roberson, Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? arXiv preprint arXiv:1610.01983, 2016.
 Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28, pages 91–99. Curran Associates, Inc., 2015.
 German Ros, Simon Stent, Pablo F Alcantarilla, and Tomoki Watanabe. Training constrained deconvolutional networks for road scene semantic segmentation. arXiv preprint arXiv:1604.01545, 2016.
 Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
 Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
 Fengwei Yu, Wenbo Li, Quanquan Li, Yu Liu, Xiaohua Shi, and Junjie Yan. Poi: Multiple object tracking with high performance detection and appearance feature. In European Conference on Computer Vision, pages 36–42. Springer, 2016.
 Chunze Lin, Jiwen Lu, Gang Wang, and Jie Zhou. Graininess-aware deep feature learning for pedestrian detection. In The European Conference on Computer Vision (ECCV), September 2018.