Human body pose estimation methods have become increasingly reliable. Powerful body part detectors [36] in combination with tree-structured body models [37, 7] show impressive results on diverse datasets [21, 3, 33]. These benchmarks promote pose estimation of single pre-localized persons but exclude scenes with multiple people. This problem definition has been a driver for progress, but also falls short on representing a realistic sample of real-world images. Many photographs contain multiple people of interest (see Fig 1) and it is unclear whether single pose approaches generalize directly. We argue that the multi person case deserves more attention since it is an important real-world task.
Key challenges inherent to multi person pose estimation
Figure 1. Method overview: (a) initial detections (= part candidates) and pairwise terms (graph) between all detections that (b) are jointly clustered belonging to one person (one colored subgraph = one person) and each part is labeled corresponding to its part class (different colors and symbols correspond to different body parts); (c) shows the predicted pose sticks.
are the partial visibility of some people, significant overlap of bounding box regions of people, and the a-priori unknown number of people in an image. The problem thus is to infer the number of persons, assign part detections to person instances while respecting geometric and appearance constraints. Most strategies use a two-stage inference process [29, 18, 35] to first detect and then independently estimate poses. This is unsuited for cases when people are in close proximity since they permit simultaneous assignment of the same body-part candidates to multiple people hypotheses.
As a principled solution for multi person pose estimation a model is proposed that jointly estimates poses of all people present in an image by minimizing a joint objective. The formulation is based on partitioning and labeling an initial pool of body part candidates into subsets that correspond to sets of mutually consistent body-part candidates and abide to mutual consistency and exclusion constraints. The proposed method has a number of appealing properties. (1) The formulation is able to deal with an unknown number of people, and also infers this number by linking part hypotheses. (2) The formulation allows to either deactivate or merge part hypotheses in the initial set of part candidates hence effectively performing non-maximum suppression (NMS). In contrast to NMS performed on individual part candidates, the model incorporates evidence from all other parts making the process more reliable. (3) The problem is cast in the form of an Integer Linear Program (ILP). Although the problem is NP-hard, the ILP formulation facilitates the computation of bounds and feasible solutions with a certified optimality gap.
This paper makes the following contributions. The main contribution is the derivation of a joint detection and pose estimation formulation cast as an integer linear program. Further, two CNN variants are proposed to generate representative sets of body part candidates. These, combined with the model, obtain state-of-the-art results for both single-person and multi-person pose estimation on different datasets. Related work. Most work on pose estimation targets the single person case. Methods progressed from simple part detectors and elaborate body models [32, 31, 19] to tree-structured pictorial structures (PS) models with strong part detectors [28, 42, 7, 33]. Impressive results are obtained predicting locations of parts with convolutional neural networks (CNN) [38, 36]. While body models are not a necessary component for effective part localization, constraints among parts allow to assemble independent detections into body configurations as demonstrated in [7] by combining CNNbased body part detectors with a body model [42].
A popular approach to multi-person pose estimation is to detect people first and then estimate body pose independently [35, 29, 42, 18]. [42] proposes a flexible mixture-of-parts model for detection and pose estimation. [42] obtains multiple pose hypotheses corresponding to different root part positions and then performing non-maximum suppression. [18] detects people using a flexible configuration of poselets and the body pose is predicted as a weighted average of activated poselets. [29] detects people and then predicts poses of each person using a PS model. [5] estimates poses of multiple people in 3D by constructing a shared space of 3D body part hypotheses, but uses 2D person detections to establish the number of people in the scene. These approaches are limited to cases with people sufficiently far from each other
that do not have overlapping body parts.
Our work is closely related to [13, 25] who also propose a joint objective to estimate poses of multiple people. [13] proposes a multi-person PS model that explicitly models depth ordering and person-person occlusions. Our formulation is not limited by a number of occlusion states among people. [25] proposes a joint model for pose estimation and body segmentation coupling pose estimates of individuals by image segmentation. [13, 25] uses a person detector to generate initial hypotheses for the joint model. [25] resorts to a greedy approach of adding one person hypothesis at a time until the joint objective can be reduced, whereas our formulation can be solved with a certified optimality gap. In addition [25] relies on expensive labeling of body part segmentation, which the proposed approach does not require.
Similarly to [8] we aim to distinguish between visible and occluded body parts. [8] primarily focuse on the single-person case and handles multi-person scenes akin to [42]. We consider the more difficult problem of full-body pose estimation, whereas [13, 8] focus on upper-body poses and consider a simplified case of people seen from the front.
Our work is related to early work on pose estimation that also relies on integer linear programming to assemble candidate body part hypotheses into valid configurations [19]. Their single person method employs a tree graph augmented with weaker non-tree repulsive edges and expects the same number of parts. In contrast, our novel formulation relies on fully connected model to deal with unknown number of people per image and body parts per person.
The Minimum Cost Multicut Problem [9, 11], known in machine learning as correlation clustering [4], has been used in computer vision for image segmentation [1, 2, 23, 43] but has not been used before in the context of pose estimation. It is known to be NP-hard [10].
In this section, the problem of estimating articulated poses of an unknown number of people in an image is cast as an optimization problem. The goal of this formulation is to state three problems jointly: 1. The selection of a subset of body parts from a set D of body part candidates, estimated from an image as described in Section 4 and depicted as nodes of a graph in Fig. 1(a). 2. The labeling of each selected body part with one of C body part classes, e.g., “arm”, “leg”, “torso”, as depicted in Fig. 1(c). 3. The partitioning of body parts that belong to the same person, as depicted in Fig. 1(b).
2.1. Feasible Solutions
We encode labelings of the three problems jointly through triples (x, y, z) of binary random variables with domains and
. Here,
indicates that body part candidate d is of class
indicates that the body part candidates d and
belong to the same person, and
are auxiliary variables to relate x and y through
. Thus,
indicates that body part candidate d is of class
), body part candidate
is of class
(
), and body part candidates
belong to the same person (
In order to constrain the 01-labelings (x, y, z) to well-defined articulated poses of one or more people, we impose the linear inequalities (1)–(3) stated below. Here, the inequalities (1) guarantee that every body part is labeled with at most one body part class. (If it is labeled with no body part class, it is suppressed). The inequalities (2) guarantee that distinct body parts belong to the same person only if neither
is suppressed. The inequalities (3) guarantee, for any three pairwise distinct body parts,
and
, if d and
are the same person (as indicated by
) and
and
are the same person (as indicated by
), then also d and
are the same person (
), that is, transitivity, cf. [9]. Finally, the inequalities (4) guarantee, for any
These constraints allow us to write an objective function as a linear form in z that would otherwise be written as a cubic form in x and y. We denote by
the set of all (x, y, z) that satisfy all inequalities, i.e., the set of feasible solutions.
When at most one person is in an image, we further constrain the feasible solutions to a well-defined pose of a single person. This is achieved by an additional class of inequalities which guarantee, for any two distinct body parts that are not suppressed, that they must be clustered together:
2.2. Objective Function
For every pair , we will estimate a probability
of the body part d being of class c. In the context of CRFs, these probabilities are called part unaries and we will detail their estimation in Section 4.
For every and every
, we consider a probability
of the conditional probability of
belonging to the same person, given that
body parts of classes
, respectively. For
probabilities
are the pairwise terms in a graphical model of the human body. In contrast to the classic pictorial structures model, our model allows for a fully connected graph where each body part is connected to all other parts in the entire set D by a pairwise term. For
probability of the part candidates d and
representing the same part of the same person. This facilitates clustering of multiple part candidates of the same part of the same person and a repulsive property that prevents nearby part candidates of the same type to be associated to different people.
The optimization problem that we call the subset partition and labeling problem is the ILP that minimizes over the set of feasible solutions
where we used the short-hand notation
The objective (6)–(10) is the MAP estimate of a probability measure of joint detections x and clusterings y, z of body parts, where prior probabilities and
are estimated independently from data, and the likelihood is a positive constant if (x, y, z) satisfies (1)–(4), and is 0, otherwise. The exact form (6)–(10) is obtained when minimizing the negative logarithm of this probability measure.
2.3. Optimization
In order to obtain feasible solutions of the ILP (6) with guaranteed bounds, we separate the inequalities (1)–(5) in the branch-and-cut loop of the state-of-the-art ILP solver Gurobi. More precisely, we solve a sequence of relaxations of the problem (6), starting with the (trivial) unconstrained problem. Each problem is solved using the cuts proposed by Gurobi. Once an integer feasible solution is found, we identify violated inequalities (1)–(5), if any, by breadth-first-search, add these to the constraint pool and re-solve the tightened relaxation. Once an integer solution satisfying all inequalities is found, together with a lower bound that certifies an optimality gap below 1%, we terminate.
Here we describe the estimation of the pairwise terms. We define pairwise features for the variable
(Sec. 2). Each part detection d includes the probabilities (Sec. 4.4), its location
, scale
and bounding box
coordinates. Given two detections d and
, and the corresponding features
and
, we define two sets of auxiliary variables for
, one set for
(same body part class clustering) and one for
(across two body part classes labeling). These features capture the proximity, kinematic relation and appearance similarity between body parts. The same body part class (
Two detections denoting the same body part of the same person should be in close proximity to each other. We introduce the following auxiliary variables that capture the spatial relations:
IOUnion, IOMin, IOMax. The latter three are intersections over union/minimum/maximum of the two detection boxes, respectively, and
Non-linear Mapping. We augment the feature representation by appending quadratic and exponential terms. The final pairwise feature for the variable
is
Two different body part classes (
We encode the kinematic body constraints into the pairwise feature by introducing auxiliary variables
are the Euclidean distance and the angle between two detections, respectively. To capture the joint distribution of
and
, instead of using
and
directly, we employ the posterior probability
as pairwise feature for
to encode the geometric relations between the body part class
. More specifically, assuming the prior probability
0) = 0.5, the posterior probability of detection
the body part label
where is obtained by conducting a normalized 2D histogram of
and
from positive training examples, analogous to the negative likelihood
. In Sec. 5.1 we also experiment with encoding the appearance into the pairwise feature by concatenating the feature
from d and
from
, as
is the output of the CNN-based part detectors. The final pairwise feature is
3.1. Probability Estimation
The coefficients of the objective function (Eq. 6) are defined by the probability ratio in the log space (Eq. 7 and Eq. 8). Here we describe the estimation of the corresponding probability density: (1) For every pair of detection and part classes, namely for any
, we estimate a probability
of the detection d being a body part of class c. (2) For every combination of two distinct detections and two body part classes, namely for any
and any
, we estimate a probability
of d and
belonging to the same person, meanwhile
are body parts of classes
, respectively. Learning. Given the features
and a Gaussian prior
on the parameters, logistic model is
parameters are estimated using ML.
Inference Given two detections d and , the coefficients
for
and
for
are obtained by Eq. 7, the coefficient
has the form
Model parameters are learned using logistic regression.
We first introduce our deep learning-based part detection models and then evaluate them on two prominent benchmarks thereby significantly outperforming state of the art.
4.1. Adapted Fast R-CNN (AFR-CNN)
To obtain strong part detectors we adapt Fast RCNN [16]. FR-CNN takes as input an image and set of class-independent region proposals [39] and outputs the softmax probabilities over all classes and refined bounding boxes. To adapt FR-CNN for part detection we alter it in two ways: 1) proposal generation and 2) detection region size. The adapted version is called AFR-CNN throughout the paper.
Detection proposals. Generating object proposals is essential for FR-CNN, meanwhile detecting body parts is challenging due to their small size and high intra-class variability. We use DPM-based part detectors [28] for proposal generation. We collect K top-scoring detections by each part detector in a common pool of N part-independent proposals and use these proposals as input to AFR-CNN. N is 2, 000 in case of single and 20, 000 in case of multiple people.
Larger context. Increasing the size of DPM detections by upscaling every bounding box by a fixed factor allows to capture more context around each part. In Sec. 4.3 we evaluate the influence of upscaling and show that using larger context around parts is crucial for best performance.
Details. Following standard FR-CNN training procedure ImageNet models are finetuned on pose estimation task. Center of a predicted bounding box is used for body part location prediction. See Appendix A for detailed parameter analysis.
Table 1. Unary only performance (PCK) of AFR-CNN on the LSP (Person-Centric) dataset. AFR-CNN is finetuned from ImageNet to MPII (lines 3-6), and then finetuned to LSP (line 7).
4.2. Dense Architecture (Dense-CNN)
Using proposals for body part detection may be suboptimal. We thus develop a fully convolutional architecture for computing part probability scoremaps.
Stride. We build on VGG [34]. Fully convolutional VGG has stride of 32 px – too coarse for precise part localization. We thus use hole algorithm [6] to reduce the stride to 8 px.
Scale. Selecting image scale is crucial. We found that scaling to a standing height of 340 px performs best: VGG receptive field sees entire body to disambiguate body parts.
Loss function. We start with a softmax loss that outputs probabilities for each body part and background. The downside is inability to assign probabilities above 0.5 to several close-by body parts. We thus re-formulate the detection as multi-label classification, where at each location a separate set of probability distributions is estimated for each part. We use sigmoid activation function on the output neurons and cross entropy loss. We found this loss to perform better than softmax and converge much faster compared to MSE [37]. Target training scoremap for each joint is constructed by assigning a positive label 1 at each location within 15 px to the ground truth, and negative label 0 otherwise.
Location refinement. In order to improve location precision we follow [16]: we add a location refinement FC layer after the FC7 and use the relative offsets from a scoremap location to the ground truth as targets.
Regression to other parts. Similar to location refinement we add an extra term to the objective function where for each part we regress onto all other part locations. We found this auxiliary task to improve the performance (c.f. Sec. 4.3).
Training. We follow best practices and use SGD for CNN training. In each iteration we forward-pass a single image. After FC6 we select all positive and random negative samples to keep the pos/neg ratio as 25%/75%. We finetune VGG from Imagenet model to pose estimation task and use training data augmentation. We train for 430k iterations with the following learning rates (lr): 10k at lr=0.001, 180k at lr=0.002, 120k at lr=0.0002 and 120k at lr=0.0001. Pretraining at smaller lr prevents the gradients from diverging.
Table 2. Unary only performance (PCK) of Dense-CNN VGG on LSP (PC) dataset. Dense-CNN is finetuned from ImageNet to MPII (line 1), to MPII+LSPET (lines 2-5), and finally to LSP (line 6).
4.3. Evaluation of Part Detectors
Datasets. We train and evaluate on three public benchmarks: “Leeds Sports Poses” (LSP) [20] (person-centric (PC)), “LSP Extended” (LSPET) [21]2, and “MPII Human Pose” (“Single Person”) [3]. The MPII training set (19185 people) is used as default. In some cases LSP training and LSPET are added to MPII (marked as MPII+LSPET in the experiments). Evaluation measures. We use the standard “PCK” metric [33, 38, 37] and evaluation scripts available on the web page of [3]. In addition, we report “Area under Curve” (AUC) computed for the entire range of PCK thresholds.
AFR-CNN. Evaluation of AFR-CNN on LSP is shown in Tab. 1. Oracle selecting per part the closest from 2, 000 proposals achieves 97.8% PCK, as proposals cover majority of the ground truth locations. Choosing a single proposal per part using DPM score achieves 23.0% PCK – not surprising given the difficulty of the body part detection problem. Rescoring the proposals using AFR-CNN with AlexNet [24] dramatically improves the performance to 56.9% PCK, as CNN learns richer image representations. Extending the regions by head size) achieves 65.1% PCK, as it incorporates more context including the information about symmetric parts and allows to implicitly encode higher-order part relations. Using data augmentation and slightly tuning training parameters improves the performance to 72.4% PCK. We refer to the Appendix A for detailed analysis. Deeper VGG architecture improves over smaller AlexNet reaching 77.9% PCK. All results so far are achieved by finetuning the ImageNet models on MPII. Further finetuning to LSP leads to remarkable 82.8% PCK: CNN learns LSP-specific image representations. Strong increase in AUC (57.0 vs. 50%) is due to improvements for smaller PCK thresholds. Using no bounding box regression leads to performance drop (81.3% PCK, 53.2% AUC): location refinement is crucial for better localization. Overall AFR-CNN obtains very good results on LSP by far outperforming the state of the art (c.f. Tab. 3, rows
). Evaluation on MPII shows competitive performance (Tab. 4, row 1).
Dense-CNN. The results are in Tab. 2. Training with VGG on MPII with softmax loss achieves 80.8% PCK thereby
(a) LSP (PC) (b) MPII Single Person Figure 2. Pose estimation results over all PCK thresholds.
outperforming AFR-CNN (c.f. Tab. 1, row 6). This shows the advantages of fully convolutional training and evaluation. Expectedly, training on larger MPII+LSPET dataset improves the results (83.0 vs. 80.8% PCK). Using crossentropy loss with sigmoid activations improves the results to 83.8% PCK, as it better models the appearance of close-by parts. Location refinement improves localization accuracy (84.8% PCK), which becomes more clear when analyzing AUC (61.5 vs. 55.6%). Interestingly, regressing to other parts further improves PCK to 86.1% showing a value of training with the auxiliary task. Finally, finetuning to LSP achieves the best result of 87.1% PCK, which is significantly higher than the best published results (c.f. Tab. 3, rows Unary-only evaluation on MPII reveals slightly higher AUC results compared to the state of the art (Tab. 4, row
4.4. Using Detections in DeepCut Models
The SPLP problem is NP-hard, to solve instances of it efficiently we select a subset of representative detections from the entire set produced by a model. In our experiments we use |D| = 100 as default detection set size. In case of the AFR-CNN we directly use the softmax output as unary probabilities: is the probability of the detection d being the part class c. For Dense-CNN detection model we use the sigmoid detection unary scores.
The aim of this paper is to tackle the multi person case. To that end, we evaluate the proposed DeepCut models on four diverse benchmarks. We confirm that both single person (SP) and multi person (MP) variants (Sec. 2) are effective on standard SP pose estimation datasets [20, 3]. Then, we demonstrate superior performance of DeepCut MP on the multi person pose estimation task.
5.1. Single Person Pose Estimation
We now evaluate single person (SP) and more general multi person (MP) DeepCut models on LSP and MPII SP benchmarks described in Sec. 4. Since this evaluation setting implicitly relies on the knowledge that all parts are present in the image we always output the full number of parts.
Results on LSP. We report per-part PCK results (Tab. 3) and results for a variable distance threshold (Fig. 2 (a)).
∗ re-evaluated using the standard protocol, for details see project page of [41] Table 3. Pose estimation results (PCK) on LSP (PC) dataset.
DeepCut SP AFR-CNN model using 100 detections improves over unary only (83.0 vs. 82.8% PCK, 58.4 vs. 57% AUC), as pairwise connections filter out some of the high-scoring detections on the background. The improvement is clear in Fig. 2 (a) for smaller thresholds. Using part appearance scores in addition to geometrical features in pairwise terms only slightly improves AUC, as the appearance of neighboring parts is mostly captured by a relatively large region centered at each part. The performance of DeepCut MP AFR-CNN matches the SP and improves over AFR-CNN alone: DeepCut MP correctly handles the SP case. Performance of DeepCut SP Dense-CNN is almost identical to unary only, unlike the results for AFR-CNN. Dense-CNN performance is noticeably higher compared to AFR-CNN, and “easy” cases that could have been corrected by a spatial model are resolved by stronger part detectors alone.
Comparison to the state of the art (LSP). Tab. 3 compares results of DeepCut models to other deep learning methods specifically designed for single person pose estimation. All DeepCuts significantly outperform the state of the art, with DeepCut SP Dense-CNN model improving by 13.7% PCK over the best known result [7]. The improvement is even more dramatic for lower thresholds (Fig. 2 (a)): for PCK @ 0.1 the best model improves by 19.9% over Tompson et al. [37], by 26.7% over Fan et al. [41], and by 32.4% PCK over Chen&Yuille [7]. The latter is interesting, as [7] use a stronger spatial model that predicts the pairwise conditioned on the CNN features, whereas DeepCuts use geometric-only pairwise connectivity. Including body part orientation information into DeepCuts should further improve the results.
Results on MPII Single Person. Results are shown in Tab. 4 and Fig. 2 (b). DeepCut SP AFR-CNN noticeably improves over AFR-CNN alone (79.8 vs. 78.8% PCK, 51.1 vs. 49.0% AUC). The improvement is stronger for smaller thresholds (c.f. Fig. 2), as spatial model improves part localization. Dense-CNN alone trained on MPII outperforms AFR-CNN (81.6 vs. 78.8% PCK), which shows the advantages of dense training and evaluation. As expected, Dense-CNN performs slightly better when trained on the larger MPII+LSPET. Finally, DeepCut Dense-CNN SP is slightly better than Dense-CNN alone leading to the best
Table 4. Pose estimation results (PCK) on MPII Single Person.
result on MPII dataset (82.4% PCK).
Comparison to the state of the art (MPII). We compare the performance of DeepCut models to the best deep learning approaches from the literature [37, 36]3. DeepCut SP Dense-CNN outperforms both [37, 36] (82.4 vs 79.6 and 82.0% PCK, respectively). Similar to them DeepCuts rely on dense training and evaluation of part detectors, but unlike them use single size receptive field and do not include multi-resolution context information. Also, appearance and spatial components of DeepCuts are trained piece-wise, unlike [37]. We observe that performance differences are higher for smaller thresholds (c.f. Fig. 2 (b)). This is remarkable, as a much simpler strategy for location refinement is used compared to [36]. Using multi-resolution filters and joint training should improve the performance.
5.2. Multi Person Pose Estimation
We now evaluate DeepCut MP models on the challenging task of MP pose estimation with an unknown number of people per image and visible body parts per person.
Datasets. For evaluation we use two public MP benchmarks: “We Are Family” (WAF) [13] with 350 training and 175 testing group shots of people; “MPII Human Pose” (“MultiPerson”) [3] consisting of 3844 training and 1758 testing groups of multiple interacting individuals in highly articulated poses with variable number of parts. On MPII, we use a subset of 288 testing images for evaluation. We first pre-finetune both AFR-CNN and Dense-CNN from ImageNet to MPII and MPII+LSPET, respectively, and further finetune each model to WAF and MPII Multi-Person. For WAF, we re-train the spatial model on WAF training set.
WAF evaluation measure. Approaches are evaluated using the official toolkit [13], thus results are directly comparable to prior work. The toolkit implements occlusion-aware “Percentage of Correct Parts (mPCP)” metric. In addition, we report “Accuracy of Occlusion Prediction (AOP)” [8].
MPII Multi-Person evaluation measure. PCK metric is suitable for SP pose estimation with known number of parts and does not penalize for false positives that are not a part of the ground truth. Thus, for MP pose estimation we use “Mean Average Precision (mAP)” measure, similar to [35, 42]. In contrast to [35, 42] evaluating the detection
of any part instance in the image disrespecting inconsistent pose predictions, we evaluate consistent part configura-tions. First, multiple body pose predictions are generated and then assigned to the ground truth (GT) based on the highest PCK[3]. Only single pose can be assigned to GT. Unassigned predictions are counted as false positives. Finally, AP for each body part is computed and mAP is reported.
Baselines. To assess the performance of AFR-CNN and Dense-CNN we follow a traditional route from the literature based on two stage approach: first a set of regions of interest (ROI) is generated and then the SP pose estimation is performed in the ROIs. This corresponds to unary only performance. ROI are either based on a ground truth (GT ROI) or on the people detector output (det ROI).
Results on WAF. Results are shown in Tab. 5. det ROI is obtained by extending provided upper body detection boxes. AFR-CNN det ROI achieves 57.6% mPCP and 73.9% AOP. DeepCut MP AFR-CNN significantly improves over AFRCNN det ROI achieving 82.2% mPCP. This improvement is stronger compared to LSP and MPII due to several reasons. First, mPCP requires consistent prediction of body sticks as opposite to body joints, and including spatial model enforces consistency. Second, mPCP metric is occlusion-aware. DeepCuts can deactivate detections for the occluded parts thus effectively reasoning about occlusion. This is supported by strong increase in AOP (85.6 vs. 73.9%). Results by DeepCut MP Dense-CNN follow the same tendency achieving the best performance of 84.7% mPCP and 86.5% AOP. Both increase in mPCP and AOP show the advantages of DeepCuts over traditional det ROI approaches.
Tab. 5 shows that DeepCuts outperform all prior methods. Deep learning method [8] is outperformed both for mPCP (84.7 vs. 80.7%) and AOP (86.5 vs. 84.9%) measures. This is remarkable, as DeepCuts reason about part interactions across several people, whereas [8] primarily focuses on the single-person case and handles multi-person scenes akin to [42]. In contrast to [8], DeepCuts are not limited by the number of possible occlusion patterns and cover person-person occlusions and other types as truncation and occlusion by objects in one formulation. DeepCuts significantly outperform [13] while being more general: unlike [13] DeepCuts do not require person detector and not limited by a number of occlusion states among people.
Qualitative comparison to [8] is provided in Fig. 3.
Figure 3. Qualitative comparison of our joint formulation DeepCut MP Dense-CNN (middle) to the traditional two-stage approach Dense-CNN det ROI (top) and the approach of Chen&Yuille [8] (bottom) on WAF dataset. In contrast to det ROI, DeepCut MP is able to disambiguate multiple and potentially overlapping persons and correctly assemble independent detections into plausible body part configurations. In contrast to [8], DeepCut MP can better predict occlusions (image 2 person from the left, top row; image 4 person 1, 4; image 5, person 2) and better cope with strong articulations and foreshortenings (image 1, person 1, 3; image 2 person 1 bottom row; image 3, person 1-2). See Appendix B for more examples.
Results on MPII Multi-Person. Obtaining a strong detector of highly articulated people having strong occlusions and truncations is difficult. We employ a neck detector as a person detector as it turned out to be the most reliable part. Full body bounding box is created around a neck detection and used as det ROI. GT ROIs were provided by the authors [3]. As the MP approach [8] is not public, we compare to SP state-of-the-art method [7] applied to GT ROI image crops.
Results are shown in Tab. 6. DeepCut MP AFR-CNN improves over AFR-CNN det ROI by 4.3% achieving 51.4% AP. The largest differences are observed for the ankle, knee, elbow and wrist, as those parts benefit more from the connections to other parts. DeepCut MP UB AFR-CNN using upper body parts only slightly improves over the full body model when compared on common parts (60.5 vs 58.2% AP). Similar tendencies are observed for Dense-CNNs, though improvements of MP UB over MP are more significant.
All DeepCuts outperform Chen&Yuille SP GT ROI, partially due to stronger part detectors compared to [7] (c.f. Tab. 3). Another reason is that Chen&Yuille SP GT ROI does not model body part occlusion and truncation always predicting the full set of parts, which is penalized by the AP measure. In contrast, our formulation allows to deactivate the part hypothesis in the initial set of part candidates thus effectively performing non-maximum suppression. In DeepCuts part hypotheses are suppressed based on the evidence from all other body parts making this process more reliable.
Table 6. Pose estimation results (AP) on MPII Multi-Person.
Articulated pose estimation of multiple people in uncontrolled real world images is challenging but of real world interest. In this work, we proposed a new formulation as a joint subset partitioning and labeling problem (SPLP). Different to previous two-stage strategies that separate the detection and pose estimation steps, the SPLP model jointly infers the number of people, their poses, spatial proximity, and part level occlusions. Empirical results on four diverse and challenging datasets show significant improvements over all previous methods not only for the multi person, but also for the single person pose estimation problem. On multi person WAF dataset we improve by 30% PCP over the traditional two-stage approach. This shows that a joint formulation is crucial to disambiguate multiple and potentially overlapping persons. Models and code available at http://pose.mpi-inf.mpg.de.
We provide additional quantitative results on LSP dataset using person-centric (PC) and observer-centric (OC) evaluation settings.
A.1. LSP Person-Centric (PC)
First, detailed performance analysis is performed when evaluating various parameters of AFR-CNN and results are reported using PCK [33] evaluation measure. Then, performance of the proposed AFR-CNN and Dense-CNN part detection models is evaluated using strict PCP [14] measure. Detailed AFR-CNN performance analysis (PCK). Detailed parameter analysis of AFR-CNN is provided in Tab. 7 and results are reported using PCK evaluation measure. Respecting parameters for each experiment are shown in the first column and parameter differences between the neighboring rows in the table are highlighted in bold. Re-scoring the 2000 DPM proposals using AFR-CNN with AlexNet [24] leads to 56.9% PCK. This is achieved using basis scale head size) of proposals and training with initial learning rate (lr) of 0.001 for 80k iterations, after which lr is reduced by 0.1, for a total number of 140k SGD iterations. In addition, bounding box regression and default IoU threshold of 0.5 for positive/negative label assignment [16] have been used. Extending the regions by 4x increases the performance to 65.1% PCK, as it incorporates more context including the information about symmetric body parts and allows to implicitly encode higher-order body part relations into the part detector. No improvements observed for larger scales. Increasing lr to 0.003, lr reduction step to 160k and training for a larger number of iterations (240k) improves the results to 67.4, as higher lr allows for for more significant updates of model parameters when finetuned on the task of human body part detection. Increasing the number of training examples by reducing the training IoU threshold to 0.4 results into slight performance improvement (68.8 vs. 67.4% PCK). Further increasing the number of training samples by horizontally flipping each image and performing translation and scale jittering of the ground truth training samples improves the performance to 69.6% PCK and 42.3% AUC. The improvement is more pronounced for smaller distance thresholds (42.3 vs. 40.9% AUC): localization of body parts is improved due to the increased number of jittered samples that significantly overlap with the ground truth. Further increasing the lr, lr reduction step and total number of iterations altogether improves the performance to 72.4% PCK, and very minor improvements are observed when training longer. All results above are achieved by finetuning the AlexNet architecture from the ImageNet model on the MPII training set. Further finetuning the MPII-finetuned model on the LSP training set increases the performance to 77.9% PCK, as the network learns LSP-specific image representations. Using the deeper VGG [34] architecture improves over more shallow AlexNet (77.9 vs. 72.4% PCK, 50.0 vs. 44.6% AUC). Funetuning VGG on LSP achieves remarkable 82.8% PCK and 57.0% AUC. Strong increase in AUC (57.0 vs. 50%) characterizes the improvement for smaller PCK evaluation thresholds. Switching off bounding box regression results into performance drop (81.3% PCK, 53.2% AUC) thus showing the importance of the bounding box regression for better part localization. Overall, we demonstrate that proper adaptation and tweaking of the state-of-the-art generic object detector FR-CNN [16] leads to a strong body part detection model that dramatically improves over the vanilla FR-CNN (82.8 vs. 56.9% PCK, 57.8 vs. 35.9% AUC) and signifi-cantly outperforms the state of the art (+9.4% PCK over the best known PCK result [7] and +9.7% AUC over the best known AUC result [37].
Overall performance using PCP evaluation measure. Performance when using the strict “Percentage of Correct Parts (PCP)” [14] measure is reported in Tab. 8. In contrast to PCK measure evaluating the accuracy of predicting body joints, PCP evaluation metric measures the accuracy of predicting body part sticks. AFR-CNN achieves 78.3% PCP. Similar to PCK results, DeepCut SP AFR-CNN slightly improves over unary alone, as it enforces more consistent predictions of body part sticks. Using more general multi-person DeepCut MP AFR-CNN model results into similar performance, which shows the generality of DeepCut MP method. DeepCut SP Dense-CNN slightly improves over Dense-CNN alone (84.3 vs. 83.9% PCP) achieving the best PCP result on LSP dataset using PC annotations. This is in contrast to PCK results where performance differences DeepCut SP Dense-CNN vs. Dense-CNN alone are minor.
We now compare the PCP results to the state of the art. The DeepCut models outperform all other methods by a large margin. The best known PCP result by Chen&Yuille [7] is outperformed by 10.7% PCP. This is interesting, as their deep learning based method relies on the image conditioned pairwise terms while our approach uses more simple geometric only connectivity. Interestingly, AFR-CNN alone outperforms the approach of Fan et al. [41] (78.3 vs. 70.1% PCP), who build on the previous version of the R-CNN detector [17]. At the same time, the best performing dense architecture DeepCut SP Dense-CNN outperforms [41] by +14.2% PCP. Surprisingly, DeepCut SP Dense-CNN dramatically outperforms the method of Tompson et al. [37] (+17.7% PCP) that also produces dense score maps, but additionally includes multi-scale receptive fields and jointly trains appearance and spatial models in a single deep learning framework. We envision that both advances can further improve the performance of DeepCut models. Finally, all proposed approaches significantly outperform earlier non-deep
Table 7. PCK performance of AFR-CNN (unary) on LSP (PC) dataset. AFR-CNN is finetuned from ImageNet on MPII (lines 1-6, 8), and then finetuned on LSP (lines 7, 9).
∗ re-evaluated using the standard protocol, for details see project page of [41] Table 8. Pose estimation results (PCP) on LSP (PC) dataset.
learning based methods [40, 28] relying on hand-crafted image features.
A.2. LSP Observer-Centric (OC)
We now evaluate the performance of the proposed part detection models on LSP dataset using the observer-centric (OC) annotations [12]. In contrast to the person-centric (PC) annotations used in all previous experiments, OC annotations do not penalize for the right/left body part prediction flips and count a body part to be the right body part, if it is on the right side of the line connecting pelvis and neck, and a body part to be the left body part otherwise.
Evaluation is performed using the official OC annotations provided by [27, 12]. Prior to evaluation, we first finetune the AFR-CNN and Dense-CNN part detection models from ImageNet on MPII and MPII+LSPET training sets, respectively, (same as for PC evaluation), and then further finetuned the models on LSP OC training set. PCK evaluation measure. Results using OC annotations and PCK evaluation measure are shown in Tab. 9 and in Fig. 4. AFR-CNN achieves 84.2% PCK and 58.1% AUC. This result is only slightly better compared to AFR-CNN evaluated using PC annotations (84.2 vs 82.8% PCK, 58.1 vs. 57.0% AUC). Although PC annotations correspond to a harder task, only small drop in performance when using PC annotations shows that the network can learn to accurately predict person’s viewpoint and correctly label left/right limbs in most cases. This is contrast to earlier approaches based on hand-crafted features whose perfor-
Table 9. Pose estimation results (PCK) on LSP (OC) dataset.
Figure 4. Pose estimation results over all PCK thresholds on LSP (OC) dataset.
mance drops much stronger when evaluated in PC evaluation setting (e.g. [28] drops from 71.0% PCK when using OC annotations to 58.0% PCK when using PC annotations). Similar to PC case, Dense-CNN detection model outperforms AFR-CNN (88.2 vs. 84.2% PCK and 65.0 vs. 58.1% AUC). The differences are more pronounced when examining the entire PCK curve for smaller distance thresholds (c.f. Fig. 4).
Comparing the performance by AFR-CNN and Dense-CNN to the state of the art, we observe that both proposed approaches significantly outperform other methods. Both deep learning based approaches of Chen&Yuille [7] and Ouyang et al. [26] are outperformed by +10.7 and +18.2% PCK when compared to the best performing Dense-CNN. Analysis of PCK curve for the entire range of PCK distance thresholds reveals even larger performance differences (c.f. Fig. 4). The results using OC annotations confirm our findings from PC evaluation and clearly show the advantages of the proposed part detection models over the state-of-the-art deep learning methods [7, 26], as well as over earlier pose estimation methods based on hand-crafted image features [28, 22, 30]. PCP evaluation measure. Results using OC annotations
Table 10. Pose estimation results (PCP) on LSP (OC) dataset.
and PCP evaluation measure are shown in Tab. 10. Overall, the trend is similar to PC evaluation: both proposed approaches significantly outperform the state-of-the-art methods with Dense-CNN achieving the best result of 85.0% PCP thereby improving by +10% PCP over the best published result [7].
Qualitative comparison of our joint formulation DeepCut MP Dense-CNN to the traditional two-stage approach Dense-CNN det ROI on MPII Multi-Person dataset is shown in Fig. 6 and 7. Dense-CNN det ROI works well when multiple fully visible individuals are sufficiently separated and thus their body parts can be partitioned based on the person detection bounding box. In this case the strong Dense-CNN body part detection model can correctly estimate most of the visible body parts (image 16, 17, 19). However, Dense-CNN det ROI cannot tell apart the body parts of multiple individuals located next to each other and possibly occluding each other, and often links the body parts across the individuals (images 1-16, 19-20). In addition, Dense-CNN det ROI cannot reason about occlusions and truncations always providing a prediction for each body part (image 4, 6, 10). In contrast, DeepCut MP Dense-CNN is able to correctly partition and label an initial pool of body part candidates (each image, top row) into subsets that correspond to sets of mutually consistent body part candidates and abide to mutual consistency and exclusion constraints (each image, row 2), thereby outputting consistent body pose predictions (each image, row 3). pairwise terms allow to partition the initial set of part detection candidates into valid pose configurations (each image, row 2: personclusters highlighted by dense colored connections).
pairwise terms facilitate clustering of multiple body part candidates of the same body part of the same person (each image, row 2: markers of the same type and color). In addition,
pairwise terms facilitate a repulsive property that prevents nearby part candidates of the same type to be associated to different people (image 1: detections of the left shoulder are assigned to the front person only). Furthermore, DeepCut MP Dense-CNN allows to either merge or deactivate part hypotheses thus effectively performing non-maximum suppression and reasoning about body part occlusions and truncations (image 3, row 2: body part hypotheses on the background are deactivated (black crosses); image 6, row 2: body part hypotheses for the truncated body parts are deactivated (black crosses); image 1-6, 8-9, 13-14, row 3: only visible body parts of the partially occluded people are estimated, while non-visible body parts are correctly predicted to be occluded). These qualitative examples show that DeepCuts MP can successfully deal with the unknown number of people per image and the unknown number of visible body parts per person.
[1] A. Alush and J. Goldberger. Ensemble segmentation using efficient integer linear programming. TPAMI, 34(10):1966– 1977, 2012. 2
[2] B. Andres, J. H. Kappes, T. Beier, U. K¨othe, and F. A. Hamprecht. Probabilistic image segmentation with closedness constraints. In ICCV, 2011. 2
[3] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR’14. 1, 5, 6, 7, 8
[4] N. Bansal, A. Blum, and S. Chawla. Correlation clustering. Machine Learning, 56(1–3):89–113, 2004. 2
[5] V. Belagiannis, S. Amin, M. Andriluka, B. Schiele, N. Navab, and S. Ilic. 3D pictorial structures for multiple human pose estimation. In CVPR’14. 2
[6] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR, 2015. 5
[7] X. Chen and A. Yuille. Articulated pose estimation by a graphical model with image dependent pairwise relations. In NIPS’14. 1, 2, 6, 8, 9, 10, 11
[8] X. Chen and A. Yuille. Parsing occluded people by flexible compositions. In CVPR, 2015. 2, 7, 8, 11, 12
[9] S. Chopra and M. Rao. The partition problem. Mathematical Programming, 59(1–3):87–115, 1993. 2, 3
[10] E. D. Demaine, D. Emanuel, A. Fiat, and N. Immorlica. Correlation clustering in general weighted graphs. Theoretical Computer Science, 361(2–3):172–187, 2006. 2
[11] M. M. Deza and M. Laurent. Geometry of Cuts and Metrics. Springer, 1997. 2
[12] M. Eichner and V. Ferrari. Appearance sharing for collective human pose estimation. In ACCV’12. 10
[13] M. Eichner and V. Ferrari. We are family: Joint pose estimation of multiple persons. In ECCV’10. 2, 7
[14] V. Ferrari, M. Marin, and A. Zisserman. Progressive search space reduction for human pose estimation. In CVPR’08. 9
1 23 45
12 3
4 5
12 3
4 5
1 2 3
4 5
Figure 5. Qualitative comparison of our joint formulation DeepCut MP Dense-CNN (rows 2, 5) to the traditional two-stage approach Dense-CNN det ROI (rows 1, 4) and to the approach of Chen&Yuille [8] (rows 3, 6) on WAF dataset. det ROI does not reason about occlusion and often predicts inconsistent body part configurations by linking the parts across the nearby staying people (image 4, right shoulder and wrist of person 2 are linked to the right elbow of person 3; image 5, left elbow of person 4 is linked to the left wrist of person 3). In contrast, DeepCut MP predicts body part occlusions, disambiguates multiple and potentially overlapping people and correctly assembles independent detections into plausible body part configurations (image 4, left arms of people 1-3 are correctly predicted to be occluded; image 5, linking of body parts across people 3 and 4 is corrected; image 7, occlusion of body parts is correctly predicted and visible parts are accurately estimated). In contrast to Chen&Yuille [8], DeepCut MP better predicts occlusions of person’s body parts by the nearby staying people (images 1, 3-9), but also by other objects (image 2, left arm of person 1 is occluded by the chair). Furthermore, DeepCut MP is able to better cope with strong articulations and foreshortenings (image 1, person 6; image 3, person 2; image 5, person 4; image 7, person 4; image 8, person 1). Typical DeepCut MP failure case is shown in image 10: the right upper arm of person 3 and both arms of person 4 are not estimated due to missing part detection candidates.
[15] G. Ghiasi, Y. Yang, D. Ramanan, and C. Fowlkes. Parsing occluded people. In CVPR’14. 7
[16] R. Girshick. Fast r-cnn. In ICCV’15. 4, 5, 9
[17] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR’14. 9
[18] G. Gkioxari, B. Hariharan, R. Girshick, and J. Malik. Using k-poselets for detecting people and localizing their keypoints. In CVPR’14. 1, 2
[19] H. Jiang and D. R. Martin. Global pose estimation using non-tree models. In CVPR’09. 2
[20] S. Johnson and M. Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC’10. 5, 6
[21] S. Johnson and M. Everingham. Learning Effective Human Pose Estimation from Inaccurate Annotation. In CVPR’11. 1, 5
[22] M. Kiefel and P. Gehler. Human pose estimation with fields of parts. In ECCV’14. 10, 11
[23] S. Kim, C. Yoo, S. Nowozin, and P. Kohli. Image segmentation using higher-order correlation clustering. TPAMI, 36:1761–1774, 2014. 2
[24] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS’12. 5, 9
[25] L. Ladicky, P. H. Torr, and A. Zisserman. Human pose estimation using a joint pixel-wise and part-wise formulation. In CVPR’13. 2
[26] W. Ouyang, X. Chu, and X. Wang. Multi-source deep learning for human pose estimation. In CVPR’14. 10, 11
[27] L. Pishchulin, M. Andriluka, P. Gehler, and B. Schiele. Poselet conditioned pictorial structures. In CVPR’13. 10
[28] L. Pishchulin, M. Andriluka, P. Gehler, and B. Schiele. Strong appearance and expressive spatial models for human pose estimation. In ICCV’13. 2, 4, 10, 11
[29] L. Pishchulin, A. Jain, M. Andriluka, T. Thormaehlen, and B. Schiele. Articulated people detection and pose estimation: Reshaping the future. In CVPR’12. 1, 2
[30] V. Ramakrishna, D. Munoz, M. Hebert, A. J. Bagnell, and Y. Sheikh. Pose machines: Articulated pose estimation via inference machines. In ECCV’14. 10, 11
[31] D. Ramanan. Learning to parse images of articulated objects. In NIPS’06. 2
[32] X. Ren, A. C. Berg, and J. Malik. Recovering human body configurations using pairwise constraints between parts. In ICCV’05. 2
[33] B. Sapp and B. Taskar. Multimodal decomposable models for human pose estimation. In CVPR’13. 1, 2, 5, 9
[34] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR,14. 5, 9
[35] M. Sun and S. Savarese. Articulated part-based model for joint object detection and pose estimation. In ICCV’11. 1, 2, 7
[36] J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler. Efficient object localization using convolutional networks. In CVPR’15. 1, 2, 7
[37] J. J. Tompson, A. Jain, Y. LeCun, and C. Bregler. Joint training of a convolutional network and a graphical model for human pose estimation. In NIPS’14. 1, 5, 6, 7, 9, 10
[38] A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. In CVPR’14. 2, 5
[39] J. Uijlings, K. van de Sande, T. Gevers, and A. Smeulders. Selective search for object recognition. IJCV’13. 4
[40] F. Wang and Y. Li. Beyond physical connections: Tree models in human pose estimation. In CVPR’13. 10
[41] X. Fan, K. Zheng, Y. Lin, and S. Wang. Combining local appearance and holistic view: Dual-source deep neural networks for human pose estimation. In CVPR’15. 6, 9, 10
[42] Y. Yang and D. Ramanan. Articulated human detection with flexible mixtures of parts. PAMI’13. 2, 7
[43] J. Yarkony, A. Ihler, and C. C. Fowlkes. Fast planar correlation clustering for image segmentation. In ECCV, 2012. 2