(MP)2T: Multiple People Multiple Parts Tracker
We present a method for multi-target tracking that exploits the persistence in detection of object parts. While the implicit representation and detection of body parts have recently been leveraged for improved human detection, ours is the first method that attempts to temporally constrain the location of human body parts with the express purpose of improving pedestrian tracking. We pose the problem of simultaneous tracking of multiple targets and their parts in a network flow optimization framework and show that parts of this network need to be optimized separately and iteratively, due to inter-dependencies of node and edge costs. Given potential detections of humans and their parts separately, an initial set of pedestrian tracklets is first obtained, followed by explicit tracking of human parts as constrained by initial human tracking. A merging step is then performed whereby we attempt to include part-only detections for which the entire human is not observable. This step employs a selective appearance model, which allows us to skip occluded parts in description of positive training samples. The result is high confidence, robust trajectories of pedestrians as well as their parts, which essentially constrain each other’s locations and associations, thus improving human tracking and parts detection. We test our algorithm on multiple real datasets and show that the proposed algorithm is an improvement over the state-of-the-art.
Highlights of the paper:
- Simultaneously accomplish multiple people and multiple parts tracking.
- Not utilize DPM part results, but use dense parts to postpone the person-track association until the temporal consistency of the parts is reached.
- Take advantage of as few as a single part’s trajectory, to infer the presence/absence and approximate trajectory of the whole person under severe occlusions.
Proposed Framework and Key Steps
We propose a framework which attempts to solve the pedestrian tracking problem by simultaneously constraining the detection, temporal persistence, and appearance similarity of the detected humans, as well as the observable or inferable parts they are composed of. This method follows three key steps:
- Step 1: Associate pedestrian detections to obtain several short tracklets.
- Step 2: Associate part detections by computing the likelihood leveraging pedestrian tracklets. Revert pedestrian associations that do not conform to part tracklets.
- Step 3: Perform simultaneous association between all tracklets jointly.
Given a video, we begin by applying a state-of-the-art human detector to detect pedestrians in all frames and also obtain the detector confidence. We then create a flow network for these detections. The successive shortest paths algorithm is then used to obtain tracklets for each set of pedestrian detections. Due to problems inherent in the surveillance task, including mis-detections, merged detections, occlusions, clutter, and false positives, these tracklets are less than ideal. They break at points of occlusion and mis-detections, while merged detections and false positives allow connections between unrelated shorter tracklets.In order to break the wrong associations, we attempt to leverage the temporal persistence of pedestrian parts. We therefore first perform explicit tracking of these parts.
For part tracking, we take the detections for all parts within the spatiotemporal tubes representing a pedestrian tracklet. Again, we employ the k-shortest path algorithm, for which we need the node and edge weights in the flow network of body parts. The following figure demonstrates a vivid part tracking.
The following figure shows Gaussian models of relative part locations.
We use pedestrian-specific appearance modeling to feed the SVM model and we shows three different models for pedestrian appearance learning in the following figure. The first row demostrates a simple average over the entire person bounding box, the middle row uses detected DPM parts, and our tracked parts are used for examples in the bottom row. Using the entire person bounding box the model is very vague. Since a certain minimum number of parts are always detected in DPM, the model contains background when the person is partially occluded. The handling of occlusion and clutter for parts while temporally aligning them, makes our model more accurate.
Merging of Pedestrian and Part Tracklets
Finally, we employ pedestrian parts to merge the tracklets into correct, high confidence trajectories, again by leveraging the flow network optimization algorithm. The entire pedestrian tracklets will now act as nodes, whose costs will be the output of the previous network flow (chosen nodes and edges). We observe that two tracklets to be associated may have a temporal gap due to problems with pedestrian detections within the gap. The figure shown below illustrates the final step of merging pedestrian tracklets.
We evaluated our method using four challenging datasets: Town Center and PETS2009 datasets which are publicly available, and two new datasets: the Parking Lot and Airport sequences.
The quantitative comparisons are show in the following table, followed by specific frames containing the results of detections and trackings in these four datasets.
|Benfold & Reid||80.3||61.3||82||79|
Moreover, we explicitly ground truthed human parts and quantified our results as follows.
|Benfold & Reid||50.8||45.4||73.8||71|
|Part2 (Left shoulder)|
|Part3 (Right shoulder)|
|Part4 (Left bottom torso)|
|Part5 (Right bottom torso)|
|Part6 (Upper legs)|
|Part7 (Left lower leg)|
|Part8 (Right lower leg)|
Hamid Izadinia, Imran Saleemi, Wenhui Li and Mubarak Shah, “(MP)2T: Multiple People Multiple Parts Tracker”, European Conference on Computer Vision 2012, Florence, Italy, October 7-13, 2012. [PDF][PPT(203MB)][BibTex]
The following specifies the part annotation for Towncener dataset:
ReadMe file (1KB): ReadMe file
Parts annotations (547KB): Parts annotations
Demo of parts annoatations(12.4MB): Video showing overlaid annotated parts