Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction for Multiple Unknown Objects via 2D Point Trackers
ECCV 2026
We present Point2Pose, a model-free method for causal 6D pose tracking of multiple rigid objects from monocular RGB-D video. Initialized only from sparse image points on the objects, our approach tracks multiple unseen objects without requiring object CAD models or category priors. Point2Pose leverages a 2D point tracker to obtain long-range correspondences, enabling instant recovery after complete occlusion. Simultaneously, the system incrementally reconstructs an online Truncated Signed Distance Function (TSDF) representation of the tracked targets. Alongside the method, we introduce a new multi-object tracking dataset comprising both simulation and real-world sequences, with motion-capture ground truth for evaluation. Experiments show that Point2Pose trades some single-object pose accuracy for broader model-free tracking capabilities, including multi-object tracking and recovery from complete occlusion.
These are real Point2Pose runs, played back in the Rerun viewer. Pick a sequence above, drag to orbit, scroll to zoom, and scrub the timeline to watch the point map, the keyframes and the mesh build up as tracking runs.
Point2Pose is initialized from only a handful of points on each object of interest. From there it runs fully model-free, in real time at 30 Hz on a single NVIDIA RTX 4090 GPU:
Prior model-free trackers rely on inter-frame matching and lose objects once they are fully occluded or leave the view. Because Point2Pose tracks persistent point queries, it re-establishes correspondences the instant an object reappears and recovers its 6D pose — with no heavyweight relocalization step.
On hand-held and manipulator sequences, Point2Pose maintains accurate 6D trajectories for several objects at once, through fast motion, clutter, and inter-object occlusion.
Point2Pose is not limited to benchmark objects. Initialized from just a few clicked points, it tracks the 6D pose of arbitrary everyday objects — no CAD model, no category prior, and no retraining.
While tracking, Point2Pose fuses depth into a per-object Truncated Signed Distance Function, producing a colored 3D model of each target on the fly — no CAD model required. Drag to rotate, scroll to zoom.
We introduce YCBMultiTrack, a new dataset for dynamic multi-object RGB-D pose tracking, built from a subset of the YCB objects. Unlike existing benchmarks — HO3D and YCBInEOAT focus on a single object, YCB-Video is mostly static, and HOT3D lacks dense depth — YCBMultiTrack captures independent object motion with inter-object and full-view occlusions, and provides ground-truth poses from simulation (Isaac Lab) and from an OptiTrack motion-capture system.
Synthetic split. A two-object sequence over time (photorealistic renders with exact depth and pose).
Real-world split. A three-object sequence over time (RealSense D435i RGB-D with motion-capture ground truth).
@inproceedings{lin2026point2pose,
title = {Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction
for Multiple Unknown Objects via 2D Point Trackers},
author = {Lin, Tzu-Yuan and Lee, Ho Jae and Doherty, Kevin and Lee, Yonghyeon and Kim, Sangbae},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026},
}