Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction for Multiple Unknown Objects via 2D Point Trackers

Tzu-Yuan Lin1, Ho Jae Lee1, Kevin Doherty2, Yonghyeon Lee1, Sangbae Kim1
1Biomimetic Robotics Lab, MIT    2Boston Dynamics
Massachusetts Institute of Technology
ECCV 2026
Paper arXiv Video Code YCBMultiTrack (Coming Soon)

Live Demo

Loading the interactive viewer…

These are real Point2Pose runs, played back in the Rerun viewer. Pick a sequence above, drag to orbit, scroll to zoom, and scrub the timeline to watch the point map, the keyframes and the mesh build up as tracking runs.

Point2Pose tracks the 6D pose of multiple unknown objects from a single RGB-D video, without CAD models or category priors. When an object is completely occluded, tracking resumes the moment it comes back into view.

Abstract

We present Point2Pose, a model-free method for causal 6D pose tracking of multiple rigid objects from monocular RGB-D video. Initialized only from sparse image points on the objects, our approach tracks multiple unseen objects without requiring object CAD models or category priors. Point2Pose leverages a 2D point tracker to obtain long-range correspondences, enabling instant recovery after complete occlusion. Simultaneously, the system incrementally reconstructs an online Truncated Signed Distance Function (TSDF) representation of the tracked targets. Alongside the method, we introduce a new multi-object tracking dataset comprising both simulation and real-world sequences, with motion-capture ground truth for evaluation. Experiments show that Point2Pose trades some single-object pose accuracy for broader model-free tracking capabilities, including multi-object tracking and recovery from complete occlusion.

How It Works

Point2Pose is initialized from only a handful of points on each object of interest. From there it runs fully model-free:

  • Long-range 2D point tracking. A learned point tracker maintains pixel correspondences across hundreds of frames, acting as a persistent data-association module — no brittle frame-to-frame feature matching required.
  • Lift to 6D. Tracked 2D points are lifted to 3D with the depth stream to build a per-object keypoint map, and each object's 6-DoF pose is recovered via map-based registration.
  • Graph optimization. Poses and the keypoint map are jointly refined for temporally consistent, drift-resistant trajectories.
  • Online TSDF reconstruction. As poses are estimated, the system fuses depth into a per-object Truncated Signed Distance Function, building a 3D model on the fly.
  • Instant occlusion recovery. Because the point tracker re-detects correspondences when an object reappears, Point2Pose snaps back to the correct pose immediately after total occlusion — no separate relocalization stage.
Point2Pose pipeline overview
Pipeline. A single 2D tracker pass produces long-range correspondences that are lifted to 3D and registered to per-object keypoint maps; graph optimization yields the pose output while depth is fused into a per-object TSDF.
Long-range point tracking. Persistent point queries (colored trails) stay attached to the object surface across hundreds of frames, providing the correspondences that drive pose estimation and occlusion recovery.

Instant Recovery from Complete Occlusion

Prior model-free trackers rely on inter-frame matching and lose objects once they are fully occluded or leave the view. Because Point2Pose tracks persistent point queries, it re-establishes correspondences the instant an object reappears and recovers its 6D pose — with no heavyweight relocalization step.

The object is carried out of view upside-down, then brought back upright — Point2Pose re-detects it and snaps to the correct 6D pose the instant it reappears.

Real-World Multi-Object Tracking

On hand-held and manipulator sequences, Point2Pose maintains accurate 6D trajectories for several objects at once, through fast motion, clutter, and inter-object occlusion.

Overview. Multiple unknown objects are tracked in 6D from a single RGB-D video, with pose recovered immediately after each object is fully occluded.
YCBMultiTrack — Real-World. Two objects are tracked simultaneously in 6D as a hand rearranges them, through fast motion and mutual occlusion.

In the Wild

Point2Pose is not limited to benchmark objects. Initialized from just a few clicked points, it tracks the 6D pose of arbitrary everyday objects — no CAD model, no category prior, and no retraining.

Banana. Dense long-range point tracks (colored dots) drive the 6D pose on a textured, deformable object.
Bottle. A clicked-once household object, tracked in 6D with no CAD model.
Plush toy. A low-texture, non-rigid object tracked through free rotation in the hand.
Phone + AirPods. Two everyday objects tracked at once through fast hand manipulation — no CAD model or category prior.

Interactive 3D Reconstructions

While tracking, Point2Pose fuses depth into a per-object Truncated Signed Distance Function, producing a colored 3D model of each target on the fly — no CAD model required. Drag to rotate, scroll to zoom.

Pitcher (HO3D AP14).
Plush toy.
Toy airplane.

The YCBMultiTrack Dataset

We introduce YCBMultiTrack, a new dataset for dynamic multi-object RGB-D pose tracking, built from a subset of the YCB objects. Unlike existing benchmarks — HO3D and YCBInEOAT focus on a single object, YCB-Video is mostly static, and HOT3D lacks dense depth — YCBMultiTrack captures independent object motion with inter-object and full-view occlusions, and provides ground-truth poses from simulation (Isaac Lab) and from an OptiTrack motion-capture system.

  • Synthetic split. Photorealistic renders with exact depth and pose, single- and two-object scenes on linear/circular trajectories.
  • Real-world split. Intel RealSense D435i RGB-D with motion-capture ground truth; single-, two-, and three-object sequences.
  • Annotations. Per-object segmentation masks (SAM 2), camera intrinsics, and per-frame visibility / full-occlusion labels.
Synthetic frame 28
Synthetic frame 94
Synthetic frame 215
Synthetic frame 276

Synthetic split. A two-object sequence over time (photorealistic renders with exact depth and pose).

Real sequence frame 1: three objects arranged
Real sequence frame 2: an object lifted
Real sequence frame 3: objects rearranged
Real sequence frame 4: final configuration

Real-world split. A three-object sequence over time (RealSense D435i RGB-D with motion-capture ground truth).

BibTeX

@inproceedings{lin2026point2pose,
  title     = {Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction
               for Multiple Unknown Objects via 2D Point Trackers},
  author    = {Lin, Tzu-Yuan and Lee, Ho Jae and Doherty, Kevin and Lee, Yonghyeon and Kim, Sangbae},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026},
}