CDIS: Cross-Dimensional Class-Agnostic 3D Instance Segmentation via 2D Mask Tracking and 3D–2D Projection Merging

Interactive Demo

These are real CDIS outputs — ScanNet200 scenes that CDIS segmented into class-agnostic 3D instance masks by tracking 2D masks across frames and merging them with 3D superpoints, with no 3D-supervised network. Pick a scene below, drag to rotate, scroll to zoom, and switch between the raw 3D scan and CDIS's predicted instances.

CDIS is class-agnostic — it labels instances, not object categories — so there is no open-vocabulary query here. Each color is a distinct 3D instance produced by tracking 2D masks across frames and merging them with 3D superpoints; structural surfaces (wall, floor) and points CDIS left unassigned are shown in gray. Computed offline and replayed here for an instant response.

Abstract

Class-agnostic 3D instance segmentation is critical for robotic systems operating in unknown environments, enabling perception of previously unseen objects for reliable manipulation and navigation. Existing approaches typically project per-frame 2D instance masks into 3D and merge them, which often breaks object identities across time and yields fragmented 3D instances. We introduce Cross-Dimensional Class-Agnostic 3D Instance Segmentation (CDIS), a zero-shot framework that explicitly tracks 2D instance masks across frames and associates them with 3D superpoints, creating a feedback loop between 2D and 3D. This cross-dimensional reasoning links temporally stable 2D tracks with spatially coherent 3D regions, producing globally consistent 3D instance labels without any 3D-specific training. Experiments on benchmark datasets demonstrate that CDIS achieves higher accuracy and consistency than state-of-the-art zero-shot methods, while remaining efficient and scalable to diverse real-world environments.

Method

CDIS produces 3D instance segmentation in three stages:

1. 2D instance tracking — a 2D mask predictor generates per-frame instance masks, which are projected into 3D using depth and warped from past frames via the relative camera transform. Instances are matched across frames by 2D mask IoU, giving temporally consistent 2D instance labels even when a single frame's segmentation is noisy.
2. 3D-guided 2D instance merging — pre-computed 3D superpoints are projected onto every frame; each superpoint is assigned the instance label it overlaps most in that frame. Instance pairs whose associated superpoints agree strongly (3D IoU) are merged, hierarchically halving the number of frame groups until a single consistent set of instance identities remains.
3. 3D instance consolidation — since multiple instance IDs can share superpoints, an iterative refinement resolves overlapping and duplicate assignments (via 3D IoMin and temporal co-occurrence analysis) until every superpoint maps to exactly one instance ID.

This bidirectional design — 2D tracking corrects short-term per-frame errors, while 3D superpoint structure resolves spatial ambiguity and prevents long-term drift — is what the paper calls cross-dimensional processing.

2D Instance Tracking

This is CDIS's real cached tracking output on scene0011_00 — 16 sampled frames spanning a ~230-frame stretch of the camera trajectory, with each 2D instance mask colored by its tracked id. Scrub through them (or press play): the same object keeps the same color as the camera moves, even as other instances enter and leave the frame.

▶ Play frame 1 / 16

Overlay is the raw RGB frame blended with CDIS's tracked 2D instance mask (from CDIS/matching_2d.py); gray means no instance at that pixel.

CDIS also ships its own visualization for the tracker's internal state: the current frame's mask (leftmost) alongside the last four frames warped into it, all colored by tracked id. Watching the right-hand panels shows exactly what the tracker is matching the current frame against, frame by frame, across the full trajectory.

Rendered by the codebase's own visualize_warped_masks_over_time debug visualization (utils/util.py).

Tracking doesn't just match against the previous frame — each frame is matched against a sliding queue of the last q_max = 5 frames (matching_2d.queue_size), so an instance can be re-identified even after going undetected for a few frames in between. This is a real example from the same scene: the door edge (highlighted red, id 679) is tracked in frame 410, missed entirely by the 2D mask predictor for three straight frames, then re-matched to the same id in frame 414 — 4 frames back, within the queue window. A predictor that only compared to frame t−1 would have assigned it a brand-new id here.

Results

Class-agnostic instance segmentation AP on ScanNet200 and ScanNet++ (paper Tables I & II).

Mask3D (a fully-supervised model trained with 3D ground-truth masks) scores higher still (AP 39.7 on ScanNet200) but requires dense 3D annotation; CDIS is zero-shot and uses no 3D-specific training.

Real-World Experiments

CDIS was also validated on real-world RGB-D data (not from ScanNet200/ScanNet++) captured with RTAB-Map visual SLAM across an office, a bedroom, and a kitchen — reconstructed via TSDF volume integration, with no 3D-specific training involved.

BibTeX

@inproceedings{kim2025cdis,
  title={CDIS: Cross-Dimensional Class-Agnostic 3D Instance Segmentation via 2D Mask Tracking and 3D--2D Projection Merging},
  author={Kim, Juno and Yoon, Hye-Jung and Park, Yesol and Zhang, Byoung-Tak},
  booktitle={2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2025},
  organization={IEEE}
}

Acknowledgements & License

This work was partly supported by the IITP (RS-2021-II212068-AIHub/10%, RS-2021-II211343-GSAI/15%, RS-2022-II220951-LBA/15%, RS-2022-II220953-PICA/20%), NRF (RS-2024-00353991-SPARC/20%), and KEIT (RS-2024-00423940/10%) grants funded by the Korean government.

This work builds on Segment Anything, CropFormer / Entity, Pointcept, SAM3D, and the ScanNet evaluation toolkit, and extends our earlier project OV-MAP. The code release is MIT-licensed (see the LICENSE file); vendored/dependent components retain their own licenses. This is a separate license from this website's own template/code.