Ov-Map: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots

Interactive Demo

This is a real OV-MAP output — a ScanNet200 scene (scene0011_00) that OV-MAP segmented into per-instance 3D masks with no 3D-supervised network. Drag to rotate, scroll to zoom. Switch between the 3D scene (real colors) and OV-MAP's class-agnostic instance masks, or click an open-vocabulary query to light up every instance the map matches to that word.

Each query is resolved by matching a CLIP text embedding against every 3D instance's open-vocabulary feature — the same mechanism a robot would use to localize an object from a spoken command. Highlighted regions are OV-MAP's predictions, computed offline and replayed here for an instant response.

Abstract

We introduce OV-MAP, a novel approach to open-world 3D mapping for mobile robots by integrating open-features into 3D maps to enhance object recognition capabilities. A significant challenge arises when overlapping features from adjacent voxels reduce instance-level precision, as features spill over voxel boundaries, blending neighboring regions together. Our method overcomes this by employing a class-agnostic segmentation model to project 2D masks into 3D space, combined with a supplemented depth image created by merging raw and synthetic depth from point clouds. This approach, along with a 3D mask voting mechanism, enables accurate zero-shot 3D instance segmentation without relying on 3D supervised segmentation models. We assess the effectiveness of our method through comprehensive experiments on public datasets such as ScanNet200 and Replica, demonstrating superior zero-shot performance, robustness, and adaptability across diverse environments. Additionally, we conducted real-world experiments to demonstrate our method's adaptability and robustness when applied to diverse real-world environments.

Method

OV-MAP's pipeline has four stages:

1. Depth supplementation — raw depth is merged with synthetic depth rendered from the reconstructed point cloud, correcting gaps and noise (e.g. from reflective surfaces) in the raw sensor depth before it's used for 2D-to-3D projection.
2. 2D mask projection — CropFormer produces per-frame class-agnostic 2D masks from the RGB-D input, which are projected into 3D using the supplemented depth.
3. Merge & dominant voting — per-frame 3D masks are merged, and a voting mechanism over mesh-segmented areas assigns each area to its dominant mask group — yielding per-instance 3D proposals without any 3D-supervised network.
4. Open-vocabulary labeling — each 3D instance is labeled with CLIP features computed from its highest-scoring 2D view, enabling open-vocabulary querying of the resulting map.

Depth Supplementation

Consumer RGB-D sensors leave large holes in their depth maps — on dark, reflective, or distant surfaces the sensor simply returns nothing. Those holes break the 2D-to-3D projection that OV-MAP relies on. OV-MAP renders a synthetic depth map from the reconstructed point cloud and uses it to fill only the missing pixels of the raw depth, leaving valid measurements untouched. Drag the slider to compare the raw sensor depth with the supplemented result on a real frame from scene0011_00.

Raw sensor depth Supplemented depth

On this frame the raw depth is missing ~37% of its pixels; supplementation brings that down to ~4%. Only the raw holes are replaced with the point-cloud-rendered synthetic depth, so real measurements are never overwritten.

Building the Map: Merging Across Views

OV-MAP never sees the whole room at once. It walks through the RGB-D stream one view at a time, lifts each view's class-agnostic 2D masks into 3D, and merges them: overlapping 3D masks from different views are fused into a single instance, and a dominant-vote step settles the rest — no 3D-supervised network anywhere. Press play to watch the per-instance map assemble, view by view.

Each point appears at the view where OV-MAP first observes it; colors are the final per-instance labels it converges to, and structural surfaces (wall / floor / ceiling) are drawn in gray. Sampled every 10th frame of the trajectory.

Presentation Video

IROS 2024 conference presentation — method overview and results walkthrough.

Results

3D instance segmentation on ScanNet200 (paper Table I).

Zero-shot generalization to the Replica dataset (paper Table II).

Real-World Validation

Beyond the ScanNet200/Replica benchmarks, OV-MAP was also validated on a real-world scan captured outside either dataset, then queried with open-vocabulary text prompts.

BibTeX

@inproceedings{kim2024ovmap,
  title={OV-MAP: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots},
  author={Kim, Juno and Park, Yesol and Yoon, Hye-Jung and Zhang, Byoung-Tak},
  booktitle={2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={13780--13786},
  year={2024},
  organization={IEEE}
}

Acknowledgements & License

This work builds on Segment Anything, Pointcept, SAM3D, CropFormer, and OpenMask3D. The code release is MIT-licensed (see the LICENSE file); vendored/dependent components retain their own licenses. This is a separate license from this website's own template/code.