Scene-Q: Confidence-Aware Coarse-to-Fine Querying of 3D Scenes with Selective VLM Reasoning

Interactive Demo

This demo replays an actual Scene-Q run on ScanNet200 scene scene0011_00: class-agnostic 3D instances from OneFormer3D, scored with the SigLIP2 encoder used for routing. Selecting an instance below displays its temperature-scaled confidence distribution over all 198 candidate labels and indicates whether it was resolved on the fast path or escalated to Qwen2.5-VL, following the routing procedure in Fig. 1.

Semantic Instance Segmentation colors each of this scene's 95 instances by Scene-Q's predicted category (identical colors indicate identical labels); this is the segmentation against which Table I's mAP, AP50, and AP25 are computed, as distinct from the class-agnostic geometry shown above. The result is Scene-Q's own saved end-to-end output for this scene (sem_mask_array_CC_BB, SigLIP2 with selective Qwen2.5-VL routing), which we verified by independently re-deriving six of its instances (below) and obtaining identical labels. Of the 95 instances, 56 (59%) were escalated to the VLM, consistent with the paper's reported range of 39–62%.

The confidence bars use the per-instance SigLIP2 embeddings saved during evaluation, re-scored with the paper's fitted temperature (T = 0.01113…) and routing thresholds (p_max ≥ 0.50, margin ≥ 0.15, normalized entropy ≤ 0.35 unless margin ≥ 0.20), following Eq. 8. Escalated instances were reprocessed through Scene-Q's SAM + Qwen2.5-VL-7B reasoning step (identical multi-view crops, bounding boxes, and prompt template), computed once offline and replayed here; selecting an instance above shows the outcome for that case.

The Query row replays the paper's four natural-language retrieval types (Table III, Fig. 5: Category, Attribute, Spatial, Affordance) on this scene, since the paper's real-world retrieval set (11 novel maps, 1968 human-authored queries) was not available on the machine used to build this site. Selecting a query above shows how its target was identified.

Abstract

Indoor mobile robots require open-vocabulary scene understanding that grounds natural-language queries in a consistent 3D map. Many existing systems ultimately rely on cosine-similarity retrieval with contrastive image–text encoders, which is efficient but brittle when labels are near-synonymous or multiple similar instances appear. We present Scene-Q, a confidence-aware coarse-to-fine querying framework that normalizes encoder scores with temperature scaling and selectively invokes a reasoning VLM only for low-confidence cases. High-confidence queries are answered by fast retrieval, while ambiguous ones are reranked over a small top-K candidate set using the original multi-view images and instance bounding boxes, enabling context-aware disambiguation at low cost. Scene-Q improves open-vocabulary 3D instance segmentation on ScanNet200 and natural-language 3D instance retrieval on real-world reconstructions, with the largest gains on spatial and relational queries while keeping a substantial fraction of queries on the fast path.

Method

Scene-Q builds a queryable 3D instance map in three stages:

1. 3D instance segmentation via 2D-guided refinement — starting from class-agnostic 3D masks (Mask3D / OneFormer3D) and per-view 2D masks (SAM / CropFormer), a superpoint-based refinement step consolidates point assignments, detaches geometrically separated components with low multi-view 2D agreement, and iteratively merges adjacent instances that are consistent across views.
2. Multi-view instance descriptors — for each refined instance, the top-k most-visible views are cropped at multiple expansion levels, encoded with an image–text encoder (CLIP or SigLIP 2), and averaged into a single ℓ2-normalized instance embedding.
3. Confidence-aware selective VLM reasoning — cosine similarities against the label set are passed through a temperature-scaled softmax to obtain a routing confidence. High-confidence queries take the encoder's top-1 prediction directly (fast path, ~50 ms), while low-confidence queries send the top-K candidate labels together with the original multi-view images and instance bounding boxes to a reasoning VLM (Qwen2.5-VL), which picks the final label from full-image context (~2 s).

Presentation Video

IROS 2026 conference presentation — method overview and real-world novel-map query results.

Results

Open-vocabulary 3D instance segmentation on the ScanNet200 validation split (paper Table I).

Scene-Q's largest gain is under the relaxed AP25 metric (+5.4 over OpenYOLO3D), and it keeps a balanced head/common/tail profile (25.9 / 24.2 / 25.7 AP) where prior closed-vocabulary methods show a strong bias toward head categories.

Ablation — selective vs. always/never invoking the reasoning VLM, evaluated on ScanNet200 with oracle (GT) instance masks to isolate semantic accuracy from geometric mask quality (paper Table II).

Selective routing outperforms both never- and always-calling the VLM for both encoders — invoking the VLM on every instance can occasionally distract the model on cases the encoder already had right, so reasoning is most valuable precisely on the ambiguous subset (39–62% of instances per scene).

Real-World Validation

Beyond ScanNet200, Scene-Q was evaluated on previously unseen 3D maps captured in our lab with an Azure Kinect and reconstructed with RTAB-Map — using the exact same temperature scaling and routing thresholds fit on ScanNet200, with no retuning on the novel maps.

Natural-language 3D instance retrieval (Hit@1) on these real-world maps, using each method's own predicted instance masks — i.e. no ground-truth masks at inference time (paper Table III).

Gains are largest on the compositional query types — Spatial and Affordance — where crop-based encoder matching fails to leverage room context and inter-object relations. With ground-truth instance masks the same ordering holds and margins widen further (Scene-Q: 75.9 / 69.2 / 61.4 / 41.2 across the four query types).

BibTeX

@inproceedings{kim2026sceneq,
  title={Scene-Q: Confidence-Aware Coarse-to-Fine Querying of 3D Scenes with Selective VLM Reasoning},
  author={Kim, Juno and Park, Yesol and Yoon, Hye-Jung and Zhang, Byoung-Tak},
  booktitle={2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2026}
}

Acknowledgements & License

This work builds on Segment Anything, CropFormer / EntitySeg, Qwen2.5-VL, OpenMask3D, and Open3DIS. The code release is MIT-licensed (see the LICENSE file); vendored/dependent components retain their own licenses. This is a separate license from this website's own template/code.