DA-Fusion: Deformable Attention-Based RGB-D Fusion Transformer for Unseen Object Instance Segmentation

IEEE ICRA 2025

IEEE International Conference on Robotics and Automation (ICRA) 2025

Yesol Park1,* · Hye-Jung Yoon1,* · Juno Kim1,* · Byoung-Tak Zhang1,2,3

1Interdisciplinary Program in AI, Seoul National University  ·  2AI Institute, Seoul National University  ·  3Dept. of Computer Science, Seoul National University
*Equal contribution

arXiv Paper Code BibTeX
Unseen object segmentation comparison: RGB-only segmentation over-segments a textured object, depth-only segmentation under-segments two touching objects, while DA-Fusion's RGB+depth fusion matches the ground truth.

DA-Fusion is a deformable attention-based RGB-D fusion Transformer for unseen object instance segmentation — it dynamically fuses RGB and depth features at multiple layers to fix the over-segmentation of RGB-only and under-segmentation of depth-only methods in cluttered logistics scenes.

Abstract

In logistics automation, precise segmentation of unseen objects is crucial for efficient robotic manipulation in cluttered environments. Tasks such as bin-picking and shelf-picking require robust perception to handle occlusions, varying object shapes, and complex spatial arrangements. Traditional RGB-based methods tend to over-segment objects due to their reliance on texture, while depth-based methods often under-segment by focusing primarily on geometric features. To address these limitations, we propose DA-Fusion, a deformable attention-based RGB-D fusion Transformer designed for unseen object instance segmentation. DA-Fusion effectively combines the strengths of both RGB and depth data, enhancing segmentation accuracy in cluttered and multi-layered object environments. We also introduce the Object Clutter Bin Dataset (OCBD), a benchmark dataset specifically tailored for evaluating bin-picking scenarios in top-down views. Extensive evaluations demonstrate that DA-Fusion outperforms state-of-the-art methods across diverse environments, making it particularly suited for real-world logistics tasks.

Method

DA-Fusion architecture: parallel RGB and depth branches pass through four Swin Transformer layers, each fusing features via Deformable Self-Attention (DS) and Deformable Cross-Attention (DC), concatenated into f1-f4, and decoded by a Mask Transformer decoder into the final prediction.
Overview of the DA-Fusion architecture. Two parallel branches extract features from RGB and depth inputs, fused at multiple layers via deformable attention, then decoded by a Mask Transformer decoder into the final segmentation masks.

DA-Fusion processes RGB and depth through parallel 4-layer Swin Transformer branches, fusing them at every layer with two deformable-attention modules:

  1. Deformable Self-Attention (DS) — refines the RGB and depth feature maps independently. Reference points are sampled over each feature map, and a learned offset network predicts where to dynamically sample the most informative key/value positions, rather than attending uniformly.
  2. Deformable Cross-Attention (DC) — fuses the two modalities. The RGB query attends to depth-sampled keys/values (and vice versa), so each modality's query can pull in the complementary information it's missing — texture for depth, geometry for RGB.
  3. Multi-layer fusion — the DS+DC outputs at each of the 4 layers are concatenated into fused features f₁–f₄, capturing complementary RGB-D information at multiple scales.
  4. Mask Transformer decoding — the fused multi-scale features are passed to a pixel decoder and Transformer decoder (following Mask2Former's class-agnostic mask prediction), producing the final per-instance segmentation masks.

Object Clutter Bin Dataset (OCBD)

Existing bin-picking benchmarks are mostly simulated or feature simple, uniformly colored, single-layer objects. OCBD is a new real-world dataset the paper introduces to close that gap: 1,000 top-down RGB-D images (640×480, captured with an Azure Kinect DK) of 30 YCB objects and 25 HOPE household items, human-annotated with instance masks even in heavily cluttered, multi-layered piles — 13,147 object instances total, averaging 13.2 objects per image.

Six example OCBD images: top row shows real cluttered bin photographs, bottom row shows the corresponding human-annotated instance segmentation ground truth.
OCBD example images with their segmentation ground truth, illustrating the dataset's diversity and complexity.

Presentation Video

ICRA 2025 conference presentation — method overview, quantitative/qualitative results, and real-robot shelf-picking and bin-picking demonstrations.

Live Demo

TODO: an interactive demo (in the style of the OV-MAP viewer) is planned here, using real samples pulled from the server DA-Fusion was run on — not yet available.

Results

DA-Fusion achieves the highest overall scores on both OCID and OSD (Table below), and consistently outperforms prior RGB-D fusion methods on the new OCBD bin-picking benchmark, with the largest gains on the Overlap F-measure and %75 (fraction of objects segmented with Overlap F-measure ≥ 75%) metrics that matter most in cluttered scenes.

Dataset Input Overlap F Boundary F %75
OCID
(2390 images) 		RGB-D (best prior, UCN+) 91.6 86.1 89.3
Ours (DS+DC) 92.1 90.0 92.9
OSD
(111 images) 		RGB-D (best prior, UCN+) 87.4 69.4 83.2
Ours (DS+DC) 92.9 88.0 92.9
OCBD
(1000 images) 		RGB-D (best prior, UCN+) 70.7 46.7 61.3
Ours (DS+DC) 91.3 88.7 87.1

F-measures are the RGB-D-input row for each method (paper Tables I & II); %75 = fraction of objects segmented with Overlap F-measure ≥ 75%.

Qualitative comparison across OSD, OCID, and OCBD datasets: UCN+, UOAIS, and MSMFormer+ all show visible over- or under-segmentation errors, while DA-Fusion's predictions closely match the ground truth in every row.
Qualitative comparison across OSD (top), OCID (middle), and OCBD (bottom). DA-Fusion segments accumulated objects in OSD, delineates objects against feature-rich backgrounds in OCID, and detects occluded objects in lower bin layers in OCBD more accurately than prior methods.
Real-robot experiments: a UR5e arm with a RealSense D435 camera performing shelf-picking, and a UR5e with an overhead Azure Kinect camera performing bin-picking, each shown with DA-Fusion's segmentation result.
Real-robot experiments. Shelf-picking (left, front-view UR5e + RealSense D435) and bin-picking (right, top-down UR5e + Azure Kinect DK), with DA-Fusion's segmentation results.

BibTeX

@inproceedings{park2025dafusion,
  title={DA-Fusion: Deformable Attention-Based RGB-D Fusion Transformer for Unseen Object Instance Segmentation},
  author={Park, Yesol and Yoon, Hye-Jung and Kim, Juno and Zhang, Byoung-Tak},
  booktitle={2025 IEEE International Conference on Robotics and Automation (ICRA)},
  pages={7490--7496},
  year={2025},
  organization={IEEE}
}

Acknowledgements & License

This work was partly supported by the IITP (RS-2021-II212068-AIHub/10%, RS-2021-II211343-GSAI/15%, RS-2022-II220951-LBA/15%, RS-2022-II220953-PICA/20%), NRF (RS-2024-00353991-SPARC/20%), and KEIT (RS-2024-00423940/10%) grants funded by the Korean government. The code release is MIT-licensed (see the LICENSE file). This is a separate license from this website's own template/code.