Seg2Grasp: A Robust Modular Suction Grasping in Bin Picking

Abstract

Current bin-picking methods that rely heavily on end-to-end learning often falter when confronted with unfamiliar or complex objects in unstructured environments. To overcome these limitations, we introduce Seg2Grasp, a modular pipeline designed for robust suction grasping in dynamic and cluttered bin scenarios. Seg2Grasp is built on a three-step process: Segmentation, Grasping, and Classification. The Segmentation module employs a Transformer-based model to generate class-agnostic object masks from RGB-D images, ensuring accurate detection across various conditions. The Grasping module uses surface normals and mask proposals to determine the optimal suction points, enhancing grasp success. Finally, the Classification module leverages open-vocabulary matching for precise object identification, enabling versatile handling of diverse objects. Real-world robotic experiments demonstrate that Seg2Grasp outperforms existing methods in success rates and adaptability, establishing it as a powerful tool for automated bin picking in industrial settings.

Note on the public code release: the published system used a fine-tuned Mask-CLIP classifier. The released code upgrades the classification module to a Qwen vision-language model (the segmented target and the full scene are shown to the VLM, which names the category) — simpler to reproduce and open-vocabulary. Segmentation and grasping follow the paper.

Method

Seg2Grasp is three specialized modules orchestrated by a single pipeline:

1. Segmentation — a Mask2Former (Swin-L) class-agnostic mask-proposal network fed the RGB image fused with an inverted, normalized depth image (its shared backbone runs on each and the outputs are averaged). Produces per-object instance masks.
2. Grasping — an analytic (non-learning) suction planner. It samples geometrically-uniform candidate points over the target's point cloud and scores each by a weighted criterion: surface angle (how top-facing the patch normal is), proximity to the object's center of gravity, and graspable-point count, returning the optimal suction point and camera-facing normal.
3. Classification — an open-vocabulary classifier. The public release uses a Qwen VLM: the full scene (for context) and the cropped target are shown to the model, which returns the object category.

Robot Demonstration

IROS 2024 presentation video — method overview followed by real UR5 bin-picking footage (from the original experiments; segmentation and classification have since been improved beyond what's shown here).

Sample Frames

24 real input frames captured across three physical bins — pick a tab below.

Hole Gray Bin Large Yellow Bin Small White Bin

The scene tabs above show real input frames from the repository's bundled demo/samples/ set, reflecting the current code release's segmentation/classification models.

Results

Real-robot picking success across difficulty levels (paper Table I). pr = pick success, or = object success, sr = segmentation success.

BibTeX

@inproceedings{yoon2024seg2grasp,
  title={Seg2Grasp: A Robust Modular Suction Grasping in Bin Picking},
  author={Yoon, Hye-Jung and Kim, Juno and Park, Yesol and Lee, Jun-Ki and Zhang, Byoung-Tak},
  booktitle={2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={2921--2927},
  year={2024},
  organization={IEEE}
}

Acknowledgements & License

This work builds on Mask2Former, detectron2, and Qwen. The code release is MIT-licensed (see the LICENSE file); vendored/dependent components (Mask2Former, detectron2, Qwen-VL) retain their own licenses. This is a separate license from this website's own template/code.