Accepted at IEEE/ASME AIM 2026

Monocular Vision Based Control Framework for Grasping

2026 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)  ·  Genova, Italy  ·  July 7-10, 2026

One RGB camera. One standard gripper. A single control pipeline that handles both soft and rigid objects, with no tactile sensors and no specialized hardware.

Shail Jadav1 Dongheui Lee1,2
1Autonomous Systems, TU Wien, Vienna, Austria    2Institute of Robotics and Mechatronics, DLR, Germany
Supported by the WWTF (SafeDiffusion, ICT25068) and the EU project INVERSE (No. 101136067).
arXiv Paper Video BibTeX

The core idea. Our method reads deformation directly from the camera and works well on soft objects (left), whereas vision-based tactile sensors show very limited sensitivity on soft materials (right).

RGB-only
No tactile · no RGB-D
30 FPS
Real-time control loop
Soft + Rigid
One unified pipeline
Standard
Position-controlled gripper

Abstract

Humans grasp anything, a soft tomato or a rigid bottle, with the same hand and a glance. We give a standard gripper that ability, using vision alone.

Grasping in unstructured environments requires handling objects with widely different mechanical properties, from soft and deformable items to rigid everyday objects. Most existing approaches address these categories separately and often rely on tactile sensing, object-specific models, or specialized grippers. We present a unified monocular vision-based grasping framework that targets both soft and rigid objects within a single control pipeline, using only RGB input and a position-controlled gripper.

The system combines open-vocabulary object detection, image segmentation, boundary-aware point assignment, real-time point tracking, and monocular depth estimation to recover object motion and geometry from vision. A key component is a language-based stiffness estimation model (StiffNET) that infers an object's expected compliance from its semantic description, providing an object-level prior for selecting the grasping strategy before contact. For deformable objects, adaptation is governed by a Procrustes-based dissimilarity measure from tracked keypoints; for rigid objects, gripper width is regulated through the scaling of tracked point distances. We validate the method with real-world pick-and-place on a Franka Emika Research 3 across lettuce, fresh mozzarella cheese, croissants, paper towels, and hard plastic bottles.

Grasping Across Mechanical Diversity

Robots must handle objects spanning a vast stiffness range, yet no single system handles both. Prior approaches each trade away generality, whether through hardware, sensing, or model assumptions.

The challenge: tactile sensors, specialized grippers, and physics models each have limitations
Three approaches, three limitations. Tactile sensors are expensive and lose sensitivity on very soft objects; specialized grippers cannot exert forces for rigid objects; physics models require precise material parameters that are hard to obtain in practice.

Framework Overview

A single RGB pipeline drives two grasping modes. A language prior selects the strategy before contact; visual feedback adapts the gripper width during the grasp.

Framework overview: monocular perception, language-driven material prior, geometric state estimation, control mode selection, adaptive gripper width control
Overview. Monocular perception and a language-driven stiffness prior feed a control-mode selector. Deformable objects use a deformation metric; rigid objects use a scaling factor, both regulating gripper width in real time.
  • Unified RGB-only grasping. One monocular pipeline lets a standard position-controlled gripper handle both compliant and rigid objects, without tactile sensing, RGB-D, mechanics models, or specialized grippers.
  • Language as a pre-contact prior. Semantic knowledge from language yields an object-level compliance prior, so the system picks the right behavior before touching the object.
  • Vision-driven adaptation. Visual feedback alone supports adaptive, stable grasping across everyday objects: a practical, sensor-efficient path for food handling and household manipulation.
Soft → track a target deformation Rigid → regulate the scaling factor

Perception Pipeline

A text prompt and the camera feed drive detection and segmentation; boundary-aware points are tracked and lifted to 3D, then compared against the initial state to estimate deformation.

Perception pipeline: object detection, segmentation, point assignment, tracking, depth estimation, Procrustes analysis, gripper control
From pixels to grasp. YOLO-World detection → SAM 2 segmentation → weighted k-means point assignment → TAPIR tracking → Depth Anything V2 → Procrustes analysis (deformation metric + scaling factor) → gripper-width control, informed by StiffNET.

StiffNET: Reading Stiffness from Language

Before the gripper touches the object, StiffNET turns a text description into a physical stiffness estimate, deciding how to grasp from semantics alone.

StiffNET architecture with CLIP text encoder and Young's modulus comparison table
Learned log-stiffness axis. A frozen CLIP text encoder plus a small network maps an object name to a physical stiffness, trained jointly on pairwise ordering and sparse measured values. StiffNET matches GPT-5.3 on anchored objects and generalizes to unseen ones.
LLM (Gemma 3) provides pairwise hardness comparisons between everyday objects
Supervision from language. An LLM (Gemma 3) generates pairwise hardness comparisons between everyday objects (e.g. "which is harder, bread or a diamond?"), giving relative ordering at scale without manual annotation.

Supplementary Video

Full walkthrough of the method and real-world grasping experiments across soft and rigid objects.

Experiments

Hardware platform and objects tested
Setup & objects. A Franka Emika Research 3 with a Franka Hand, 3D-printed PLA fingers, and a RAZER Kiyo-X webcam (RGB only) running at 30 FPS. Objects span the deformable-to-rigid spectrum: lettuce, mozzarella, croissant, paper towel, and a hard plastic bottle.
Grasping lettuce, mozzarella, croissant, and paper roll
Grasping multiple objects. Pick-and-place for deformable objects: lettuce, mozzarella cheese, croissant bread, and a paper roll. The controller adjusts finger width to track the target dissimilarity D (deformation vs. the initial condition) and hold a stable grasp.
Grasping lettuce over time
Grasping lettuce. As the robot lifts, the scaling factor drops while the object is not yet secured, raising the reference dissimilarity and closing the gripper. Once grasped, all signals stabilize.
Grasping a rigid plastic bottle
Grasping a rigid bottle. With minimal deformation, control relies on the scaling factor (SF): as the arm ascends, SF falls and the controller narrows the fingers until SF and width stabilize.

StiffNET vs. a Large Language Model

Young's modulus estimates (GPa). StiffNET recovers physically meaningful stiffness from language alone, used here only to select the control mode.

ObjectTrue (GPa)GPT-5.3StiffNET
Anchored (ground truth available)
Tofu0.00010.00010.0001002
Wooden Block10109.092
Steel200200210.3
PVC (Polyvinyl Chloride)2.52.52.568
Generalization (no ground truth)
Cucumbern/a0.0020.03191
Walnutn/a618.18
Scissorsn/a200157
Power Bankn/a35.451
Plastic Bottlen/a1.52.1

BibTeX

@inproceedings{jadav2026monocular,
  title     = {Monocular Vision Based Control Framework for Grasping},
  author    = {Jadav, Shail and Lee, Dongheui},
  booktitle = {2026 IEEE/ASME International Conference on Advanced
               Intelligent Mechatronics (AIM)},
  address   = {Genova, Italy},
  year      = {2026}
}