Autonomous Microsurgical Needle Manipulation with Real-Time Stereo Keypoint Tracking using Neural Network
Accepted for publication in IEEE Transactions on Automation Science and Engineering on July 10, 2026. Xiaofeng Lin is a co-author of this study. DOI: 10.1109/TASE.2026.3711084
Overview
Reliable autonomous suturing requires a robot to localize and grasp small needles with sub-millimeter accuracy while the target may move or become partially occluded. This work integrates neural-network-based stereo perception, automatic camera-to-robot calibration, geometry-constrained grasp-point estimation, and closed-loop visual servoing for autonomous microsurgical needle manipulation.
System
The Neural Stereo Keypoint Tracker detects the needle tip, midpoint, endpoint, and forceps tips in synchronized stereo images, then triangulates their 3D positions. The tracker runs at approximately 55 FPS. A markerless calibration routine aligns the camera and robot coordinate frames using predefined robot landmarks. A five-state controller updates the grasp trajectory from 3D keypoints at approximately 50 Hz and includes recovery logic for temporary keypoint loss.
Experimental validation
- Mean 3D reconstruction error: 0.46 ± 0.90 mm across 100 images.
- Automatic calibration RMSE: 0.475 ± 0.047 mm across five camera placements.
- Closed-loop grasping: 72 successful trials out of 80, spanning two normal and two microsurgical needle types under static and dynamic conditions.
- Dynamic-condition baseline: closed-loop control achieved 36/40; the one-shot open-loop baseline achieved 0/40.
The same framework extracted a microsurgical needle from an artificial blood vessel in 7/10 trials and completed human-to-robot needle handover in 10/10 trials.
Scope and next steps
These results validate needle-handling subtasks on a controlled bench-top robotic platform; they do not establish clinical efficacy or complete autonomous suturing. Current limitations include frame-by-frame tracking without temporal motion priors, localization instability during occlusion, and the absence of force feedback. Future work will investigate temporal tracking, grasp-pose planning, force sensing, and multi-instrument coordination.
Publication
Y. Wang, X. Lin, S. A. Heredia Perez, and K. Harada, “Autonomous Microsurgical Needle Manipulation with Real-Time Stereo Keypoint Tracking using Neural Network,” IEEE Transactions on Automation Science and Engineering, accepted for publication, 2026. DOI