晓锋's Website

A Robotic Platform for Micromanipulation

(Cover Image: Prototype of AI Robot Platform Developed by Our Research Group – Harada Lab)

The development of a robotic platform and micro-robotic tools aims to enable more accurate, precise, and flexible operations than those performed by human scientists, allowing for scientific experiments that cannot be completed by human scientists alone.

In life sciences, experiments often require highly accurate, precise, and complex manipulations under a microscope. However, even skilled scientists frequently encounter difficulties due to the physical limitations of the human body. Moreover, while current robotic technology excels in performing routine tasks involving rigid objects, it struggles with unconventional tasks that involve small and flexible specimens, such as plant and animal samples.

In this project, development progresses in two phases. First, through remote control of the robots or automated operations, our goal is to surpass the accuracy, precision, and flexibility of skilled scientists in handling samples during life sciences experiments. Next, we aim to integrate AI with the robotic system to enable fully automated experimentation. Currently, we are developing a prototype platform and micro-robotic tools for integration with AI.

As a platform for autonomous scientific exploration, we have developed the AI robot platform shown in Figure 1. The platform consists of four robotic arms, each with distinct functionalities, positioned around the experimental object under the microscope. These arms can be equipped with various experimental tools, and depending on the requirements, the AI-equipped robotic arms can coordinate and execute tasks with the necessary tools. We have implemented a control system that automatically prevents collisions between the robotic arms and between the arms and the surrounding environment, allowing precise operations in the limited field of view of the microscope. We have successfully demonstrated the platform’s remote control capability, performing high-precision and complex experimental manipulations on real samples. This remote control will be used to collect training data for the AI’s skill development.

Figure 1: Components of the Micromanipulation Robotic System

This robot is currently capable of performing surgical procedures on small mammals, such as mice. It allows independent remote control of the four robotic arms using up to two operator consoles and four haptic-feedback master controllers. The robotic platform has successfully demonstrated ultra-long-distance remote operations via Internet, as shown in the video shared below. With a latency of less than 100 ms, even beginners can quickly become proficient in its operation.

Video 1: Robot-Assisted Craniotomy Surgery on a Mouse Using Master-Slave Control
Video 2: Ultra-Remote Operation via 5G (Kyoto, Japan – Tokyo, Japan), Demonstrated at IROS 2022

The system also provides a simulation platform with kinematic modeling and a digital twin, as illustrated in the following slide.

Digital Twin is a virtual modeling technology that creates an accurate replica of a physical object or system in a digital environment. By collecting and analyzing data from sensors, IoT devices, and other sources, a digital twin can reflect the real-time state and behavior of its physical counterpart.

In simple terms, digital twin technology allows observation and simulation of the operation of real-world physical objects in a virtual space, enabling prediction, optimization, maintenance, and decision-making. For example, in manufacturing, digital twins help engineers monitor the status of production equipment, predict failures, and optimize processes. In urban management, they can be used for traffic management, energy distribution, and other areas to enhance operational efficiency.

The robotic system includes kinematic simulation based on CoppeliaSim, which prevents collisions between the robotic arms and the hardware environment at the software and algorithm level. The digital twin environment based on Nvidia Isaac Sim provides high-quality datasets for the robot’s reinforcement learning training and simulations, significantly improving the efficiency of experiments and scientific discovery.

Future work will focus on further validating and demonstrating the system’s advanced capabilities and robustness through a broader range of scientific and animal experiments.

References: