Optimization of target acquisition and sorting for object-finding multi-manipulator based on open MV vision

1 Introduction

With the continuous breakthrough and innovation in multi-rotor unmanned aerial vehicles (UAV) technology, with its advantages of flexible movement, simple operation, stability, and reliability, it has played an increasingly important role in agricultural plant protection, military security, meteorological monitoring, disaster relief, and other fields. In recent years, UAVs have been used in a wide range of civil applications, including urban planning, environmental monitoring, agriculture, governance, utilities, and commercial uses. UAVs are less expensive to acquire and operate than manned aircraft, which are utilized in traditional photography [1]. They also create photographs with comparable or superior spatial resolution. The disadvantage is that a single flight’s region coverage is limited. UAVs can (and often must) fly at extremely low altitudes (less than 200 m, mainly below clouds) and as frequently as resources (e.g., batteries) allow. UAVs can thus play an essential role in monitoring smallholder agricultural systems, where persistent cloud cover makes picture acquisition difficult for optical remote sensing systems [2,3].

UAV is mainly used by pilots to monitor the specified locations in the form of aerial photography, and it cannot actively interact with the surrounding environment, which has certain application limitations [4]. UAV – robotic arm structure, also known as flying robotic arm, has greatly expanded the application field of UAV. In addition, it can replace manual operations to reduce risks, improve working environment, and improve work efficiency [5]. The main research content is the autonomous navigation function of multi-rotor flight arm system based on open multiple-view (MV) visual dispatching system, and the target identification and grasping research based on navigation. The motion planning problem of humanoid robot arm grasping space object is a typical motion planning problem with multiple constraints in high-dimensional C-space, which is a challenging frontier research field. For the complex system of tightly coupled humanoid robot, the complex problem of motion planning in high-dimensional space involved by humanoid robot can be decomposed into a series of sub-problems in low-dimensional space, the constraint coordination scheme is utilized and the subproblem is solved incrementally [6]. Automatic sorting robots will be responsible for handling, sorting, packaging, and other tasks, among which, the sorting operation is to intelligently pick, sort, and mix a variety of goods according to the order requirements. Diverse environments on the production floor, shortage of experienced workers, functional correctness, and reliability are just a few of the issues that are preventing smart industrial robots from being used more widely. The automation and deployment of industrial robot systems replace repetitive, meaningless jobs that offer a significant danger of injury, as well as low-skill work that formerly needed human intellect. To be precise, the robot must be capable of making order decisions, handling movable objects, and interacting with information. At this time, robots require a high level of comprehensive perception, decision-making, and execution technologies, which necessitates the use of a variety of robot control technologies such as visual recognition, autonomous judgment, and motion planning [7].

Zhang et al. constructed a target grabbing system for robotic arm based on Kinect, and used the depth difference method to extract the contour of desktop object to complete the target detection. RRT algorithm is used to plan the trajectory of the manipulator to achieve the target grasping, but this method cannot be effectively applied to the non-uniform objects. How to carry out trajectory planning under spatial constraints is also the key to ensure the work efficiency of the sorting robot [8]. Candelieri et al. proposed a trajectory planning method of robot joint space under multiple constraints based on trapezoidal velocity curve, it can give full play to the driving performance of the robot joint while making the movement as smooth as possible [9]. Yin et al. adopted the energy storage systems method to reduce the selection error of feature points, and optimized the binocular feature matching algorithm points with the improved RANSAC algorithm, the effectiveness of the method is proved by the indoor and outdoor robot self-positioning comparison experiments [10]. UAVs can be controlled remotely by a pilot at a ground control station or autonomously by following a pre-programmed flight plan. The history of these systems may be traced back to hobby and professional radio-controlled aircraft, as well as military applications, such as flying over hostile or otherwise dangerous area. UAVs are planes that do not have a pilot on board. UAVs are gaining popularity among scientists as innovative instruments for remote sensing [11]. Due to this unique properties of data that it can acquire, the UAV is able to fills the unfilled niche when compared to more traditional aircraft or satellite-based platforms. Because of its low operating altitude, it can generate data with ultra-high spatial resolution across relatively tiny geographical areas [12]. For topographic modeling and feature recognition, Zhao et al. [13] and Lin et al. [14] employed UAV-based LiDAR. Stefanik et al. [15] employed UAVs for stereo-image 3D landscape modeling. According to Rudol and Doherty [16] and Hinkley and Zajkowski [17], UAVs are assisting in the fast replacement of traditional aerial photography in the field of remote sensing by producing high spatial resolution aerial photographs of the earth’s surface. UAV images may be used for a variety of purposes, including large-scale mapping, urban modeling, and flora structure mapping. UAVs can conduct effective surveys in disaster-prone or physically unreachable places, as well as can quickly assess the damage caused by landslides, floods, and earthquakes to enable relief efforts.

On the basis of the above planning technology, this study proposes research on open MV visual scheduling, hardware system, software system, and overall structure layout and function of the sorting robot by studying the operating requirements of the unmanned storage application scenario. The sorting motion path of the six degree of freedom (DOF) manipulator in the storage scenario is developed based on the operating requirements of goods sorting, the features of the grab surface of warehousing items are examined, and two end grab strategies, bent suction and side suction, are proposed.

2 Grasping and sorting by mechanical arm

Based on the D–H convention, the reference coordinate system, linkage coordinate system, and workspace of the right arm of the humanoid robot with seven DOF are set to q ₁ = [q ₁, …, q ₇] ^T , sin q _i (s _qi ) and cos q _i (c _qi ) are denoted as s _i and c _i , respectively, then the homogeneous matrix ⁱ⁻¹ T(q _i ) between coordinate system i and i − 1 can be expressed as Eq. (1):

The forward kinematics equation of the humanoid robot’s right arm can be described by the homogeneous coordinate transformation matrix of the position and pose of the end effector Σ₇ of the right arm relative to the reference coordinate system Σ₀ Eq. (2):

The right arm of the robot described in Eq. (2) has redundancy, and can be described by rotation around the center of the elbow through the wrist and the Σ₀ origin axis. Once the position of the elbow is determined, an analytical form of the inverse kinematics of the arm can be derived. According to the elbow circle of the right arm, the position ⁰ T4 (Eq. (3)) of the elbow joint of the right arm can be deduced and calculated.

The position of the end effector of the right arm relative to the right elbow joint is (4):

Thus, the 7-DOF inverse kinematics problem of the right arm is decomposed into Eqs. (3) and (4), which represent two sub-inverse kinematics problems with smaller dimensions. By multiplying both sides of Eq. (3) by T 1 − 1 o , Eq. (5) can be obtained:

The equation formed by matrices (3,4) and other elements on both sides of Eq. (5) can be obtained:

Multiply both sides of Eq. (5) by the inverse matrix T 1 − 1 2 ( q 2 ) again to get:

By comparing the matrix elements on the left and right sides of Eq. (8), q ₃ and q ₄ can be obtained:

According to Eq. (4):

The equation formed by the corresponding elements of the matrices on both sides of Eq. (11) can be solved as follows:

2.1 Based on open MV visual scheduling research

The open MV is a low power consumption, low volume, and low price machine vision module based on the STM32H7, OV7725 camera module, and the FLIR Lepton infrared thermal imaging module. The open MV on-board processor runs up to 400 MHz, and has a faster graphics processing algorithm, whose RAM is up to 1 MB, which enables it to support large model matching pictures, lenet digital recognition support, and running large neural network models. At the same time, in the small hardware module, the core machine vision algorithm is efficiently implemented in C language, equipped with Micro Python interpreter, which can realize the access and control of the bottom layer of the hardware through Python scripting language. Machine vision algorithms on the open MV include looking for color blocks, eye tracking, QR code recognition, face detection, edge detection, logo tracking, etc. STM32H7 by open MV has rich hardware resources and interfaces such as UART, I2C, SPI, PWM, ADC, DAC, and GPIO to expand peripheral capabilities. The μ secure digital (SD) card slot having 100 Mbps reading and writing capability uses the open MV camera to take photos and record video, and the machine vision material can be extracted from the SD card. Open MV connects the integrated development environment on the computer via universal serial bus interface. Integrated development environment helps with programming, debugging, and updating firmware. Robot control technology with visual awareness is a difficult issue. In the space and material of a collaborative robot arm, follow the search scenario of this theme. The positioning capability of the 3D coordinate of the depth camera and the binocular camera is more suitable for the requirements of the scene; however, during the automatic classification process, the autonomy of the camera is significantly limited by space, for example, when using eye-to-eye method to penetrate the target of the lower shelf, the mechanical arm can only try to avoid noise with the shelf captured by the party. Greatly limiting the working space of the manipulator, therefore, visual positioning system uses eye-to-hand method. The binocular structure was combined with the grabbing strategy on the fixed side of the sorting robot platform to achieve the collision free multi-target positioning and grabbing experiment [18].

In open MV machine vision applications, the relationship between a three-dimensional space point and its presence in a two-dimensional image is closely related to the camera imaging model, Figure 1. In the relationship between pixels coordinate system and the physical coordinate system of the imaging plane, the coordinates in the pixel coordinate system (O _uv ) are (u, v), the origin O ₀ of (O _uv ) is generally selected at the upper left corner of the image; however, (u, v) does not represent the physical location of the pixel, instead, the physical position of the pixel is described in the imaging plane physical coordinate system (O _xy ), the origin of (O _xy ) is O ₁, the intersection point of the camera optical axis and the image plane, its X-axis and Y-axis are parallel to the U-axis and V-axis of the pixel coordinate system, respectively [19]. Set the coordinate of O ₁ as (u ₀, v ₀) in the pixel coordinate system, assuming that the actual physical dimensions of pixels on the imaging plane in the X and Y directions are dx and dy, then the corresponding relationship between pixel coordinates and imaging physical coordinates is:

(13) u = x d x + u 0 v = y d y + v 0 .

Figure 1

Pixel coordinate system and image coordinate system.

Expressed in the form of homogeneous coordinate matrix:

(14) x y 1 = d x 0 − u 0 d x 0 d y − v 0 d y 0 0 1 .

As shown in Figure 2, the coordinate p(x _c, y _c, z _c) of the object in the camera is described by the camera coordinate system O _c, the origin of P is generally the optical center of the camera [20], and the z axis of the coordinate system is the optical axis of the camera. When the binocular camera observed a coordinate point p(x _c, y _c, z _c) of barium O ₁, coordinate point p(x _c, y _c, z _c) is represented differently in the two camera coordinates as shown in Figure 2. Figure 2 is the mapping between camera coordinate system O _c and imaging plane physical coordinate system (O _xy ), the relationship between the actual position of object O _c and the question of (O _c) is:

(15) x = f X C Z C , y = f Y c Z c .

O _w represents the world coordinate system, and the homogeneous transformation matrix can be used between it and O _c. T w c represents the transformation of coordinates, assuming that the object’s world coordinate is p _w, the homogeneous coordinate is [X _W, Y _W, Z _W, 1] ^T , the camera coordinate system is [X _C, Y _C, Z _C] ^T , then the relationship between the two coordinate systems can be expressed as follows:

(16) X C Y C Z C 1 = X W Y W Z W 1 = R 3 × 3 t 3 × 1 0 1 × 3 1 ,

where R represents the rotation transformation matrix, t represents the translation transformation vector, and then, according to Eqs. (13)–(16), the relationship between pixel coordinates (u, v) and p _w can be obtained as follows:

(17) Z C u v 1 = 1 / d x 0 u 0 0 1 / d x v 0 0 0 1 f 0 0 0 f 0 0 1 0 R 3 × 3 t 3 × 1 0 1 × 3 1 .

Figure 2

Camera imaging model.

Formula (17) represents the position mapping relationship between the points in O _c and the world coordinate system O _w.

Open MV (machine vision) system is the process of object spatial position perception, due to the influence of illumination change, external noise, perspective distortion, and other factors, the processing of two-dimensional images inevitably involves the problem of visual maladaptation, that is, the characteristics of the corresponding points projected from the spatial positioning points of the target goods to the physical imaging plane of the binocular camera are different [21,22]. In the storage environment, the binocular visual recognition system of the sorting robot is faced with the unknown category and location of goods. For a feature point or piece of image located in an image, there may be multiple similar candidate matches in another image, so other information or constraints are required for further judgment. In order to determine a unique match, the matching accuracy of the object capture information between images will determine the final object spatial positioning accuracy [23,24]. The spatial representation of grasping target is reconstructed based on the parallax principle, which provides a reliable basis for the space motion planning of manipulator arm and the precise grasping operation of actuator. According to the information points of the target image, the matching strategy of polar line geometry is adopted to complete the three-dimensional matching of the capture information, which effectively guarantees the uniqueness of the capture point of the object, so as to realize the accuracy of the spatial positioning of the object.

3 Experimental analysis

Unmanned storage scenario has narrow aisles, many shelves, and dense goods. In the process of dynamic operation, get open MV 3D location information with the visual scheduling system. According to the position information of the goods, the manipulator can obtain the end attitude through the inverse solution of the analytical method, then plan specific subtasks or sequences of actions. Given the starting points, intermediate points, end points, and some necessary constraints, plan and describe the trajectory of the manipulator. The flexibility of the end actuator is used to adjust the grasping strategy to avoid collision between the manipulator and surrounding obstacles, stable and fast cargo grabbing can be realized. The control system diagram of the sorting robot is shown in Figure 3.

Figure 3

Control system diagram of sorting robot.

Select products with typical shapes in the unmanned storage scene, as shown in Figure 3, these goods are in lying or are in vertical state. This scheme simulates the human perspective and grasping mode, select the surface closest to the outside of the shelf when the goods are placed naturally at the grasping surface. The combination of steering gear and flexible sucker cleverly adds a 180° to the end of the manipulator arm. DOF and the suction angle can be set freely. For different shapes of goods, the manipulator end actuator adopts two grasping strategies, the first grasping strategy is called the forward suction position, and the second grasping strategy is called the side suction position. The increase in the end actuator extends the motion range of the manipulator arm, and the end of the manipulator arm can keep the end grasping state further deep into the shelf, the operation comfort of the manipulator and the flexibility of grasping goods are improved. Under the condition that the end posture of the manipulator remains the same, the first approach extends the extension length of the manipulator by 16 cm, the second side suction attitude extends the extension length of the manipulator by 19 cm. The increase in the end actuator extends the motion range of the manipulator arm, improves the operation comfort of the manipulator arm and the flexibility of grasping goods. The end actuator’s flexibility is employed to alter the grasping strategies to minimize collisions between both the manipulator and surrounding obstructions, allowing for steady and quick cargo grabbing. The manipulator arm’s end is skillfully turned 180° thanks to the combination of steering gear and flexible sucker. The suction angle may be flexibly set due to the degree of freedom. The first approach extends under the constraint that the manipulator’s ultimate position stays the same.

4 Conclusion

A robotic system that can identify and locate multiple targets in an unmanned storage environment and has the ability to execute, is developed, according to the characteristics of various targets and environments in the aerial storage scene. Multi-target identification and positioning classification robot was designed and verified by simulation and practice. Simulation and construction of a gallery-type unmanned storage scenario, analysis, and design of a sorting robot control module were performed to solve problems in the storage environment, such as narrow channels between shelves, the small distance between the goods layers of the shelves, the variety of goods and irregular shapes, and the camera image model was studied, the open MV vision planning structure was used for positioning multi-objective goods, based on the results of an eye-corrected and open MV visual planning test. The 3 D coordinate accuracy of the open MV visual planning system is proven to be 8%. When the target merchandise order distance is greater than 5 cm, the open MV match success rate is 89.7%. By increasing the offset of the uneven target goods, the sorting and picking success rate is increased from 6 to 85%, so that the end of the mechanical arm can reach the position of the cargo easily, conveniently and quickly.