Robotic Hand Grasping: a review focused on multiple objects manipulation

2.1.1 Hand design

Gripper design can be bioinspired, or physics based. The bioinspired designs include anthropomorphic hands [11], bird claws [12], lobster grippers [13] and tentacles [14]. Regarding anthropomorphic hands [4], their three main characteristics are: (a) presence of morphological features such as an opposing thumb, a palm, fingers, and phalanges; (b) similarity between the movement of the gripper during the grasping and the behavior of the human hand; and (c) the size of the gripper compared with that of the human hand. The development of these devices covers a century, starting with a prosthetic hook in 1912, while robotic hands as Robonaut, iCub, and RBO have been designed and built just in the last decades. Their design is focused on three aspects: joints, actuation, and transmission. Regarding hand joints, they are divided into rigid, flexible, dislocatable, and soft continuous. In a rigid joint, the links are connected by fixed mechanical elements. In a flexible joint, at least one flexible mechanical component is included. In a dislocatable joint, the flexible components allow to withstand some disarticulations. In a soft continuous joint, the finger is built using continuously flexible materials. Actuation systems can be rigid, with active impedance/admittance control, serial elastic, with explicit stiffness variation, agonist-antagonist variable-stiffness, or variable-impedance. Rigid actuation systems have negligible compliance, so they hold a specific position when external forces are applied on their output. Actuators with impedance/admittance control consider relevant compliance in their output, using fine tuning of control gains and/or the integration of an output force sensor. In serial elastic actuators, a spring is placed between a conventional actuator and an output mechanical component. Actuators with explicit stiffness variation are an evolution of the serial elastic actuators that allows to adjust the stiffness of the spring, by using a second actuation system. Actuators with agonist-antagonist variable-stiffness combine two prime motors, each motor is connected to the output shaft through a non-linear spring, and the position and stiffness of the output shaft is controlled by synchronous or opposite motions of the prime motors. Variable impedance actuators are another evolution of the serial elastic actuators, which combines springs and dampers that can be adjusted to change the mechanical impedance. Transmission systems are divided into fully actuated (number of actuators is equal to the DOF of the system), coupled (actuators number is higher than the DOF number), and underactuated (actuators number is lower than the DOF number). Anthropomorphic hands are applied in rehabilitation devices, industrial activities, and human-robot interaction [11].

Physics-based designs usually correspond to industrial grippers that can grasp the object by one of the following principles [2]: force closure (friction gripper), form closure (jaw gripper), magnetism, suction, indentation (needle gripper), electrostatics, Van der Waals forces, ice applying, acoustic levitation, optical pressure (Laser), Bernoulli, and adhesion. However, object dimensions limit the application of some grippers. For example, needle grippers only can be applied at macroscale, while adhesive grippers only can be applied at microscale.

Traditionally, fully actuated robots with only rigid components have been used in the industry, due to rigid components that allow higher precision of the tool positioning [15], so robotic hands also have been developed following this criterium. Additionally, the industry usually has a controlled environment, i.e., its conditions are known, so a gripper can only be operated using movement control [16]. However, when the environment is non-controlled, robot performance can be affected by external disturbances, therefore it is necessary to add force control strategies or flexible components in the robot. In line with the above, grippers with flexible components or soft hands have been proposed as alternatives. Regarding transmission systems in grippers, they should be as light as possible to allow them to save power in the manipulation actuators, which in turn allows them to pick up higher loads. This objective can also be achieved using a lower number of actuators, so the use of underactuated grippers has increased in the last years [11].

2.1.2 Grasping task implementation

Grasping task implementation can be divided into the following stages: object detection, grasping planning and control. It is important to mention that the above stages are not necessarily together in a grasping task and depend on the automatization level. For example, a teleoperated gripper usually only requires a monitoring step that corresponds to open-loop control [7]. Other example is the manipulation of unknown objects using tactile-based blind grasping [17].

Object detection is usually carried out by computer vision (Figure 2), which captures RGB and depth images, hence RGB-D cameras are mainly used [18]. Considering technological evolution, visual object detection approaches are divided in traditional and deep learning based [19]. Traditional object visual detection was developed from three important methods: VJ detector, Histogram of Oriented Gradients (HOG) detector, and deformable part-based model (DPM) [20]. Deep learning based object visual detection became a competitive option when a large Convolutional Neural Network (CNN) was trained to classify 1.2 million high-resolution images into 1,000 different classes, and obtained on its test data top-1 and top-5 error rates of 37.5 % and 17.0 % respectively, which were considered better that results recorded in the state of the art of that time [21]. CNN architecture allows to extract features from raw pixels to high level semantic information [22] with more robustness and expressive ability than the traditional detection methods [19]. Research about object detection based on deep learning usually divides its methods into two families: two-stages detectors and one-stage detectors. While two-stage detectors divide the process into proposal generation and region classification, one-stage detectors consider all positions on the image as potential objects and aim to classify each region of interest as either background or a target object [22].

Figure 2:

Visual detection of an apple using ASPOSE software. Original image source: www.wallpaperbetter.com/es/hd-wallpaper-zsonk.

Grasp planning can be divided into two steps: synthesis and selection. While the first step consists of generating poses (contact points and/or joints positions) that ensure feasible grasps, the second step consists in choosing the pose with the best grasp [23]. The above activities can be realized by analytical, or data driven approaches [24].

Analytical grasping planning approaches apply mathematical formulations that can include object model, gripper model, task constraints and environment constraints [5]. Considering that force closure is a relevant concept in analytical approaches, it is important to mention the theoretical framework proposed by Yoshikawa [25], who defined two typical constraints imposed by a constraining mechanism on an object: passive and active closure. In the first case, the current position and orientation of the object is maintained against an arbitrary external force applied to the object without changing the joints forces of the constraining mechanism. If passive closure requires preliminary constant contact forces generated by the actuators or the gravity, the condition is called passive force closure (Figure 3a). If these forces are not required, the condition is called passive form closure (Figure 3b). Regarding the active closure, it is present when arbitrary forces and torques can be exerted on the object through a set of contacts (Figure 4). The above condition is also called Active Force Closure (AFC) due to changing of the actuators forces is necessary.

Figure 3:

Passive closure. (a) Object subject to passive force closure. (b) Object subject to passive closure. (c) Contact forces in both cases if there is frictionless point contact.

Figure 4:

Active closure. (a) Object subject to active force closure. (b) Contact forces applied to the object if there is frictional point contact (red arrows indicate normal force components and green triangles indicate friction cones).

Two early grasp models were proposed by Nguyen. One model considered virtual springs at the fingertips, so a stiffness matrix was obtained, and the grasp stability was evaluated using a potential function [26]. The other one consisted in constructing independent contact regions for the fingertips such that the grasped object could present force closure. The model was applied to polygonal and polyhedral objects considering three cases: grasp with two soft finger contacts, grasp with three hard finger contacts, and grasp with seven frictionless point contacts [27]. On the other hand, an early grasping optimization approach was proposed by Li and Sastry [28], which consisted of three quality grasps measures: smallest singular value of grasp matrix, volume in wrench space, and a task-oriented quality measure that considered task ellipsoids in the wrench space. The smallest singular value of the grasp matrix indicates how far the grasp configuration is from falling into a singular configuration. Volume in wrench space considers all the singular values of the grasp matrix with the same weight, and often maximizes the volume of an ellipsoid in the wrench space that represents the global contribution of all the contact forces. Task oriented quality measure quantifies the ability of the grasp to counteract expected disturbances during task execution [29].

A quality measure, which became the most popular, is the radius of the largest ball (Figure 5). This measure corresponds to the largest perturbation wrench (vector of forces and torques in the center of mass of the object) that the grasp can resist in any direction [29]. The radius of the largest ball was proposed in two works. In one case, Kirkpatrick, Mishra and Yap [30] proposed a quantitative form of the Steinitz’s theorem that was applied to compute closure grasps by a multi-finger robotic hand. The other work was carried out by Ferrari and Canny [31], and consisted in solving the problem of planning optimal grasps for two-jaw and three-jaw grippers considering the wrench space, i.e., the space of vectors composed by forces and torques in the mass center of the object.

Figure 5:

Largest minimum resisted wrench (radius of the largest ball). W ̄ is the grasp wrench hull, while Q _lrw is the grasping quality measure. The wrench vector corresponds to forces f and torques τ in the mass center of the object.

A definition of grasping and manipulating forces for multi-finger robotic hands was proposed by Yoshikawa and Nagai [32]. The grasping force was defined as an internal force that satisfies the static friction constraint. The manipulating force was defined as a fingertip force that produces the specific resultant force, it is not in the inverse direction of the grasping force, and it is orthogonal to the grasping force component [32]. A study of the three-finger case was presented by Ponce et al. [33], who proved new sufficient conditions for equilibrium and force closure that are linear in the unknown grasp parameters. These conditions reduced the stable grasp regions in configuration space to constructing the three-dimensional projection of a five-dimensional polytope, allowing the use of linear optimization to compute maximal independent contact regions that ensure force closure. The study was extended to the four-finger case [34]. A recent work about analytical grasping planning approaches was developed by Qiu and Kermani [35], who considered the nonlinear friction cone model, and formulated the boundary of the grasp wrench space as a continuous function, allowing to analyze the grasp as a least-square problem.

Data-driven grasping planning approaches use database or known previous grasps to find feasible grasps in a specific case [36]. The database mainly includes the prior knowledge object (known, unknown or familiar) and/or the hand configuration (multi-finger or gripper), while the candidate grasps can be obtained directly of sensory data (heuristic techniques) or learning from human demonstrations, labeled examples, or trial and error (learning techniques). These approaches became popular in the 2000s due to the development of simulators such as GraspIt! [37] and Open Dynamics Engine [38], which provide a large database to train different algorithms. Additionally, the availability of high-quality cameras, depth sensors and powerful computational resources, allowed to relate perceptual features with grasp-success [18]. Regarding heuristic grasping planning approaches, Brook, Ciocarlie and Hsiao [39] proposed a framework that considered multiple objects representation such as 3D model and points cloud. The framework was implemented to evaluate the success probability of a candidate grasp, considering the sensed data uncertainty of a real object. In the same line, Weisz and Allen [40] evaluated the probability that a grasp reaches force closure under pose errors. The study demonstrated that the radius of the largest ball is a poor predictor of that probability. Regarding learning techniques, they can be divided into classical machine learning and deep learning. Classical-machine-learning-based grasping planning approaches are mainly based on Supervised Learning (SL) and Reinforcement Learning (RL). While approaches based on SL are trained with labeled data, RL approaches are trained by reward and penalty. An early study based on SL was carried out by Pelossof et al. [41] and consisted in taking advantage of GraspIt! to create a database that contained grasps examples and the quality of each grasp. The object model was built by composite super-quadrics, while the hand model was based on Barret’s design. The database was used to train and test a Support Vector Machine. Regarding RL, Wheeler, Fagg and Grupen [42] formulated the pick and place task as prospective behavior based on children, and proposed an intelligent agent that used Q-learning and ɛ-greedy exploration to gain experience during interaction with the environment. The agent was used to control a two-arm robot equipped with hands, vision system and tactile sensors. Regarding Deep-learning-based grasping planning approaches, they mainly implement CNN in a grasp detection architecture, but also can use other models of ANN. Those approaches appeared shortly after the success of Krizhevsky, Sutskever and Hinton [21] in object detection by deep learning (DL), and some works based on parallel-plate grippers were evaluated by 5-fold cross validation using the Cornell Grasp Dataset (CGD). At present, the highest accuracy in the CGD is 98.2 % (Figure 6) and was achieved by Ainetter and Fraundorfer [43], who proposed an end-to-end trainable CNN-based architecture that combines grasp detection and semantic segmentation in parallel to refine the candidate grasps prediction. The architecture was also tested in the Jacquard Grasping Dataset (JGD), obtaining an accuracy of 92.95 %.

Figure 6:

Improvement of grasping detection accuracy on CGD. The image was edited from Ref. [44].

Regarding grasping control, its approaches usually aim to ensure the grasp stability from the dynamic viewpoint unlike the planning case that focuses on a stable equilibrium. Therefore, the model-based approaches and the implementation of control strategies in virtual environments should consider a dynamic model of grasping. For example, Zhao et al. [45] developed a dynamic model considering a multi-finger robotic hand in contact with the target object as an entire system which is composed of multiple finger subsystems and an object subsystem. Then, the grasping forces were decoupled with the control torques of the finger joints and finally the manipulation control of the multi-finger hand was reformulated as a constraint-following problem. In the same line, a dynamic model for each finger of an anthropomorphic hand was implemented by Rojas et al. [46] to computationally validate a force/position controller with bounded actions and gravity compensation, considering elastic contact. Other study was developed by Liu et al. [47] to study the contact control problem of grasping a non-cooperative satellite by a space robot. From the study, a position-based impedance control strategy was proposed, considering a grasp model based on Hertzian contact.

Considering that developing a dynamic model can be a complex or unavailable process, some control approaches are mainly based on sensing information. Psomopulou et al. [48] designed a pinching controller for a gripper of two three-DOF robotic fingers with revolute joints and soft hemispherical tips. The controller only received finger proprioceptive measurements. It adjusted the relative finger orientation during the pinching of an arbitrary shape object by rolling soft fingertips, and allowed the transition from an initial contact state without equilibrium to a dynamically stable grasp. Additionally, the grasping force manipulability was improved for facilitating subsequent tasks. Regarding learning approaches, Vasquez et al. [49] presented an approach for soft grasping with objects of different shape, stiffness and size using a biologically inspired Spiking Neural Network (SNN) to control an anthropomorphic hand. An interesting work that included planning and control was developed by Pardi et al. [50] to improve the performance of a dual-arm robot. Considering that one arm had to support an object while the other arm had to apply a tool to the object, the strategy combined an optimization-based planning algorithm with an integral adaptive sliding mode controller to ensure robustness of the grasp under external forces transmitted by the tool.

This section ends with an example of task implementation (detection + planning + control) that corresponds to the work of Ge et al. [51], who proposed a framework based on a 3D detection network. The architecture was divided into three modules: image capture, 3D detection and robot control. The grasping task was carried out as follows. An RGB-D camera sent RGB and depth images to the image capture module. Then, the 3D detection module, which was based on YOLO, obtained the center point, 3D bounding box and classification label for each object. This output information was used to generate an initial grasping posture that was later optimized by key points adjustment. Finally, the robot control module used the hand-eye calibration to achieve the grasp of the target object.

3.3.1 First kind of AFC

In this case, the AFC focuses on one of the grasped objects (Figure 8). If a stationary condition is assumed, the first kind of AFC is expressed as follows [61]:

where w _B is the wrench vector, i.e., the forces and moments applied to the center of mass of the object.

Figure 8:

First kind of AFC.

3.3.2 Second kind of AFC

In this case, the AFC focuses on the center of mass of multiple objects (Figure 9). If a stationary condition is assumed, the second kind of AFC is expressed as follows [61]:

where G _mc is the grasp matrix that relates the center of mass one object with the mass center of the multiple objects system. p ̈ c m and α _cm are the translational and rotational acceleration of the center mass of the multiple objects, respectively.

Figure 9:

Second kind of AFC.

3.4.1 Multi-object diameter function

The multi-object diameter d(t), which is a function of time t, is defined as follows [68].

where L g t is the gripper opening length, b l t is the minimum distance between the left finger and the objects, and b r t is the minimum distance between the right finger and the objects. Figure 10 presents the initial conditions for multiple objects grasping, considering the diameter function. In this case, a success multiple objects grasping must satisfy [68]:

where d _min is the minimum possible final multi-object gasp diameter (Figure 11).

Figure 10:

Initial conditions of a frictionless multiple objects grasping that allow to implement the diameter function [68].

Figure 11:

Minimum final multi-object grasp diameter [68].

3.4.2 Intersection area

Given a rectangle S that corresponds to the gripper internal region (Figure 12), if A i t is the area of the intersection polygon for an object O _i during multiple objects grasping, a success grasp of m objects must satisfy [68]:

where t _f > 0 is the time when the gripper becomes stationary after closing. Considering that the intersection area changes as the gripper closes, if A i 0 = 0 , the distance between O _i and S must be shorter than a distance ɛ to ensure a success grasp.

Figure 12:

Conditions to ensure intersection areas [68]. The intersection polygons are drawn with purple lines.

If a value ɛ = 0 is assumed, the intersection area condition becomes [68]: