Programming and control for skill-based robots

Janne Saukkoriipi; Tapio Heikkilä; Jari M. Ahola; Tuomas Seppälä; Pekka Isto

doi:10.1515/eng-2020-0037

Article Open Access

Programming and control for skill-based robots

Janne Saukkoriipi , Tapio Heikkilä , Jari M. Ahola , Tuomas Seppälä and Pekka Isto

Published/Copyright: May 9, 2020

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information

From the journal Open Engineering Volume 10 Issue 1

Abstract

This paper considers programming and control of skill-based robots. Robot skills are used to integrate and synchronize robot actions and sensor data in a consistent way. Skill-based approach provides a framework for configurable robot systems, enabling quick setups and start-ups of applications. In the paper we will introduce skill programming and skill control concepts in more detail and how they relate to usage of models and sensors. We will also give a practical example for programming and implementing skills for a grinding application.

Keywords: robot skill; programming; sensor; control

1 Introduction

With ever-increasing demands for agility and flexibility, robotized manufacturing is facing challenges and barriers to the use of robots because; the high cost of engineering the work cells (i.e., the design, fabrication, and installation of jigs, fixtures, conveyors, and third-party sensors and software). Robots must be able to perform their tasks in environments with greater uncertainty than current systems can tolerate [1],where the robustness implies wide usage of a variety of sensor technologies. Robust robot operations can be based on behavioral models supporting the integration of sensors into robot operations - this is how the robot “skills” are formulated. A skill is typically defined as a high level entity as a sensory-motoric decomposition pattern, where skills are decomposed to action and trajectory level representations [2]. Skill representations support reusability and when the skill layer is abstract it provides a non-expert and intuitive programming interface, but still relies on a device level setup by an expert user [3]. Reusability is two-fold, it can be supported by simpler, or conversely by more complex skills [4]:

simpler skills are easier to re-use and may be related to generic description on a symbol level, but
hardware independence can only be achieved to a certain degree, but is more difficult if skills are simple.

Skills can be hierarchic and at the lowest level there are atomic actions representing device interfaces or API functions. Hierarchic structure in skill-based control and programming will go as follows: a “task” is described as the highest-level of operation like for example “Single part feeding”, “Welding with localization” and “Pick-up with contact-force” [5]. A task can be decomposed to “skills” such as “Locate box”, “Pick-up box” and “Place box” [5, 6, 7], which are formed from the lowest-level entities called “primitives”, which can be described as “device operations”. Further, the “primitive” layer can be seen to represent as the conventional robot work sequence, consisting of commands such as “Move linear”, “Open gripper”, “Get point-cloud” and “Get force-values”. Skills should be designed in a way, where parametrization supports an easy and quick re-configurability of the skills for different situations in varying applications, for example phase dependent locations/points, poses, forces and so on.

To our knowledge, there are no off-line programming tools for skill based robot systems, where both robot path programming - especially with compliant motion control primitives - and sensor programming were integrated. Schou et al. [8] presents a similar hierarchical functional modelling and parametrization approach to task-level programming. But in their system, only some parameters were specified off-line (e.g. velocity, object type etc.) and all the locations were taught on-line. Whereas in our system, all skills are fully programmed off-line (with only updating the specific parameters based on information gained from the other skills. In their paper, Schou et al. also differentiate computational procedures from skills to “services”, which is a challenging generalization since almost every task, skill and primitive include some sort of computational procedures which can be implemented as SW services.

In addition, as we have been developing robot skills especially relying on vision feedback and compliant contact motions, and for that there are only few, if any, reported references in the literature. Zeiss has reported on modeling principles [9], where force and torque have been interconnected in the task representation with the a pose-wrench approach (wrench=force /torque parameters) but it considers only static path points and not compliance control.

In this paper we will introduce methods and tools for robot skill programming and an example of skill based control system. Skill execution control - or dynamics - is specified as UML action diagrams, the implementation of which may be distributed and implemented as robot programs or PLC programs. Skills are specified with (configuration) parameters, and our integrated tool supports setting and computing these, using a robot off-line programming tool with our extensions. In the experiments we will show examples of skills that rely heavily on off-line programming methods.

2 Skill models and implementations

Ideally, skill models are designed and programmed by expert users but they could be re-configured for different situations even by operators who have no experience in robot programming. This is achieved by making skills intuitive object-centered robot abilities, where they are applied on physical entities instead of 3D coordinates [5]. In practice, this can be accomplished with a proper parametrization of programs, where a set of inputs will define the whole structure of a program. The program would automatically give all the necessary parameters for the sensors and control systems in a system. Robot skills are divided to primitive skills and compound skills, composed of primitive skills and other compound skills. These are described further below.

2.1 Primitive skills

Feature based localization

Fundamentally, rough global localization of the target object is needed to produce the initial pose parameters for the fine localization. Rough location of the object can be determined based on known features of the object e.g. eigenvectors and center points of the 3D point clusters, or other known vectors, like surface normals or interest points, like corners. The non-orthogonal estimate of the rotation matrix Re can be computed as

(1)Re=N2N1+

in which N₁ and N₂ are matrices including surface normal vectors and have more columns than rows. N1+denotes pseudoinverse which can be computed as

(2)N1+=N1TN1N1T−1

required that matrix N1N1Tis invertible. The matrix N₁ include vectors in the object’s CAD model coordinate system

(3)N1=n¯1…n¯i

and the matrix N₂ include vectors in the sensor’s coordinate system

(4)N2=n¯m1…n¯mi

The orthogonal estimate of the rotation matrix is constructed from Re by taking cross products of the columns of the rotation matrix estimate

(5)v¯z=v¯x×v¯y

in which v_x = Re_1:3,1 and v_y = Re_1:3,2.

(6)v¯x=v¯y×v¯z

The initial, i.e. rough, rotation matrix is constructed from the orthogonal unit vectors.

(7)Rinit=v¯xv¯xv¯yv¯yv¯zv¯z

The initial translational vector T_init is computed as

(8)T¯init=p¯c2−Rinitp¯c1

in which p_c1 is the corresponding center point in the object’s CAD coordinate system and p_c2 is the cluster center point in the sensor’s coordinate system.

Verification vision

In the fine localization, the target object’s pose is estimated by fitting surface models to the measured 3D-points. Here the equations for fitting planar surfaces are presented and equations for fitting other surface types have been presented in [14]. The rotation correction ᐃR is computed as a product of rotation matrices

(9)ΔR=Rot(x,Δϕx)Rot(y,Δϕy)Rot(z,Δϕz)

in which

(10)Rot(x,Δϕx)=1000cos⁡(Δϕx)−sin⁡(Δϕx)0sin⁡(Δϕx)cos⁡(Δϕx)

(11)Rot(y,Δϕy)=cos⁡(Δϕy)0sin⁡(Δϕy)010−sin⁡(Δϕy)0cos⁡(Δϕy)

(12)Rot(z,Δϕy)=cos⁡(Δϕz)−sin⁡(Δϕz)0sin⁡(Δϕz)cos⁡(Δϕz)0001

where ᐃφ_x, ᐃφ_y, and ᐃφ_z are the rotation corrections with respect to the coordinate axes.

The translational correction ᐃT is defined by three translational components

(13)ΔT=ΔtxΔtyΔtz

The correction vector ᐃ is defined as

(14)Δ=ΔϕxΔϕyΔϕzΔtxΔtyΔtz

The correction vector is calculated

(15)Δ=−(JTJ)−1JT⋅E

in which J is the Jacobian matrix gathered from partial derivatives of the error functions e_i with respect to correction vector ᐃ.

(16)J=∂e1∂Δ⋮∂eN∂Δ

E is the error vector.

(17)E=e1⋮eN

The partial derivatives are computed as follows

(18)∂ei∂Δ=∂ei∂pMi∂pMi∂Δ

If the reference feature is a plane the error e_i is computed as

(19)ei=pMi−ppTnpnpTnp

in which p_Mi is the i:th measured point and the plane surface is defined by point p_p and normal n_p. The partial derivative of e_i respect to p_Mi is

(20)∂ei∂pMi=1nTTnTnTT

The partial derivative of p_Mi with respect to ᐃ is

(21)∂pMi∂Δ=∂pMi∂Δϕx∂pMi∂Δϕy∂pMi∂Δϕz∂pMi∂Δtx∂pMi∂Δty∂pMi∂Δtz

(22)∂pMi∂Δ=0−zMiyMi−100zMi0−xMi0−10−yMixMi000−1

The updated T and R of the target object pose are calculated as

(23)T=RΔT+T

(24)R=RΔR

Compliant motion control

The compliant motion control primitive, “Cartesian impedance motion control” lays the basis for motion accuracy and the compliant contact motions. The impedance controller regulates the relation between robot’s position x and contact force F according to time domain equation

(25)F=Mtx¨−x¨0+Btx˙−x˙0+Ktx−x0

in which M_t is the target mass, B_t is the target damping, K_t is the target stiffness, x₀ is the nominal position trajectory and x is the actual position. A schematic diagram for the corresponding one degree-of-freedom impedance controlled motion is given in Figure 3.

Figure 1

Primitive skill: impedance controlled contact motion

Figure 2

Compound skill: grinding of an object feature, like plane or edge, with contact to followed object surfaces

Figure 3

The schematic control system transfer functions of the impedance controlled robot

The feasible values for target model parameters depend on the dynamics of the robot, stiffness of the environment and the characteristics of the contact task. Examples of derived impedance parameters for two target systems are [14]:

Parameters for hard contact (K_e = 100*10³ N/m)
1. Target model frequency ω_t = 0.8 * 2 Hz = 1.6 Hz
2. Target stiffness K_t = 100 N / 0.01 m =10000 N/m.
3. Target mass M_t = 10000 N/m / (3.2π rad/s)² = 98.95 kg
4. Target damping ξ_t = 14.75 and B_t = 29351 Ns/m.
5. Target impedance model G_t = 98.95s² + 29351s + 10000
Parameters for soft contact (K_e = 1000 N/m)
1. Target stiffness K_t = 0 N/m (i.e. mass-damper)
2. Target mass M_t = 20 N / 1 m/s² = 20 kg
3. Target damping ξ_t = 1.0 and B_t = 282.0 Ns/m (critically damped)
4. Target impedance model Gt=20s2+282s

A detailed description of designing robust impedance controller for the industrial robot arm is given in [14] and an analysis of the impedance controller’s capability to follow inclined and curved surfaces while keeping the contact force in desired limits is given in [15].

2.2 Skill modeling and compound skills

The compliant motion control primitive, “Cartesian impedance motion control” is outlined as a UML activity model in Figure 1. It has two types of input parameters. The setup parameters, the “configuration” parameters are nominal force, compliance frame, and impedance control parameters (stiffness, damping and inertia). In addition, it has control parameters, i.e. measured forces/torques and as output parameters, the pose increments. As a sequence, it is simple: monitor the contact forces and torques, execute the impedance control compensator, and execute the incremental Cartesian motion increments with the robot. Typically, the compliance frame coincides the tool frame but certain tasks may require offsetting the compliance frame respect to the tool frame. The nominal force is a force offset that deviates the position of the robot to the desired direction in compliance with the target impedance model.

The surface processing skill represents a compound skill. There are many tasks where the management and control of the tool contact to the target object is critical. The surface processing skill relies on the impedance controlled motion sub-skill. It has as setup input parameters process and tool specific nominal force, and tool coordinate frame, which further forwarded to the impedance motion control primitive (the compliance frame is set to the contact point, i.e. the tool end tip) and also the tool dependent maximum speed. The surface processing skill includes motions for approaching the target, getting into contact, executing the process motions in contact and impedance controlled, and finally departing form the target. This is outlined in the Figure 2.

2.3 Skill programming

Programming of a skill can be based on off-line or on-line programming methods. In on-line programming, skill parameters are taught to the system using the actual object in a real world. On the other hand, off-line programming methods rely on using off-line programming tools together with 3D CAD-models of robots, tools, devices, and the target objects to configure the skill parameters. Skill parameters vary with different actions; they should define the actions for all the components in a robot system. Our proposal for the data flow for skill programming is given in Figure 4. Starting from the skill model and object models (or the real object), the skill parameters are acquired, and transformed into forms needed throughout the skill hierarchy.

Figure 4

Skill programming can be done off-line based on geometric models or on-line based on sensor data

2.4 Skill control

The robot system control model represents the architecture,which enables the robot to integrate and synchronize sensor data and robot actions. The control architecture (Figure 5) is based on proprietary and open interfaces, the latter of which are in our case based on ROS (Robot Operating System) [10]. Proprietary interfaces are implemented for connecting devices, i.e. sensors and robot and data interfaces with adapters to ROS. Key functional components representing the objects (or “swim lanes”) in the skill models, include an Object Detector component and a variety of 3D Point Cloud acquisition components (or Point Cloud sources), based on 3D cameras (based on time-of-flight or triangulation) or cooperation of a set of motion provider components (robots, transfer axis and conveyors) together with 2D laser profiler sensor components. Crosswise usage of these are supported over ROS based interfaces for 3D Point Cloud sources and 2D profiles.

Figure 5

A flexible system architecture for integrating sensor data and robot control

3 Case example: Programming and control grinding skills

“Grinding with localization” – task is used as a case example for off-line programmed skills. It consists of lower-level skills such as global rough localization with a 3D-depth sensor, precise localization with a 2D-profile scanner and straight seam grinding. All the skills are programmed inan off-line programming tool RoboDK [11] and implemented in a robot cell, composed of KUKA KR120 R2500 PRO industrial robot with grinding tools, Intel’s RealSense D415 depth sensor and microEpsilon’s 2D-profile scanner. The experimental work object consisted mainly of plane-like surfaces, so all the skills were mainly designed for objects with such features. The process sequence will go as following:

Programming global localization
Programming of the precise (local) localization and related scanning motions
Global localization
Precise localization
Straight seam grinding

First, the work object will be localized globally in the robots workspace with accuracy of a few centimeters. For this, the sensor system needs a set of reference features, in this case a set of 3 planar surface segments and some metric parameters of these (e.g., size and nominal directions and center points). These reference features are programmed with the robot programming tool. The estimated initial position of the work object is used to program scanning movements for the robot to localize the object more accurately. This means, that the robot will use a more accurate 2D-profile scanner to scan local features of the object to get enough data for more precise localization. In precise localization phase, the measured data will be combined with reference data to produce a sub-millimeter-level accuracy for the object localization before applying the grinding motions. Finally, after the object has been localized, the grinding movements can be executed.

4 Experimental results and discussions

4.1 Global localization

The aim of global localization is to locate roughly the work object within the robot workspace. Accuracy of the localization depends on the sensor used in the process. Generally, it should be within a few centimeters, depending on the view angle of the scanning sensor and scanning distance. The global localization sensor used in the experiments is Intel’s RealSense D415, which is a 3D-depth sensor that uses stereo vision with infrared projector to create a depth image of the scene (see Figure 7). 3D-depth sensors are used because they offer a suitable range with reasonable accuracy for this purpose. With multiple external sensors, the whole workspace of industrial 6-DOF robot can be monitored reliably. If the object is known to be located at certain side of the robot, a single 3D-depth sensor was determined to be sufficient.

Figure 6

Skills in “grinding with localization” – task

Figure 7

Colored 3D depth image captured with RealSenseD415 [12]

Localization itself was performed by comparing and matching the reference and measured surfaces. Based on the camera data, the measured surfaces can be represented in robot coordinate system and matched against the reference surfaces. On the other hand, the reference surface normals are represented in 3D-models coordinate system. Thus, the comparison of surface normals will result a position of a 3D-models coordinate system relative to the robot’s coordinate system.

The RoboDK tool was used to gather the reference data from 3D CAD-model; it was already used to program scanning and grinding movements for the robot. The surface data (a 3D point and a surface normal) were obtained by selecting sets of three points, one set from each of three reference surfaces of the 3D-model and calculating the normal and reference points for every surface (see Figure 8). Surface normals and related 3D points from three non-parallel surfaces needed to be gathered to express unambiguously position and orientation of the object. Adding the points and calculation of the normal vectors and the 3D points were done through Python API. On the other hand, corresponding measured surfaces (3D points, surface normal) were solved by segmenting the measured point cloud and using principal component analysis (PCA) for the segments as seen in Figure 6. Direction of the least variance was defined as the surface normal for a segment (blue axis). Main challenge of the segmentation phase is related on finding suitable segmentation parameters, which is very dependent on the geometric surfaces of the work object.

Figure 8

Gathering reference (on the left) and corresponding measured surface points and normals [12]

4.2 Precise localization

After global localization had given an estimate of the object’s location and orientation, scanning movements were updated based on programmed nominal paths. Scanning movements were composed of using a robot with a 2D-profile scanner attached to its flange, performing scanning motions in proximity of the object. 2D-profile scanners were used due to their high-level accuracy, which increases accuracy of the localization significantly. Scanning movements resulted a 3D-point cloud of the features that were within the sensors measurement range, which was then filtered and segmented into different regions using Point Cloud Library’s [13] point cloud processing algorithms.

After filtering and segmentation, different regions of the point cloud represented different planes of the object. The same reference surfaces (obtained the same way as for global localization) were sent to the pose estimator that was integrated into same software, which was used to capture and process the point cloud. Pose estimator corrects iteratively the accurate position parameters so that the distance between the measured point and corresponding surface in 3D- model will be minimized. The pose estimator provides a more accurate pose of the target object in the robot coordinate system, which was used to update position of the target object in the robot program (“user coordinates”). Scanning movements, processed point cloud and fitted point cloud can be seen in Figure 9 and Figure 10.

Figure 9

Scanning the object with microEpsilon 2900-100/BL profile scanner [12]

Figure 10

Segmented point cloud in precise localization, obtained by scanning motions and fitted into 3D-model [12]

4.3 Grinding skills

Even though the object was quite accurately localized after precise localization, some localization errors are still present. Even with high-level equipment and calibration routines, errors will occur due to manufacturing defects that will make the actual object to deviate from what is derived based on the 3D CAD-model. These localization errors were compensated by applying impedance-controlled compliant motions for the grinding movements. In these force-controlled motions, the robot used a force/torque sensor integrated into its flange to measure the contact forces and torques between the object and the tool. Besides encountering localization errors, it was used to enhance even the effect of grinding by applying nominal force to the surface.

Seam grinding skill is based on four input parameters, i.e. target frames that define the characteristics of the seam. Targets were placed to the end regions of the seam, as shown in Figure 11, where target frame pairs were in the start and end of the seam on adjacent surfaces. All the approach and grinding movements were based on these four input target frames. Target frames were placed easily in RoboDK by using a “Teach target(s) on the surface” feature, which sets a target frame so that the Z-axis is perpendicular to the surface and pointing downwards. After this, targets were only needed to be fine-tuned in location and orientation around Z-axis. After the targets had been placed, the paths of the grinding skill were then created and simulated (see Figure 12). The simulation showed if the path could not be completed due to crossing singularities or issues with reachability. All of this made the skill configuration and validating the program for another seam relatively easy and fast.

Figure 11

Defining location and orientation of the seam [12]

Figure 12

Full path of the grinding program visualized [12]

Seam grinding was designed to be in two phases. In the first phase, the seam will be grinded with long vertical movements with angle grinder to remove all excessive material from the seam. Objective of this phase is to result a sharp-pointed seam without any extra material left from welding. In the second phase, the seam will then be rounded by using a sander. Both phases will use the same four input targets to form the grinding programs. Results can be seen in Figure 13.

Figure 13

Grinding skill in different phases in left to right: seam after welding, seam after vertical grinding and fully grinded seam [12]

5 Summary

In this paper we have shown the design and implementation of off-line programmed skills. Skills such as global localization, scanning movements, precise localization and straight seam grinding were used to localize and grind the object. Programmed skills were tested successfully in a real robot environment.

References

[1] A Roadmap for U.S. Robotics (2013) - From Internet to Robotics, 2013 Edition, 129 p. Available in: http://archive2.cra.org/ccc/files/docs/2013-Robotics-RoadmapSearch in Google Scholar

[2] Boada MJ, Barber R, Salichs MA. Visual approach skill for a mobile robot using learning and fusion of simple skills. Robot Auton Syst. 2002;38(3-4):157–70.10.1016/S0921-8890(02)00165-3Search in Google Scholar

[3] Andersen RH, Solund T, Hallam J. Definition and Initial Case-Based Evaluation of Hardware-Robot Skills for Industrial Robotic Co-Workers, Conf. ISR ROBOTIK; 2014.Search in Google Scholar

[4] Weser M, Zhang J. Autonomous Planning for Mobile Manipulation Services Based on Multi-Level Robot Skills, The 2009 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Oct 11-15, 2009, St. Louis, USA.10.1109/IROS.2009.5353901Search in Google Scholar

[5] Pedersen MR, Nalpantidis L, Andersen RS, Schou C, Bøgh S, Krüger V, et al. Robot skills for manufacturing:from concept to industrial deployment. Robot Comput-Integr Manuf. 2016;37:282–91.10.1016/j.rcim.2015.04.002Search in Google Scholar

[6] Heikkilä T, Ahola JM. Robot skills – modeling and control aspects. 14^th IEEE/ASME International Conference on Mechatronic and Embedded Systems and Applications. July 2-4 2018, Oulu, Finland10.1109/MESA.2018.8449184Search in Google Scholar

[7] Morrow J, Khosla P. Manipulation Task Primitives for Composing Robot Skills. Proc of the 1997 IEEE Int’l Conf on Robotics and Automation, Albuquerque, New Mexico. https://doi.org/10.1109/ROBOT.1997.60680010.1109/ROBOT.1997.606800Search in Google Scholar

[8] Schou C, Andersen RS, Chrysostomou D, Bøgh S,Madsen O. Skil-based instruction of collaborative robots in industrial settings. Robot Comput-Integr Manuf. 2018;53:72–80.10.1016/j.rcim.2018.03.008Search in Google Scholar

[9] Stphan Z. Manipulation Skill for Robotic Assembly. MsC Thesis, Technical University of Darmstadt, 2014. 117p.Search in Google Scholar

[10] About ROS [Internet]. [cited 2019 Feb 7] Available from: http://www.ros.org/Search in Google Scholar

[11] RoboDK. [Internet]. [2019; cited 2019 Feb 7] Available from: https://robodk.com/Search in Google Scholar

[12] Saukkoriipi J. Design and implementation of robot skill programming and control, Master’s Thesis 2019, University of Oulu, Finland.Search in Google Scholar

[13] Point cloud library. [Internet] [2018; cited 2019 Feb 4] Available from: http://pointclouds.org/Search in Google Scholar

[14] Heikkilä T, Ahola JM, Viljamaa E, Järviluoma M. An interactive 3D sensor system and its programming for target localising in robotics applications. Proceedings of the IASTED international Conference Robotics 2010, Robo 2010. IASTED, p. 89-96.10.2316/P.2010.703-056Search in Google Scholar

[15] Ahola JM, Koskinen J, Seppälä T, Heikkilä T. Development of impedance control for human/robot interactive handling of heavy parts and loads. 1 Jan 2015. 2015 ASME International Conference on Mechatronics and Embedded Systems and Applications. American Society of Mechanical Engineers ASME. Vol 9.10.1115/DETC2015-47045Search in Google Scholar

Received: 2019-07-02

Accepted: 2020-02-27

Published Online: 2020-05-09

This work is licensed under the Creative Commons Attribution 4.0 International License.

Articles in the same Issue

https://doi.org/10.1515/eng-2020-0037

Keywords for this article

robot skill; programming; sensor; control

Creative Commons

BY 4.0