3D reconstruction study of motion blur non-coded targets based on the iterative relaxation method

2.1 Iterative relaxation method construction

2.1.1 Iterative method

The iterative method starts from a given initial vector x ( 0 ) and constructs an appropriate iterative formula that gradually computes the vector x ( 1 ) , x ( 2 ) , … such that the sequence of vectors { x ( k ) } converges to an exact solution. In this way, for a suitably large k , x ( k ) can be taken as an approximate solution. After the initial vector x ( 0 ) is determined, the estimated value vector can be derived by the following equation:

(1) T x ( k ) = T x ( k − 1 ) + c ,

where k = 1 , 2 , 3 , … . The surface can be assumed to be a cartesian surface z = f ( x , y ) , and the normal to the surface can be replaced by n ( p , q , 1 ) , where p = − ∂ H ∂ x , q = − ∂ H ∂ y , establishing the origin coordinates as shown in Figure 1, where n represents the direction of the normal, θ is the angle of the normal to the z axis, and ϕ is the angle of the normal plane to the x axis. Assuming that the initial vector u ( 0 ) ∈ D is given, the following iterative algorithm is used to calculate u ( k + 1 ) when k = 0 , 1 , 2 , … until u ( k ) the termination condition is satisfied:

(2) W x ( k + 1 ) = ( 1 − α ) W x ( k ) + α T y ( k ) + α P ( u ( k ) ) W y ( k + 1 ) = − β T x ( k ) + ( 1 − β ) W y ( k ) + α Q ( u ( k ) )

where the constant α > 0 , β > 0 .

Figure 1

The coordinate of surface material.

2.2 Iterative image export with relaxation factor

From the above relaxation factor construction, it is clear that the projection matrix R is an M × N matrix. Now, multiply both sides of equation (4) by R T at the same time to obtain the homogeneous set of equations of equation (4) as

Abbreviate equation (5) as

where A = R T R , b = R T y . Satisfying condition a i j ≠ 0 decomposes A into

where the real constant, w > 0 , is called the relaxation factor. Here, it can be expressed as

From the iterative formula for the system of linear equations, we have

The iterative method expressed by iterative equation (9) is called the successive super-relaxation method in order to avoid finding the inverse matrix of L + D w . By making the following change to its iterative equation (9) with L + D w left and collapsing it, we obtain

Due to the special nature of the projection matrix storage, the formula is already very large for just 128 × 128 pictures, which not only takes tens of hours to compute, but also requires a lot of memory space. In practical applications, the size of images is usually 512 × 512 or larger. Therefore, it is unrealistic to use this equation in image reconstruction, and for this reason, equation (10) is modified as follows:

The algorithm derived from equations (10) and (11) is called the iterative method with relaxation factor. By changing x j ( k + 1 ) to x j ( k ) in equation (10), the last two terms in equation (11) are combined into R s x ( k ) . This allows the ( k + 1 ) th iteration to calculate the approximation of the image vector at the same time when the program is executed, instead of calculating x j ( k + 1 ) − 1 and then x j ( k + 1 ) , thus greatly reducing the reconstruction time. The iterative method with relaxation factor is a joint iterative reconstruction method or simultaneous iterative reconstruction method, which is helpful to overcome the sensitivity of the reconstructed image to measurement noise.

3.1 Traditional projective motion path (PMP) motion blur model

The traditional PMP motion blur model holds on the premise that the degradation process of the image is linearly translational invariant; however, this ideal situation is difficult to encounter in real life. This requires that the motion blur image must be partitioned into multiple subregions, and the PMP is consistent at any position in the same subregion. However, such a process makes the whole modeling process tedious and computationally intensive.

Assume that the pixel intensity of any pixel point on an image is determined by the intensity of light received by the photosensitive element on the imaging sensor during the exposure time. Then, the imaging process can also be expressed as

where I ( x ) is the captured image, σ I ( x , t ) d t is the image recorded by the camera during an infinitesimal time interval d t starting from a certain moment t . x is the zeta image coordinate of 3 × 1 and T represents the exposure time. If N moments are sampled within the exposure time and N is large enough, then σ I ( x , t ) d t can be △ I ( x , t i ) approximated instead, so that the integral operation of the entire image can be expressed as a summation approximation.

Disregarding noise, when there is no relative motion between the camera and the scene being photographed, there is

When there is relative motion between the camera and the scene being captured, I ( x ) is equal to the sum of △ I ( x , t i ) for multiple unaligned images. The PMP motion blur model uses △ I ( x , t i ) = △ I ( h i x , t i ) to represent the relative position relationship between the unaligned image at the current moment and the corresponding pixels of the image at the previous moment, where h i is a 3 × 3 single response matrix. Assuming that all h i are known, the clear imaging △ I ( x , t i ) at any moment can be I 0 ( x ) expressed as

Therefore, the PMP motion blur model can be expressed as

where B ( y ) is the motion blur image, the model represents the motion blur image as a superimposed average of multiple clear images, each of which has a unique planar projection transformation relationship with I 0 ( x ) .

3.2 Motion blur model based on the iterative relaxation method

Unlike the traditional PMP model, our model based on the iterative relaxation method effectively restores the motion of the captured target and the imaging process during the exposure time. Our model based on the iterative relaxation method is shown in Figure 2, because the motion blur effect of the image is formed by the instantaneous exposure integral of the target along the motion path during the exposure time. Therefore, the exposure time can be discretely sampled into N moments, N large enough. The “reference position” of the selected non-coded point is the center of the circle with the origin O w of the world coordinate system O w X w Y w Z w and the plane of the non-coded point is located in the plane X w O w Y w . At moment i ( i = 1 , 2 , … N ) , the non-coded point is moved from the reference position to the current “moment” position by a rotation transformation R i and a translation transformation t i . This is then projected onto the camera imaging plane through a perspective transformation to obtain a clear image of the non-coded point at i . After N different moments in the exposure time, a total of N images are projected on the camera imaging plane, and the average image produced by their superposition is the motion blur image produced during the entire exposure time.

Figure 2

Motion blur model based on the iterative relaxation method.

According to the camera imaging model, the projection process of the non-coded pattern at moment i can be quantitatively represented as

where ( X , Y , 0 ) denotes the coordinates of any point on the encoded pattern in the world coordinate system at the reference reference pose. I i is the clear image at the i th sampling moment, ( u ( I i ) , v ( I i ) ) denotes the pixel coordinates in the clear image I i . I ( ( u ( I i ) , v ( I i ) ) ) denotes the gray value of the clear image I i at pixel location ( u ( I i ) , v ( I i ) ) , and C ( X , Y , 0 ) denotes the gray value of the coded pattern at point ( X , Y , 0 ) .

Since the spatial attitude of the encoded point described by the attitude matrix R i can be represented by three angles of rotation around axes X ( w ) , Y ( w ) , and Z ( w ) in turn. The angle of rotation about axis X ( w ) is called the deflection angle, the angle of rotation about axis Y ( w ) is called the pitch angle, and the angle of rotation about Z ( w ) is called the slew angle. In this article, these three angles are written in vector form as r i = r x r y r z T . The spatial position at the current moment i is represented by D ( i ) as the coordinates of that spatial position. For the same moment i , the rotational R i and translational transformations t i are the same for any point on the coded pattern. Therefore, the projection process of all points on the non-coded pattern can be summarized by the function g ( • ) , with

According to the mechanism of motion blur effect formation, there are

where B is the final output blurred image. Substituting equation (18) into equation (19), we obtain

The SMP motion blur model established in this article, is related to the pose vector r ( i ) and D ( i ) the spatial position of the target at each moment, in addition to the parameters of the camera itself. r ( i ) describes the motion pose of the target, and D ( i ) describes the motion path of the target.

It can be seen that the motion blur model based on the iterative relaxation method more intuitively expresses the quantitative relationship between the spatial motion path of the target and the motion blur effect.

3.3 Non-coding point 3D reconstruction

The result of optimizing the initial value of the 3D spatial coordinates of the coded point localization center D cinit is recorded as D copt , and then the relationship between the left and right camera positions in the combined camera imaging model and binocular stereo vision. The pixel coordinates D coptl ′ and D coptr ′ of the optimized localization center D copt of this coding point in the corresponding left and right images, respectively, can be expressed as

For camera calibration, a single response relationship is established between the camera imaging plane and the calibration plate plane. The single-response matrix H is obtained by the correspondence between the two groups of five feature points. Then, the single-response relationship between the pixel coordinate systems O 2 l u l v l and O 2 r u r v r of the left and right corresponding images and the divided area can be established, respectively,

where D f is the spatial coordinate of any point on the delimited area, D l ′ and D r ′ are the pixel coordinates of the imaging point of point D f in the left and right corresponding images, respectively. H l and H r represent the single-response relationship between D f and D l ′ and D r ′ , respectively, since the single-response matrix is invertible.

Therefore, the single-response relationship H l r between the corresponding points D l ′ and D r ′ in the left and right corresponding images can be expressed as

When there is occlusion or large image noise in the captured image, it will largely reduce the gray value and the reconstruction drift will occur. At this time, the non-coding point 3D reconstruction process is shown in Figure 3.

Figure 3

Non-coding point 3D reconstruction process.

4.1 Experimental setup

4.1.3 Experimental procedure

Set the camera frame rate, exposure time, gain, and other parameters to obtain clear and stable images. Set the laser wavelength, power, scanning speed, and other parameters to obtain clear and stable motion blur coding points. Lay the motion blur non-coded points on the moving target surface to simulate the actual application scene.

The experimental flow of 3D reconstruction of moving objects in this article is shown in Figure 4, which is generally divided into two parts circled by the dashed boxes (serial numbers 1 and 2, respectively) in the figure. The first part is to calculate the central initial and optimized intra-frame motion paths of motion blur non-coding points using the motion blur non-coding target 3D reconstruction algorithm based on the iterative relaxation method.

Figure 4

Experimental flow of 3D reconstruction of motion target.

4.2 Data acquisition

First, ten simultaneous timing images were acquired at different exposure times to obtain different degrees of motion blur non-coded points. Second, ensure the uniform position and angle of the camera and laser projection system to obtain stable and accurate motion blur non-coded points. Finally, the acquired image data are saved to the computer for subsequent processing.

4.3 Reconstruction results and error analysis

The actual size of all non-coded points used in this article’s experiments are 25 × 25 red spheres at the end of camera calibration. First, the 3D reconstruction technique of motion blur visual features is established to study the 3D spatial coordinate points of these non-coded points at rest. The motion paths of the red blob under the original coordinate axes are shown in Figure 5, where Figure 5(a) shows motion path 1 of the blob at exposure time △ t , 0.5, 0, − 0.5  m in the x axis, respectively. 0.5, 0, − 0.5  m in the y axis, respectively. 0, 2, 4, 6, 8 m in the z axis, respectively. The blob was in a spiral pose in the 3D coordinate space, and the camera was calibrated at the end of under the camera calibration, the trajectory of the ball in three rotations was taken. Figure 5(b) shows the motion paths of the red ball in the 3D coordinates after z , with the shooting time interval △ t being 0.5 s. Finally, the motion paths of each coded point at moment t = 0 for different exposure times were obtained using an iterative relaxation method based on the motion blur non-coding target 3D reconstruction algorithm, including the initial intra-frame motion path (initial path) and the optimized intra-frame motion path (optimized path).

Figure 5

The motion path of the ball under the original coordinate axis.

Now that the intra-frame motion paths of the non-coded points have been optimized, the blurring degree of the coding point images can be evaluated according to the blurring degree formula defined in the previous section in order to have a more intuitive understanding of the degree of motion blur generated at the exposure times set in this experiment. In this study, the velocity of the moving target can be simulated by extending the exposure time, but considering only the velocity factor is not comprehensive enough. Because speed and exposure time are interrelated, we need to consider the combined index of the two – the blur level. Experiments at different blur levels can more comprehensively verify the adaptability and stability of the algorithm for different situations. Therefore, the degree of fuzziness is more reflective of the situation in real applications and more accurate in assessing the performance and reliability of the algorithm.

The average blur degree [16] of all non-coded targets in the experiment at each exposure time calculated separately and the average reconstruction error of the motion path within the frame calculated are shown in Table 1. Where ε i and ε j denote the average reconstruction errors of the initial path and the optimized path, respectively. It can be seen that the reconstruction accuracy under the optimized path has improved by two orders of magnitude compared with that under the initial path at different exposure times. However, the reconstruction error basically remains around 0.042, and the maximum does not exceed 0.05. The model algorithm execution of the article takes approximately 3 min.