An innovative binocular vision-based method for displacement measurement in membrane structures

2.1 Overall flow

In image recognition, the accuracy of results hinges significantly on the methodology used for extraction. This article introduces a novel approach to pinpointing the target point. Figure 1 illustrates the comprehensive flow of the image recognition process, wherein the innovative method proposed in this study is applied during the target point identification phase. In the process of measuring the deformation of binocular stereo vision, it is common to install artificial identification points on the key parts of the surface of the object to be measured and then establish the binocular visual measurement system to identify and process the identification points.

Figure 1

Calculation flow of image recognition.

2.2 Identification of effective areas based on SIFT characteristics and threshold correlation methods

The membrane surface is marked with key points, and image data are acquired using a binocular vision measurement system. During image acquisition, the range of pictures taken by the camera is much larger than the marked area, making it difficult to avoid interference of invalid areas during the extraction of the image identification point. In this article, a method based on SIFT characteristics and threshold correlation is proposed to identify the effective area of the identification point. This method is divided into two steps: identifying the candidate area for the identification point and then determining the effective identification point area. The common identification points are generally the ones shown in Figure 2a. However, this article proposes a new improved marking point, as shown in Figure 2b.

Figure 2

Marking points: (a) common marking points and (b) proposed marking point.

First, the acquired image area is filtered and then binarized. In the binary image analysis, the algorithm of a connected area marker is adopted by marking the target pixels in the binary image. Each individual connected area forms an identified area block, so as to further obtain the outline, centroid, and other parameters. The two-pass algorithm is used to analyze and extract the connected areas. For the identification of the effective area of the identification point, the SIFT local feature detection algorithm is proposed, and the Fast Library for Approximate Nearest Neighbors (FLANN) region feature matching algorithm is combined to complete the extraction and matching of the SIFT feature points on the template image and the candidate area image. The SIFT feature point extraction is divided into four steps: (i) constructing scale space, (ii) difference of Gaussian (DOG) space extreme value detection, (iii) key point direction allocation and description, and (iv) keypoint matching. The Gaussian kernel function is used to construct a scale space, as presented in Eq. (1).

where x, y are the coordinates of pixels. The image continuously transforms the scale by the Gaussian kernel function and finally obtains a multi-scale spatial sequence. The scale space of a two-dimensional image can be represented as follows:

where G ( x , y , σ ) is a gauss-scale variable Gaussian function, L ( x , y , σ ) is called Gaussian-scale space, and σ is the size determining how smooth the image is.

To detect extremum points and present in scale space, each pixel is compared to all adjacent points of its image threshold and scale threshold. According to the DOG definition formula, the local maximum value is searched for in different scale spaces, and the detection of the key points of the DOG space is completed by comparing them with the neighboring points. The DOG definition formula is as follows:

where k is a parameter depending on the test situation. For the local extremum points of the detected DOG space, the DOG function is expanded using the Taylor series by fitting to the DOG function of the scale space, as shown in Eq. (4), where D is the differential operator. Determine the location and scale of key points and eliminate some unstable edge feature points to improve the accuracy of matching

The key point description subset is established for the template image and the screened candidate area image, and the target is identified by comparing the key descriptors in the two sets, and the similarity of the key point descriptors is measured by Euclidean distance matching method, as shown in Eq. (5).

To exclude feature points without matching relationships due to the confusion and complexity of the image background, the SIFT key points in the template image are extracted using the FLANN feature matching nearest neighbor algorithm and combined with the k-nearest neighbor method. The first two key points are found in the candidate area according to the minimal distance rule. If the ratio between the two distances is judged to be an error match point, the matching process is shown in Figure 3. By matching the template features and the characteristics of the current candidate area, the number of key points and matches is obtained. If the number of key points and matches surpass the set threshold, the area is considered a valid area.

Figure 3

FLANN feature matching: (a) error match point and (b) match the points correctly.

2.3 Common methods of detection

Since the light intensity and the noise are sensitive to the change of the environment in practice, this article proposes an adaptive Canny edge detection algorithm and combines the OTSU (proposed by Ōtsu Nobuyuki) algorithm [21] to identify the target in the image. In the circular marking point often used in binocular stereo vision measurement experiments, when the position of the circular marking point is not perpendicular to the optical axis of the camera, whether the Steger method is used to extract the center of the ring or the OpenCV native blob method to obtain the center of the ring, there is a certain accuracy problem between the actual center of the measured feature point and the center of the feature point on the captured image. The optical axis of the camera and the position of the identification point are not always vertical during the actual shooting process and, in practice, the membrane structure usually has large displacement due to its flexibility.

The characteristics of the membrane structure should be fully combined when posting the marking point. It is necessary to choose an identification point that can be easily detected and identified and can accurately reflect the actual deformation displacement of the corresponding part. This article makes a new improvement on the basis of the circular identification point, that is, a pair of right-angle sectors are added to the center of the inner ring position. Figure 4 shows the result of image binarization using the OTSU method on the target point.

Figure 4

OSTU binarization results: (a) target point on specimen and (b) gray map of the target point.

Two methods are considered to detect the target point: right-corner angle detection and ellipse edge detection. In this article, the right-corner edge is used to obtain the center coordinate at the corner point, noted as P rc . The edges of the outer ring that identify them are then inspected. Since circles tend to appear elliptical in camera imaging, the benefits translate into the detection of elliptical edges. There are two ellipses in the image. The center coordinate detected from the edges of the outer ellipse is noted as P oc , and the coordinates derived from the inner ellipse are noted as P ic . Right-angle edge detection and ellipse edge detection are shown in Figure 5. The center coordinates (P _wcr) of the obtained inner and outer ellipses are weighted and averaged, and the pixel coordinates after weighted fusion are obtained, and the formula is as follows:

Figure 5

Marking point edge recognition: (a) right angle corner detection and (b) ellipse edge detection.

2.4 Proposed detection method

The right-angle method and the ellipse edge method are typically effective at obtaining accurate results when the angle or ellipse is situated on a plane. However, significant deformations occur on the surface of a membrane structure during the load applied, causing the initial right angle and circle to undergo large deformations and not lay on the plane. Therefore, there can be significant errors in the detection of these methods. Interestingly, the right-angle method usually yields a far point, and the ellipse-edge method yields a near point.

Therefore, the weighted fused ellipse center coordinates are fused with the center coordinates based on the edge detection of corner points by a second weighted average, and the result is regarded as the final pixel coordinates that identify the center of the point, with the following formula:

where P wcr is the center coordinate of the identity point of a weighted average fusion, P rc is the center coordinate of the fit of the right-angle method, and the center coordinate of the identification point, P wc the secondary weighted average fusion is the final pixel coordinate of the identification point. Hence, considering the complex environment of binocular vision technology and the object to be measured, the center coordinates can be weighted and averaged by the results of three methods combining Eqs. (6) and (7). The final formula is shown in Eq. (8).

where α 1 , α 1 , α 1 are the coefficients and α 1 + α 2 + α 3 = 1 . The coefficients are determined by the shape of the target figure. In this article, the weighted coefficients are chosen as the three Gauss points 0.5, 0.25, and 0.25, respectively.

In the article, a method for image matching is presented that integrates a grayscale-based normalized cross-correlation (NCC) algorithm with Euclidean distance matching. This technique calculates the cross-correlation values between a template image and a sequence image to determine their degree of matching, focusing on the grayscale values. Additionally, it employs Euclidean distance matching, where the nearest Euclidean distance between identification points in the template and those in a sequence image sub-graph is compared against the sub-progressive Euclidean distance of all corresponding points. This comparison helps in identifying correct matching pairs. The method is applied to both left and right image sequences, ensuring accurate matching of identification points over time. The principles and process of this matching technique are visually detailed in Figure 6.

Figure 6

Principle of marker point matching.

3.1 Specimens

In this study, the experiments have been completed in collaboration with Shanghai WeiBo Membrane Company of Yancheng City, Jiangsu Province. Using the self-made membrane tension equipment, the four levels of tension membrane were loaded. The membrane selected for the test is made of polyvinylidene difluoride (PVDF). The material properties are shown in Table 1.

The specimens used in the tests were divided into four sizes. Specimens 1 and 2 correspond to the two design schemes under the uniaxial test of planar tensile conditions, specimen 3 is a design under the biaxial test of planar tensile conditions, and specimen 4 is a saddle-shaped tensile membrane. The cutting process and production of the specimens were carried out by the Weibo Company, as shown in Figure 7.

Figure 7

Production process of specimens: (a) specimen cropping and (b) high-frequency welding machine.

The design scheme is shown in Figure 8, where there are five object points in specimens 1, 2, and 3 and six object points in specimen 4. The layout of the object points is shown in Figure 8, and practical object points can be seen in the specimens, as shown in Figure 9.

Figure 8

Schematic diagram of the test piece design: (a) specimen 1, (b) specimen 2, (c) specimen 3, and (d) specimen 4.

Figure 9

Physical view of the test piece: (a) specimen 1, (b) specimen 2, (c) specimen 3, and (d) specimen 4.

The physical drawings of the four specimens are shown in Figure 6. To ensure the precision of the marker points, we adopted a distinct marking method for specimen 2 compared to specimens 1, 3, and 4. Specifically, for specimen 2, we marked an annulus, whereas for specimens 1, 3, and 4, we marked the points as a ring edge plus an annulus.

3.2 Test apparatus and instruments

The test apparatus was independently developed and designed by our research team. The support structure is a welded steel bracket support system with a total size of 3,080 mm × 3,080 mm and an overall height of 1,500 mm, and the brackets were made of square steel pipes measuring 40 mm × 40 mm. The design drawing and actual structure are shown in Figure 10.

Figure 10

Test support device: (a) design drawings and (b) equipment for support.

The test was conducted to consider different pretension levels and various load cases. The pretension was applied at the edge of the membrane. To apply point load to the surface of the membrane, bolts were fastened onto the membrane surface. The bolt weights were 0.2, 0.5, and 1 kg, respectively, to represent three different point load cases. The pretension device consisted of a self-locking winch and a wire rope fixed around the support. For flat tension membranes, the membrane boundary was connected to the membrane using strip steel plywood and wire rope. For surface tension membranes, wire ropes and manual winches were connected using rigid splint connections at the corners. During the test, the rotating manual winch tightened the wire rope, thereby providing a certain tension to the membrane, as shown in Figure 11.

Figure 11

Test equipment: (a) the planar membrane test and (b) the surface membrane test.

The test utilizes the DYLY-103 S tensile force sensor to measure the pretension value. The sensor has a measuring range of 1.0–1.5 tons with an accuracy of 0.02%. To acquire displacement data of each measurement point on the membrane surface, this article employs the HBVCAM binocular camera model as the image acquisition device of the binocular stereo measurement system. The specific parameters of the camera are listed in Table 2.

To evaluate the precision of the method, multiple HG-C1200 Panasonic laser displacement sensors were deployed. These sensors feature a measurement range of ±80 mm and offer an accuracy of 0.1% along with a resolution of 0.01%.

3.3 Experimental process

There are 14 working conditions in total, considering different pretension loads and the rise-to-span ratio of the membrane surface. A summary of the test conditions is shown in Table 3.

For each working condition, the deformation measurement of every point on the membrane surface took into account four factors: tension force, the arrangement of data acquisition devices, the application of load, and the data acquisition process itself. In the setup phase, the laser displacement sensors were securely mounted beneath the membrane surface, aligned with each measurement point. The binocular camera was positioned at a diagonal angle above the membrane, supported by a tripod to maintain absolute stability. It was crucial to ensure that the camera remained completely stationary throughout the process. The camera’s light was oriented at an approximate 45° angle to the ground, and it was positioned roughly 1.0 m from the membrane surface, as depicted in Figure 12.

Figure 12

Binocular camera acquisition process: (a) binocular camera placement and (b) image collection.

Prior to load application, the internal and external parameters of the binocular camera were calibrated [22]. Figure 13 illustrates the checkerboard calibration plate employed in this process. At each stage of loading and unloading, the membrane structure was captured using the binocular camera. It is important to clarify that the 16 images captured by the cameras were taken at random angles within a range of 0–60 degrees. This approach was deliberately chosen to test the robustness of the calibration method rather than adhering to a set of fixed angles.

Figure 13

Camera calibration: (a) left camera and (b) right camera.

4.1 Measurement accuracy test

In order to verify the accuracy of different extraction methods, three extraction methods were used: the right angle fitting method named M1, weighted fusion based on the inner and outer circles named M2, and based on the weighted method of M1 and M2, named M3. The specimen with a pretension load of 320 kN was used for validation. The error between the calculated displacement and the actual displacement was then calculated using the Euclidean distance method and shown in Table 4, where the “Dis” means the displacement detected by laser measurement and “Cal” represents the calculated value. It should be noted that for the M1, M2, and M3 methods, the coefficients of ( α 1 , α 2 , α 3 ) used for Eq. (3) are (1, 0, 0), (0, 0.5, 0.5), and (0.5, 0.25, 0.25), respectively.

The errors in measuring displacement using lasers for methods M1, M2, and M3 are presented in Figure 14. This article suggests a method that combines the corner point, inner circle, and outer circle in a weighted manner to obtain the center coordinate of the identification point. As illustrated in Figure 14, this approach has a relative error of less than 5%, which is considerably lower than the results obtained from M1 and M2. Although the identification point extraction based on corner point fitting and elliptical fitting can demonstrate changes in the target measurement point’s displacement, it does not offer the same level of accuracy as the proposed approach in this article.

Figure 14

Errors in laser measurement of the specimens.

4.2 Displacement results

After completing the calibration of the binocular camera, the images collected by the camera at each stage were combined, and image processing based on the OpenCV 4.4.0 environment on the Visual Studio 2017 platform was used to obtain the 3D displacement information of each measurement point on the membrane surface using binocular stereo vision technology. The displacement data of each measuring point on the membrane surface were collected using a laser displacement sensor. The displacement results of each measuring point under different working conditions and three loading stages were compared. In this section, only the displacement results of measurement points A and C under pretension of the four membrane types are reported, as shown in Figure 15. It is important to highlight that only the out-of-plane deformation of the membrane, subjected to three loading stages, was considered, while the in-plane deformation was disregarded.

Figure 15

Results of displacement of specimens 1–4 at points A and C measured at 480 N and the rise-to-span ratio 1/8: (a) measurement point A of specimen 1, (b) measurement point C of specimen 1, (c) measurement point A specimen 2, (d) measurement point C specimen 2, (e) measurement point A of specimen 3, (f) measurement point C of specimen 3, (g) measurement point A of specimen 4, and (h) measurement point C of specimen 4.

The relative error of the displacement measured at 480 N with the span ratio of 1/8 is shown in Figure 16. It should be noted that the axis label of “test position” in Figure 16 corresponds to the data from Figure 15 presented in order. It can be seen that the relative error of all cases is less than 10%.

Figure 16

Relative error of the displacement at the case of 480 N and span ratio 1/8.

Due to the variety of membrane structures, in order to explore the applicability of the binocular visual measurement method to different membrane surface forms, the displacement results of each measuring point under the full load state of the four pretension membrane forms are compared, as shown in Figure 17. There are five test positions named as “A,” “B,” “C,” “D,” “E,” respectively, for each case. Thus, each figure displays five pairs of data. Figure 18 shows the displacement data obtained from specimen 4 with different span ratios. It is evident from Figures 17–18 that the displacement error is very small when compared to the measurements obtained through a laser displacement sensor.

Figure 17

Displacement results of specimen 3 at full load under different pretensions: (a) displacement on specimen 1 of 320 N, (b) displacement on specimen 1 of 480 N, (c) displacement on specimen 2 of 320 N, (d) displacement on specimen 2 of 480 N, (e) displacement on specimen 3 of 320 N, and (f) displacement on specimen 3 of 480 N.

Figure 18

Results of displacement of specimen 4 with a different rise-to-span ratio: (a) displacement with a ratio of 1/8 and (b) displacement with a ratio of 1/11.

Figure 19 illustrates the relative error of displacement for specimens 3 and 4. The results are the relative error derived from Figures 17 and 18 for each case. Notably, the majority of the measured values have a relative error of less than 10%. However, there is one exception – point E in the 480 N case – with a relative error of 16%, which is highlighted in Figure 19. The observed error at this point is attributable to significant local deformation in the membrane. The deformation caused by this localized effect appears to have a disproportionate impact on the measurement accuracy, resulting in the comparatively larger relative error observed at point E.

Figure 19

Relative error of the displacement of specimens 3 and 4.

From the measurement results, it can be seen that the binocular visual measurement method proposed herein is consistent with the displacement acquisition result of the identification point and the measured result of the laser displacement sensor. There is no significant change in the error between the two, which can indicate that the accuracy of the proposed measurement method will not be affected by the change in the form of the tension-drawn membrane.