Improved deep learning segmentation of outdoor point clouds with different sampling strategies and using intensities

1 Introduction

The importance of outdoor objects in supporting the environment and infrastructure cannot be overstated [1]. The outdoor digital twin has been attracting attention since the introduction of large data sets and the development of 3D data acquisition technology [2]. Outdoor environments, which encompass natural elements and man-made structures, play a vital role in shaping our surroundings. Understanding and modeling these outdoor situations is crucial for various applications, ranging from urban planning and landscape design to environmental monitoring and infrastructure development [3].

Modeling 3D outdoor objects has traditionally relied on methods such as terrestrial surveys, close-range photogrammetry, and laser scanning [4,5]. Among these techniques, laser scanning has emerged as the most promising approach for accurate and automated 3D modeling. Laser scanning methods can be divided based on their platform into terrestrial, airborne, unmanned, and mobile laser scanners [6,7]. On the ground, terrestrial laser scanning (TLS) offers unparalleled precision and detail, making it an ideal choice for capturing the complexity of outdoor environments [8,9,10,11,12].

The successful use of deep learning for various applications also invites researchers to use it for outdoor points cloud tasks, such as classification and segmentation. The use of deep learning to process point cloud data has increased recently because of its accuracy and adaptability to new data [13]. Chen et al. [14] and Liu et al. [15] used a deep learning method for tree classification. Chen et al. [14] proposed the Point Cloud Tree Species Classification Network (PCTSCN) to overcome challenges in classifying tree species from laser data acquired from TLS and unmanned laser scanning. PCTSCN utilizes geometric sampling and improved Farthest Point Sampling for local and global feature extraction from laser data. By incorporating tree point cloud intensity and spatial information, PCTSCN aims to enhance classification accuracy through a nuanced consideration of both local and global characteristics. The trained models achieve 96% classification accuracy and a kappa coefficient of 0.923. Liu et al. [15] used PointNet++ to classify a tree point cloud data set acquired from a backpack laser scanner and achieved 98.26% classification accuracy. In more depth, the research analyzes the influence of leaf point clouds on classification, dissecting individual tree point clouds into leaves and wood, utilizing five downsampling methods for experimental design: farthest point sampling, K-means, random, grid average sampling, and nonuniform grid sampling.

Another research by Chen et al. [16] used PointNet to segment tree crowns, achieving 85, 87, and 86% for recall, precision, and F-Score, respectively. The methodology involved voxelization to subdivide point clouds into voxels, serving as training samples for the PointNet deep learning framework to identify tree crowns at the voxel scale. Subsequently, height-related gradient information refined initial segmentation results, accurately delineating tree crown boundaries, and the effectiveness of the approach was validated by comparing retrieved tree crown breadths with field measurements. Krisanski et al. [17] deployed PointNet++ to segment forest environments and obtained 95.4% overall accuracy. The point cloud data are obtained from various sensing systems across diverse forest conditions. The model, trained and evaluated on seven manually segmented data sets, achieves notable per-class segmentation accuracies, enabling the extraction of a precise Digital Terrain Model with a mean error of 0.04 m and 99.9% completeness, as demonstrated in a benchmarking study.

Several previous studies show that the deep learning method is usable for tree or vegetation segmentation and classification, producing accurate results even where none of them obtain a classification score below 80%. Notably, Liu et al. [15] and Krisanski et al. [17] leverage PointNet++ for tree species classification, while Chen et al. [16] adopt PointNet, and Chen et al. [14] employ Farthest Point Sampling – methods that align with the effectiveness of PointNet++. Building upon these foundations, this research is motivated to employ a deep learning architecture, specifically PointNet++, to address the inherent challenges in outdoor semantic segmentation.

While numerous methods and techniques are available for segmentation and information extraction from point cloud data, most of them have primarily focused on homogeneous objects, using geometric and radiometric features. However, the full potential of TLS, which provides intensity values for each point, has yet to be optimally explored. Several studies show that using intensity information obtains remarkable results and can improve accuracy. Harnessing the rich intensity data provided by TLS has the potential to unlock new dimensions of precision and detail in segmentation and information extraction processes, further enhancing the overall efficacy of these methodologies.

Li et al. [18] provided a supervised classification algorithm of ALS altimetry data combined with laser intensity. The method involves terrain data generation through initial height filtering, subsequent raw laser intensity filtering, and the analysis of statistical results combined with height information, particularly crucial for distinguishing between objects of similar height, such as trees and buildings, through their distinct laser impulse reflections. The result shows that intensity in point cloud data highlights the difference between objects (trees and buildings) compared with only using height slope information. Reymann and Lacroix [19] combined intensity and multiple echoes for outdoor point cloud classification. The research presents an empirical evaluation of a low-cost LiDAR, emphasizing the integration of features derived from intensity and multiple echoes within a hierarchical classification scheme to enhance the overall performance of the classifier. The result shows that the use of intensity can improve classification accuracy compared with only using geometry point cloud classification.

Sun et al. [20] combined intensity and geometric (spatial) information for automatic wood–leaf classification. The research proposed a three-step classification and verification method that leverage intensity and spatial information, utilizing intensity thresholds, neighborhood density, and voxelization to automate wood-leaf classification. The classification results were then compared with manual standards, employing Overall Accuracy (OA), Kappa coefficient (Kappa), and Matthew’s correlation coefficient. Several research studies show that using the intensity attribute of point cloud data can achieve automation, accuracy, and high-speed classification.

According to similar previous research, intensity potentially can be used to extract information from point cloud data. Intensity reflects the material characteristics of objects and can be widely used to characterize terrain features and classification [21]. Intensity from vegetation and the ground can be easily detected and distinguished [18]. Intensity measures the amount of reflected light and is much less dependent on the scene illumination conditions than the intensities perceived by passive cameras [19]. The intensities’ values depend on the scanning features (radiation range and reflecting ability of the objects) and the environmental conditions (season and weather) [22].

In addition to intensity data that still receives little exploration, most research also does not specify the distribution of the data set used in deep learning processing, but it is important because the method highly depends on the data set. Research on the effective sampling method for deep learning is scarce. Johnson and Khoshgoftaar [23] studied the effect of three data samplings with high imbalance data with deep learning. The research delves into the challenging task of training predictive models with class-imbalanced data, emphasizing the increasing difficulty posed by extreme levels of imbalance in the era of big data. The results show the hybrid between random over-sampling and under-sampling is identified as a preferred method for effectively addressing high imbalance within big data problems, maximizing both performance and efficiency.

Katharopoulos and Fleuret [24] proposed a sampling schema by focusing training with data that gives the biggest change in parameters or conducting an importance sampling. The research addresses the computational inefficiencies in deep neural network training, where a significant portion of computation is dedicated to well-handled and potentially ignorable examples. The proposed solution introduces a principled importance sampling scheme aimed at concentrating computation on “informative” examples, thereby diminishing variance in stochastic gradients during training. However, most of these works are still search theories with little applicability to wider problems.

In the field of deep learning for outdoor point cloud semantic segmentation, various studies, including those by Liu et al. [15], Krisanski et al. [17], Chen et al. [14,16] have successfully demonstrated the power of deep learning methods, showcasing high accuracy in segmenting objects like trees. While existing literature has explored the use of XYZ coordinates and RGB radiometric color, the incorporation of intensity as an additional parameter has not been thoroughly examined in the context of deep learning semantic segmentation. Li et al. [18], Reymann and Lacroix [19], and Sun et al.’s [20] works, for instance, delve into intensity for outdoor classification but do not touch upon deep learning techniques. Similarly, Johnson and Khoshgoftaar [23] also Katharopoulos and Fleuret [24] analyze the impact of sampling on deep learning but do not specifically address point cloud data or semantic segmentation. Thus, this research aims to bridge this gap by focusing on the preparation of data sets for deep learning semantic segmentation, exploring the use of additional parameters, and conducting comprehensive experiments. This research compares different sampling strategies in data set preparation, varying the inclusion of intensity and non-intensity data, and assesses the performance of deep learning models for semantic segmentation in outdoor environments.

This research serves as a bridge in the literature, connecting the gaps between data set preparation, the use of additional parameters, and the semantic segmentation process in point cloud data. It distinguishes itself by comparing random and stratified sampling methods, which are rarely discussed in the existing body of literature. Following data set preparation, the study examines the effectiveness of intensity as a parameter in point cloud data for deep learning semantic segmentation. By comparing models with and without intensity, employing both random and stratified sampling, the research conducts a thorough evaluation of deep learning performance. In summary, our work not only contributes to the understanding of the impact of different sampling strategies and intensities in semantic segmentation but also builds upon previous studies by addressing these aspects in the specific context of outdoor environments using deep learning techniques.

Therefore, the objective of this research is to analyze and evaluate different sampling strategies and optimization techniques, and explore the intensity information of point cloud data. The primary aim is to enhance segmentation performance using deep learning algorithms, particularly focusing on distinguishing outdoor vegetation from man-made structures. By harnessing the intensity data and two sampling strategies, this research seeks to unlock new possibilities and improve the accuracy and efficiency of outdoor object segmentation.

2 Methods

The research was conducted in AGS Park located in an area of the Faculty of Engineering at Universitas Gadjah Mada. This park features a combination of low and medium vegetation as well as various man-made objects, including park lamps, gazebos, chairs, tables, and a dome structure. The scanning of the park occurred in the morning, providing sample illumination for the visibility of all objects in the area. The selection of this park as the research location was motivated by the diverse range of outdoor objects.

This research uses two types of data sets, with and without intensity. In general, this research method comprises data acquisition, data processing, the deep learning process, and analysis. PointNet++ architecture is chosen as the deep learning model, a deliberate choice over alternatives such as PointNet. PointNet++ offers distinct advantages by effectively capturing both local and global features of point cloud data [25], providing a nuanced understanding that distinguishes it from the original PointNet. Additionally, this research method extends to data set preparation, a facet often overlooked in existing research. By employing several sampling strategy methods, this research contributes a comprehensive understanding of the data set preparation process, enhancing the reproducibility and reliability of segmentation results. Moreover, the methodology diverges from conventional approaches by incorporating intensity analysis in the segmentation process. While traditional deep learning segmentation rarely involves intensity analysis, this research recognizes the significance of intensity variations between vegetation and non-vegetation in the research area.

Point cloud data were acquired using Leica RTC 360 TLS with high-resolution settings. The scanning process uses the full power of the TLS instrument with specifications outlined in Table 1. The choice of the Leica RTC360 was crucial for several reasons. The high-resolution parameters are used to ensure the capture of high-resolution point cloud data, allowing for a detailed representation of the outdoor environment. The use of this laser scanner also facilitated efficient data acquisition due to its portability and easy-to-use instrument, contributing to the overall speed and accuracy of the scanning process. These specifications not only align with industry standards but were also chosen to meet the specific requirements of research objectives.

Each point in the point cloud data acquired has radiometric features or RGB colors and intensity information, which can be used in segmentation. The data are registered with the cloud-to-cloud method into one data set using Leica Cyclone Register 360 with an RMSE of registration under 0.001 m. The cloud-to-cloud registration method with a remarkably low RMSE ensures accurate alignment of point cloud data. To eliminate noise and outliers in the data, this data set is then filtered using CloudCompare with two filtering tools, the statistical outlier removal filter for removing outliers based on statistical calculations [26] and the radius outlier removal filter for eliminating outliers based on the number of neighbor points within a certain radius [27]. This dual-filtering approach not only eliminates outliers effectively but also addresses potential biases and errors, ensuring that the final point cloud data set is reliable and reflective of the true outdoor environment. The high specification parameters of the Leica RTC360 laser scanner, including its precision in capturing radiometric features and RGB colors, contribute to minimizing acquisition errors. To further enhance data quality, a meticulous noise reduction process is conducted using two filtering tools, ensuring the robustness of data processing techniques. From this step, the data set was prepared for the deep learning process. Figure 1 shows point cloud data in AGS Park.

Figure 1

Point cloud data in AGS Park.

In assessing the robustness of data processing methods, the diverse conditions encountered during the outdoor scanning process are considered. The scans were conducted in a park comprising low, medium, and high vegetation, representing a range of environmental complexities. Importantly, the data acquisition was performed exclusively in the morning under clear sunlight conditions, ensuring optimal visibility without the presence of clouds or rain. This deliberate choice aimed to capture the park in its entirety with high clarity. Moreover, the effectiveness of data processing methods extends beyond these controlled conditions. The use of advanced instruments with high precision parameters contributes to the adaptability of research methods to varying outdoor scenarios. The cloud-to-cloud registration method enhances alignment accuracy even in landscapes with diverse vegetation densities. The noise filtering method demonstrated consistency to eliminate noise and outliers from the data. This adaptability ensures the reliability and robustness of data processing methods, even when confronted with varying environmental conditions.

This research uses the PointNet++ model to segment point cloud data. PointNet++ architecture is the extension of PointNet created by Qi et al. [28]. PointNet++ introduces a hierarchical set of abstract layers to capture features at different scales [29]. PointNet++ also reaches an important result for classifying and segmenting point cloud data. The applicability of PointNet++ in vegetation analysis is the reason for using this architecture. Point cloud data was collected using TLS at 13 stations across the park, as shown in Figure 2. The data cover most of the park, particularly at the center and south of the park, while the northern area has fewer point clouds.

Figure 2

Scan stations map.

The prepared point cloud data set was split into training, validation, and testing data. This research uses 70% of the data for training, 10% for validation, and 20% for testing. This research uses two strategies, with a random and stratified sampling strategy to analyze the impact of the data set on deep learning performance. The distribution of the data set with different sampling strategies can be seen in Figure 3. Other researchers have used different sampling strategies for the machine learning or deep learning approach. Magar and Barati Farimani [30] used the mean absolute error and proposed two sampling strategies tested with CGCNN architecture, while Liu et al. [31] improved the sampling strategy for few-shot learning to optimize the distribution of data and recognized fine-grained objects in a remote sensing image.

Figure 3

Distribution of data set with different sampling strategies.

In addition to different sampling strategies, this research also divides the data set used in two conditions: a point cloud with and without intensity. Intensity information in point cloud data is useful for solving various problems in object detection and classification [19,32,33,34]. Both conditions were then sampled with two types of sampling, as described before. The ground truth data used was manually annotated using CloudCompare to classify point cloud data into structure and vegetation classes. Each data set, with and without intensity data and with different strategies, was then trained with PointNet++ architecture. This research employed the PointNet++ deep learning architecture for its proven effectiveness in point cloud segmentation tasks. The choice of PointNet++ is grounded in its ability to capture local and global features of point cloud data, making it well-suited for our segmentation objective. This research in total created four models with the same objectives to segment vegetation and structure classes. All four models were then trained with the same hyperparameters as shown in Table 2 to ensure a fair comparison of the models. The selected parameters for our PointNet++ model were determined through a careful optimization process.

A batch size of 16 was chosen to balance computational efficiency and model convergence. It allows for efficient parallel processing during training. The number of epochs was chosen, striking a balance between training duration and convergence. The Adam optimizer was chosen for its adaptive learning rate properties, which can effectively handle the non-stationary nature of point cloud data with the learning rate selected to ensure stable and steady model convergence during training. A decay rate was applied to the learning rate to facilitate fine-tuning and prevent overshooting during training. The research uses 4,096 points per batch and is based on the trade-off between computational efficiency and preserving key features in the point cloud data. The selection of parameters aims to strike a balance between model complexity, training efficiency, and segmentation accuracy, aligning with the specific requirements of the outdoor point cloud segmentation task.

All four models were then trained with 32 epochs, creating a trained model and a training log containing the training accuracy and the training loss score. The training accuracy (Figure 4) shows that all models reach high accuracy, with a final score of ±0.975. The training accuracy increases for each epoch, showing that each model learns the data set provided with different parameters and different sampling strategies. The training loss score decreases for each epoch and gets a score of ±0.052 for each model. The training accuracy and the mean loss score differ for each model but have the same pattern.

Figure 4

Training accuracy and mean loss score in the training process.

To evaluate the segmentation performance, the metrics of the mean intersection of union (mIoU) are used. The mIoU is a common method for evaluating segmentation tasks [35], including semantic segmentation. The IoU is a ratio between the area of the intersection or overlap and the area of union [36]. The formula used to calculate the IoU is shownas follows:

The area of overlap indicates an area of points correctly classified by the models. It has the same label as the ground truth and the predicted segmentation. Meanwhile, the area of union represents a combined area of points, i.e., the prediction, the ground truth, or both. The IoU also can be formulated as a ratio between true positives (TPs) and the sum of TPs, false positives (FPs), and false negatives (FNs). The TPs represent the points segmented correctly, while FPs and FNs represent false segmentation. The formula for the IoU is formulated as follows:

IoU was assessed for each class after segmentation, whereas mIoU was used to summarize the overall performance of segmentation models. It considers the individual IoU values for each class and provides an average score, giving equal importance to each class.

In addition to the IoU, OA is also used to assess the overall performance of the model. Unlike the IoU which focuses on the accuracy of individual classes, OA provides a holistic evaluation of the model’s ability to correctly classify all classes in the data set. OA is calculated as the ratio of correctly segmented points to the total number of points in the entire point cloud data set, as provided by the formula. It provides a single, aggregated percentage reflecting the model’s overall effectiveness in capturing the spatial distribution of different classes. A higher OA indicates a more accurate overall segmentation, making it a valuable metric for evaluating the global performance of segmentation models across diverse data sets.

3 Results

3.1 Semantic segmentation

The semantic segmentation result of outdoor point cloud data shows that deep learning, particularly the PointNet++ architecture used in this research, successfully separates vegetation and structure. With different strategies, this research creates four models and aims to analyze the impact of different sampling methods for creating training data sets. Both sampling strategies show an accurate prediction to segment outdoor point cloud data (Table 3). Vegetation and structure classes can be separated from each other. However, the random sampling strategy fails to predict several structural objects, such as the dome located in the center of the park, while the stratified sampling strategy successfully detects the dome as a structural object.

Table 3

Semantic segmentation result

Ground truth	Prediction
	Random sampling	Stratified sampling
	With intensity
	Without intensity
	With intensity
	Without intensity
	With intensity
	Without intensity

Vegetation classes achieve accurate segmentation results using random sampling strategies, with an IoU score over 0.80, and an even higher IoU score of 0.95 with the stratified approach. Although their distributions of vegetation classes differ, both approaches show accurate results. The difference is in the structure class. The stratified sampling outperforms the random one in the segment structure class. The small objects inside the data set also are well predicted in both strategies, even if they are not smooth or there are no clear boundaries between classes, such as small vegetation inside a pot, a gazebo, or a park lamp. The ground, road, and chair as part of the structure class get accurate predictions into all models, with only a slight difference between them. The random and stratified strategies clearly differ in terms of the dome building. The dome is segmented as vegetation in random strategies but correctly classified as a structure in stratified strategies.

The reason for different results with different sampling strategies is highly related to the training data set used. With stratified sampling, the structural objects receive a proportional distribution, making the deep learning model learn and predict more accurately compared with the random strategy. Figure 3 shows that the structure and vegetation distribution with random strategies are imbalanced. In contrast, both classes obtain a 70% distribution with stratified sampling. This prediction gives information about how impactful the distribution of training used for the model is and how it is related to the final segmentation.

In addition to radiometric information such as RGB color, all point cloud data also stored intensity information that represents materials reflection characteristics [21]. The intensity values can vary because of several factors, such as instrumental and atmospheric effects, the target scattering characteristics, and the measurement geometry [37]. The difference in intensity values between objects can be used as a parameter for deep learning to segment the point cloud data and also gives a difference to the result. For random strategies, the data set with intensity is more accurate than the one without intensity. However, in stratified strategies, the impact of intensity in the data set is unsubstantial. To measure the performance of each model after training, the mIoU score was calculated, as shown in Figure 5.

Figure 5

mIoU score comparison after validation.

The mIoU score aggregates the IoU score per class after validation and summarizes it into a single score. The mIoU score clearly shows that the stratified strategy outperformed the random strategy. The difference between them is also considerable, up to 10%. This result shows that stratified sampling is much more accurate in predicting and segmenting the point cloud data in this research.

The use of intensity in point cloud data also gives different mIoU score results for both strategies. The use of intensity improves the model’s accuracy under the random sampling method but does not have a significant impact under stratified sampling, where both data get nearly the same mIoU score. The mIoU and OA scores are 0.98 and 0.9816, respectively, without the use of intensity, decreasing to 0.9605 and 0.9635 with intensities. The mIoU evaluation shows that the segmentation problem can be resolved with different approaches using different sampling methods and different data set formats.

3.2 Performance evaluation

The model performance was assessed using the IoU score per class, mIoU, and OA after testing. The IoU score for each class with different data sets and strategies is shown in Figure 6. The plot shows that for all classes, the stratified sampling strategy outperforms the random strategy. The difference between both strategies is even up to 15%, showing the superiority of the stratified approach over random sampling.

Figure 6

IoU score comparison after the test.

All classes with all methods receive high IoU scores, over 50% with the lowest one being the vegetation class with random sampling and without intensity. For stratified sampling, all classes even reach a 95% IoU score, with or without intensity data. This result means that even with random sampling and without intensity information, the model still achieves high accuracy. From the IoU score per class comparison, the use of intensity in the data set gives an improvement for the random sampling method but not the stratified method. This result is obtained because, in the stratified method, the model learned the data pattern from the balanced distribution of the data set. Consequently, the use of intensity does not apply to every scenario or every approach. The different distributions of the data sets caused by using different sampling methods improve the significant performance after evaluation. It concludes the high dependency between the deep learning method and the data set used, either for training or validation, or testing. The performance of all models is shown in Figure 7 below.

Figure 7

Performance score comparison.

The IoU score for each class, the mIoU score, and the OA show that the stratified strategy can surpass the random strategy by 10%. However, the impact of intensity inside the point cloud data depends on the strategy. The use of intensity increases model performance by 7% using the random sampling strategy but does not impact model performance using the stratified strategy.

3.3 Intensity analysis

Based on a performance evaluation, the use of intensity in the data set improves model performance with random sampling but not the stratified method. This finding provides an insight into why these data or information is probably not applicable or appropriate for all situations. The information and pattern of intensity data are probably the reasons why intensity can improve model performance with the random sampling method but not the stratified method.

The intensity of the data set shows a varied distribution, even within one vegetation class. According to vegetation intensity, stems or wood always have higher intensity than leaves. This result is similar to the findings of Tao et al. [38], who separated wood–leaf using TLS with the geometric method, where wood also has higher intensity values than leaves. The pattern is the same for all trees or vegetation in the data set, in that stems have higher intensity values. This pattern or intensity distribution between stems and values is a piece of important information and can be used to separate both classes.

In addition to one vegetation type, vegetation species differ in intensity. Based on intensity, there is still confusion about separating intensity values between low- and high-leaf vegetation. This confusion is probably due to the overlapping intensity between the two types of vegetation. Low and high vegetation in some intensity ranges have the same values, either the stems or the leaves, making it difficult to separate the low- and high-leaf vegetation.

The structure class also has a varied intensity distribution among man-made objects. A structure comprises ground and non-ground objects that vary in intensity, where the dome has a higher intensity than the ground. The dome receives the highest structure class, highlighting it and differentiating it from other structural objects. The distribution of intensity for several objects from the vegetation and structure classes is shown in Table 4 below.

Table 4

Intensity distribution

Object	Intensity
Low vegetation	0.001–0.094
Leaves/high vegetation	0.091–0.253
Stems	0.168–0.271
Road (block paving)	0.026–0.139
Rock chair	0.051–0.195
The dome and gazebo	0.194–0.423
Park lights	0.138–0.387

The intensity of structural objects shows a pattern and is affected by various factors, such as laser scanner systems, atmospheric effects, distance and incidence angle, and target surface physical characteristics [39,40,41]. From our point of cloud data, an intensity pattern created between the objects can be seen: the dome, lamp lights, chairs, and finally the road. Each object has a different pattern and distribution of intensity due to different distances, angles, environments, and materials between structures and vegetation with the TLS instrument. An example of the intensity distribution for structure and vegetation classes is shown in Figure 8.

Figure 8

Intensity of structures and vegetation.

Both classes, vegetation and structure, have different intensity patterns, and there are even variations within each class. In the vegetation class, the intensity varies between stems, wood, low vegetation, and high vegetation. These variations are probably caused by the different materials of wood and leaves. However, the difference between low and high vegetation creates assumptions that the intensity also becomes greater with higher objects. Meanwhile, the intensity of structure classes also varies among man-made objects. The variations in structure surface properties and the incidence angle create different intensity patterns [37]. The patterns of structures and vegetation are different and even varied. Structures have a higher intensity pattern than vegetation. The vegetation intensity peaks approximately between 0.05 and 0.15, while the structure is relatively higher at 0.25 to 0.35.

The difference between structure and vegetation becomes a point for separating both classes for segmentation and classification. However, as can be seen from the results, the random sampling strategy failed to segment the dome object, with or without intensity information. The intensity information based on the intensity graph (Figure 9) shows that the vegetation and structure can be separated by the intensity values, but the results of the random sampling method refute this finding. The stratified sampling method is unaffected by these intensity variations.

Figure 9

Graph of intensity of structures and vegetation.

Although the random sampling fails to segment the dome as a structure, the use of intensity does improve model performance. With the intensity value added to the dataset with the random sampling method, model parameters are increased to learn and predict more accurately to segment vegetation and structure. The use of intensity itself is not applicable for all scenarios, proven because stratified sampling does not have an effect whether using intensity or not. The intensity represented by the data is the linear object height, which probably does not contribute greatly to deep learning with stratified sampling.

4 Discussion

According to results and performance evaluation, the stratified sampling method outperforms the random sampling methods. The use of the stratified method improved the accuracy of the model by 10%. With or without intensity data, the stratified sampling is superior to random sampling strategies for both classes, vegetation, and structure.

This result is similar to the findings of another study comparing different sampling strategies for machine learning and deep learning. Mehta et al. [42] compared several data sampling methods, and the result shows that the stratified method outperformed random and simple latent space sampling. Lang et al. [43] also reached a similar conclusion showing the disadvantages of random sampling compared to other methods.

According to the performance evaluation, the average score of the four models created in this research is also reasonable compared with similar research. This research obtains an OA of 0.944, similar to other research using deep learning methods for outdoor segmentation. This OA score is not the highest compared with other research but higher compared with several other methods not using PointNet++. The OA scores of our study and those of other studies are compared in Table 5.

By using the PointNet++ model, this research obtains a higher OA score compared with other research. However, all the OA scores compared are above 90%, meaning the models are accurate for all the tasks. This research also obtains OA scores, with and without intensity data, and with random and stratified sampling methods. The intensity analysis shows a different pattern of intensity within one class and between classes. The use of intensity for prediction with two sampling methods also causes variation in the performances of the models. The use of intensity improves accuracy under random sampling but does not affect the results of the stratified method.

The use of intensity for point cloud classification has limited application scenarios [44]. According to Kaasalainen et al. [45] and Calders et al. [46], the intensity information is affected by many factors, such as the distance and incidence angle, and needs to be calibrated. These factors probably caused inconsistency in intensity data and did not affect deep learning performance in stratified sampling. The use of intensity in some applications still separates wood and leaves with difficulty [20]. Levashev [22], who segmented intensity point cloud data into individual objects, showed similar results. In some cases, the information about intensities does not allow for effective segmentation.

With the limitation of intensity data, several studies switched to using geometric features to accelerate model performance. The geometric features are represented by the point cloud itself and are unaffected by many factors, likewise the intensity. PointNet++ is chosen as the deep learning architecture, creating four models with different sampling strategies, random sampling and stratified sampling, and different data sets, with and without intensity information in the data. A limitation lies in the specificity of our dataset, which was derived from a particular park environment. Future work could benefit from a more diverse set of outdoor environments to assess the robustness of our methodology across varied contexts. Additionally, the performance evaluation focused on segmentation accuracy, and future research could explore additional metrics to provide a more comprehensive understanding of model performance. Despite these limitations, this research lays a foundation for future research to expand and refine outdoor point cloud segmentation methodologies, providing avenues for broader applicability and increased robustness in diverse environmental settings.

In contrast to prior studies, these research results yield distinctive findings that significantly contribute to the field of semantic segmentation in outdoor environments using deep learning. The extensive performance evaluation reveals a noteworthy advancement with the implementation of the stratified sampling method, outperforming random sampling by a substantial accuracy. Remarkably, this superiority persists across both vegetation and structure classes, irrespective of the presence or absence of intensity data. Surprisingly, the study diverges from existing literature by showcasing that the inclusion of intensity information in point cloud classification does not enhance the overall performance of the deep learning model. This finding not only challenges previous assumptions but also underscores the nuanced applicability of intensity data, revealing its limited effectiveness under certain conditions. These nuanced results not only distinguish this research from prior works but also carry implications for the practical implementation of deep learning techniques in outdoor semantic segmentation. The recognition of the superior performance of stratified sampling and the limited utility of intensity information in certain scenarios provide valuable insights that can guide future endeavors in optimizing segmentation accuracy for outdoor digital twin applications.

5 Conclusion

According to the experimental result, it can be concluded that the sampling method used for datasets highly impacts model performance. The use of stratified sampling balances the distribution of both classes. With random sampling, there are possibilities of imbalance between classes, making the model fail to predict accurately. Meanwhile, stratified sampling ensures that both classes have an equal distribution of data sets. The semantic segmentation result showed the stratified method outperforms the random method. Performance evaluation of each model reveals the use of the stratified method boosted the performance of the model by up to 10%. However, all four models obtain a high IoU score per class and an mIoU score >80%, meaning that all models are accurate, but a model with the stratified method obtains the highest performance. Moreover, this research obtains a reasonable and good performance compared with other research.

For the intensity, these data can be useful but are not applicable or suitable for all situations. The performance evaluation shows an improvement in the model when using intensity with random sampling compared to not using intensity. However, the use of intensity with stratified sampling does not affect the performance of the model, meaning that the model still predicts accurately even without intensity information. The intensity analysis reveals that intensity values are distributed in and between classes, making them useful information for separating objects that have highly different intensity values. This nuanced observation provides a practical perspective on the conditional applicability of intensity data in outdoor point cloud segmentation, aligning with these research findings in intensity analysis.

The utilization of Deep Learning algorithms, notably PointNet++, in outdoor environments extends their applicability beyond interiors, showcasing the adaptability of such methodologies in addressing real-world complexities. The demonstrated superiority of the stratified sampling strategy over random subsampling not only optimizes data set quality for geospatial analyses but also holds broader implications for machine learning practices. Additionally, while intensity information within point cloud data may not universally enhance model performance, the insights gained contribute valuable considerations for practitioners, guiding the judicious selection of parameters for more effective model training and segmentation outcomes. These findings collectively advance the field by enhancing segmentation methodologies in outdoor settings and offering practical guidelines for the broader integration of deep learning in geospatial data analytics.