Visualization of large-scale user association feature data based on a nonlinear dimensionality reduction method

3.1 Construction of NLDR-based LSUAFD visualization platform

In the digital age, the emergence of massive data has brought more challenges of data complexity and multidimensional nature. These data contain immeasurable information value in LSUAFD, but they also pose challenges in data processing and interpretation [17]. This study designs an LSUAFD visualization platform based on a combination of NLDR methods, specifically t-SNE, LargeVis, and PCA. Firstly, PCA performs preliminary processing on the data to reduce its dimensionality to a lower level and lower computational complexity. Next, t-SNE and LargeVis utilize their nonlinear properties to further reduce the dimensionality of the data, aiming to obtain richer structural information. This combination not only effectively reduces the computational burden but also maximizes the preservation of data features. Figure 1 shows the basic architecture of the platform.

Figure 1

Platform architecture design.

The platform adopts B/S architecture, develops backend services based on the Django framework, integrates MySQL database and distributed file storage system, and supports efficient storage and parallel processing of large-scale user data. The front-end part is mainly composed of data display plugins SlickGrid, JS visualization, and calculation modules, which facilitate the intuitive display and fast operation of HDD. The gender similarity coefficients of users u and v are represented by S ( u , v ) . When the user’s gender is consistent, S ( u , v ) = 1 ; when the user’s gender is inconsistent, S ( u , v ) = 0 . The formula for calculating the gender similarity between u and u is shown in Eq. (1).

In Eq. (1), S u represents the gender of u . S v represents the gender of v . The formula for calculating the age similarity S ( u , v ) between u and v is shown in Eq. (2).

In Eq. (2), A u represents the age of u . A v represents the v ’s age. By weighting and synthesizing gender similarity and age similarity, the user’s attribute similarity can be obtained, and its calculation formula is shown in Eq. (3).

In Eq. (3), α represents the proportion of gender. ( 1 − α ) represents the proportion of age, and its value range is [ 0 , 1 ] . Compared with the computational complexity (O(N²)) of Uniform Manifold Approximation and Projection (UMAP) on large-scale data, LargeVis optimizes the time complexity to O(N log N) through the hierarchical clustering tree structure. This is more suitable for processing the user data in this study. In addition, t-SNE has a more obvious advantage over Isometric Mapping (Isomap) because t-SNE effectively alleviates the crowding problem when mapping high-dimensional space to low-dimensional space through t-distribution fitting, thus having more advantages in visualization effect. In the data preprocessing stage, PCA is selected as the initial dimensionality reduction step. This is because PCA can quickly reduce the dimension to 50–100 dimensions while maintaining the global structure of the data, providing efficient input for subsequent NLDR algorithms. Especially on the dataset of this study, the fast processing capability of PCA enables subsequent NLDR algorithms to work more efficiently. Further analysis shows that the efficiency of LargeVis is significantly higher than that of other algorithms. The main reason for its high efficiency is closely related to its graph construction method and the characteristics of the dataset. LargeVis uses a local neighbor graph construction method. Compared with the traditional global graph structure, it can more effectively capture the local structure of the data, thereby significantly improving the computational efficiency. At the same time, the dataset of this study has strong inter-point correlation. LargeVis can better handle these correlations and the sparsity of the dataset, thereby reducing unnecessary complexity during calculations and further improving the overall efficiency. In contrast, other algorithms need to process more computing resources when constructing a global graph, resulting in lower efficiency on large-scale datasets. This indicator will directly affect the user feature mapping effect of the visualization platform. The data dimension reduction module integrates PCA, t-SNE, and LargeVis algorithms, supporting the whole process from HDD to low-dimensional visualization. Figure 2 shows the six core functional modules of the platform.

Figure 2

Functional module design framework.

The platform consists of six main functional modules: user management, file management, statistical analysis, graph analysis, DDR, and high-dimensional visualization. Through this platform, LSUAFD can be directly calculated, thereby visualizing and displaying massive user information. Using user interest feature data to predict ratings can increase data density, which can reflect user similarity. The mathematical expression for the similarity of ratings between users u and v is shown in Eq. (4).

In Eq. (4), Sim sco ( u , v ) is the similarity in scores between u and v . After obtaining user rating similarity and attribute similarity, a weighted synthesis is performed to calculate the comprehensive similarity. The formula is shown in Eq. (5).

In Eq. (5), Sim attr ( u , v ) means the user attributes similarity. Sim sco ( u , v ) is the user ratings similarity. λ , ( 1 − λ ) ∈ [ 0 , 1 ] represents the weight of user attribute characteristics and ratings in user similarity. The larger the weight, the greater the impact on user similarity. After obtaining the similarity and similarity of users, a rating prediction is made for target user u based on the rating data of the target user similarity and the target user rating. Therefore, the calculation for user u 's predicted rating P u , i for product i is shown in Eq. (6).

In Eq. (6), v represents a similar user of user u . S ( u ) is the set of similar neighbors of u . Sim ( u , v ) is the comprehensive similarity between u and v . R v , i represents v 's rating of product i . R v ¯ represents the average rating of v for all products. DDR not only reduces computational complexity and improves algorithm efficiency but also facilitates data visualization and understanding [18]. The basic principle and process of the DDR module are shown in Figure 3.

Figure 3

The principle and basic flow of nonlinear data reduction. (a) Linear dimensionality reduction, (b) nonlinear dimensionality reduction, and (c) nonlinear data dimensionality reduction process.

DDR converts HDD into low-dimensional data while preserving important features and structural information of the original data. DDR is the process of selecting representative features from the original features and constructing new features based on the original features. First, the original HDD is preprocessed, and then, an appropriate dimensionality reduction algorithm is selected, and the corresponding parameters are set. Secondly, by utilizing the selected or extracted features, HDD is mapped to a low-dimensional space. Finally, for the dimensionality-reduced data, machine learning algorithms can be applied, or the internal information of the data can be visualized directly.

3.2 Implementation of NLDR-based LSUAFD visualization platform

In the data-driven modern society, LSUAFD provides researchers with a rich source of information, but it also brings complexity to processing and parsing. Data visualization as an intuitive explanatory tool has enormous potential and challenges in processing such complex data. Therefore, in this study, the LSUAFD visualization platform combines three different dimensionality reduction algorithms: PCA, t-SNE, and LargeVis. Among them, PCA is used for its efficiency in processing large amounts of data, while t-SNE and LargeVis are advanced NLDR algorithms that can accurately reveal the inherent correlations in HDD while maintaining the data structure [19]. Through the comprehensive application and optimization of these three algorithms, different application scenarios and analysis needs can be satisfied. The basic PCA principle is shown in Figure 4.

Figure 4

Basic principles of PCA.

The DDR algorithm is a widely used linear dimensionality reduction technique. It maps the original data into a new coordinate system through coordinate transformation to achieve the goal of reducing data dimensions and preserving key features in the data. The expression for the distance between the original data point and the dimensionality reduced data point x ˆ i is shown in Eq. (7).

In Eq. (7), w i represents the standard orthogonal basis vector. If some coordinates in the newly obtained coordinate system are discarded and the dimension is reduced from d to d ' , the projection of data point x i in the low-dimensional space is z i . z i j = w j T w j represents the j th coordinate of x i in a low-dimensional space. The mathematical expression for the variance of data points in the maximum low-dimensional space based on maximum separability is shown in Eq. (8).

In Eq. (8), W T x i represents the projection of data point x i in a low-dimensional space, which is then solved using the Lagrange multiplier method and eigenvalue decomposition. The DDR algorithm discards the feature vector corresponding to the smallest individual feature value to achieve DDR, achieving a denoising effect to some extent. The t-SNE algorithm belongs to the NLDR algorithm and is suitable for reducing the HDD to 2D or 3D for visual display [20]. The basic principle of the t-SNE algorithm is shown in Figure 5.

Figure 5

Basic principles of t-SNE method.

The t-SNE algorithm constructs a probability distribution between high-dimensional spatial data points, where similar data points correspond to higher probabilities and dissimilar data points correspond to lower probabilities. The calculation formula is shown in Eq. (9).

In Eq. (9), p j ∣ i represents a higher probability for similar data points. σ i represents the confusion level of the main parameters. The calculation formula for the low probability corresponding to dissimilar data points is shown in Eq. (10).

In Eq. (10), p i j represents a lower probability for dissimilar data points. p i ∣ j represents the number of nearest neighbors of high-dimensional and low-dimensional distributions of data points. The goal of the t-SNE algorithm is to calculate the similarity between mappings y i , … , y d , y i ∈ R d ′ , y i , and y j in a low-dimensional space, as shown in Eq. (11).

In Eq. (11), q i j represents the similarity between y i and y j . y i and y j represent mappings in low dimensional space. The similarity between high and low dimensional distributions is measured by KL divergence, and its calculation formula is shown in Eq. (12).

In Eq. (12), C represents the similarity between two dimensions, which minimizes KL divergence through gradient descent. The formula for calculating the conditional probability distribution in low-dimensional space is shown in Eq. (13).

In Eq. (13), f ( x ) = 1 1 + exp ( x 2 ) represents the calculation of the entire dataset. The calculation formula for the optimization objective is shown in Eq. (14).

In Eq. (14), γ represents the weight of the negative sample edge. The LargeVis algorithm has better time complexity than the t-SNE algorithm, providing an effective dimensionality reduction method for the visualization of large-scale HDD. The process of the DDR module based on three different dimensionality reduction algorithms is shown in Figure 6.

Figure 6

Data dimension reduction module process.

The DDR module process shown in Figure 6 mainly consists of three parts: algorithm selection, parameter adjustment, and visualization. PCA, t-SNE, and LargeVis algorithms are used for dimensionality reduction of HDD.

4.1 Module functional testing

The experimental data need to be preprocessed. The KNN interpolation method is used to handle missing values during data cleaning, and outliers are identified and corrected by the IQR method. Z-score normalization is then used to scale the feature values to the [−1, 1] interval. Then, based on the variance threshold method (retaining features with variance >0.5) and correlation coefficient analysis (eliminating redundant features with |r| > 0.9), the MNIST dataset is preprocessed to reduce the dimensions from 784 to 216, and the protein dataset is reduced from 103 to 68. The parameters of the t-SNE algorithm are determined by grid search: the perplexity range is [15,50], the learning rate is [100, 500], and the number of iterations is [500, 2,000]. After cross-validation, the optimal parameter combination for the MNIST dataset is perplexity = 251, learning rate = 0.5, iter = 1,215, at which point the KL divergence is stable at 0.82 ± 0.05. The LargeVis algorithm uses an adaptive learning rate strategy, with the initial learning rate set to 10 and decayed by 5% every 500 iterations, ultimately achieving a dimensionality reduction difference of 0.12 on the protein dataset.

To verify the module functionality of the LSUAFD visualization platform designed for this study, three specific datasets are used for testing and validation. One is the MNIST handwritten digit database, which uses CSV format and contains 784 features and 1 label, with a total of 10,000 data items. The second is the protein structure and function set, which has 103 features and 14 labels, totaling 1,500 data items. The third is the diversity of food nutrition, with 14 features and 1 label, totaling 7,637 pieces of information. Each dataset is used to demonstrate the platform’s ability to process NLDR data. At the same time, the t-SNE algorithm is used in the study, with a perlexity of 30, 1,000 iterations, and a learning rate of 200, to balance local and global information. Regarding the O (N²) computational complexity of t-SNE, the Barnes Hut algorithm is used for approximate calculation, significantly improving processing efficiency. The parameter configuration of the LargeVis algorithm is set to 15 nearest neighbors (K), 0.1 learning rate, and 1,000 iterations (num_iter). These parameters have been optimized to ensure dimensionality reduction quality and improve computational efficiency. The computational complexity of the LargeVis algorithm is O (N log N), making it suitable for processing large-scale datasets. The testing environment used in the experiment is shown in Table 1.

In Table 1, the test environment parameters are Intel Core i5-4210U CPU and 4GB RAM. Its running window is 64-bit Windows 10, the browser is Chrome version 60.0.3112.113, the database is MySQL 5.7.17, and the Python version is 3.5.2. The role of these parameters in executing software and data processing tasks will be further discussed below. The test results of the data density distribution function are shown in Figure 7.

Figure 7

Data density distribution functional testing.

In Figure 7, there is a clear variation pattern between information carrying capacity and data density. In the 0–10 and 60–90 intervals of information carrying capacity, data density shows a very high level, both exceeding the threshold of 1,000. However, when the information carrying capacity is between 10 and 50, the data density is lower than the density values in other intervals, all below 400. This indicates that the volatility of data density highlights the impact of information carrying capacity on data generation. The results of box plot distributed functional analysis tests on odd-numbered datasets, such as the MNIST handwritten digit database, and even-numbered datasets, such as protein structure and function datasets, are shown in Figure 8.

Figure 8

Box-chart-distributed functional analysis tests on different data sets. (a) Test of box chart distribution function analysis on odd numbered data sets, (b) box chart distribution function analysis test for even data sets.

In Figure 8(a), the platform performs well in analyzing odd datasets and can accurately identify outliers in the dataset. The error is mainly distributed between 0.4 and −0.4, indicating that the interpretation of the distribution is quite accurate. In Figure 8(b), for even-numbered datasets, the platform’s analytical capabilities are relatively weak. The distribution of errors is relatively uneven, and the errors are mainly between 0.5 and −0.3. The user satisfaction with the visualization platform is shown in Figure 9.

Figure 9

User satisfaction with visual platforms.

In Figure 9, there is a clear correlation between the increase in user usage of the visualization platform and their satisfaction. As users use visualization platforms more frequently, the number of individuals with a positive satisfaction index remains in the majority. However, the number of users with a negative satisfaction index shows a gradually decreasing trend. In the specific satisfaction index, the index of users who are satisfied with the visualization platform is generally 0.29. The index of users who are dissatisfied with the visualization platform is mostly −0.32. The problems or difficult-to-understand and operate functions encountered by these users during the use of the platform have led to a decrease in satisfaction. This visualization platform reflects the relationship between the platform’s effectiveness and user satisfaction, providing valuable information for future improvement and optimization.

4.2 DDR module testing

In the LSUAFD visualization platform, the DDR module implements PCA, t-SNE, and LargeVis algorithms, to verify the platform’s dimensionality reduction capability by applying these dimensionality reduction methods to three datasets. The data after dimensionality reduction are presented in a 2D view. Testing the protein structure and function dataset to ensure that the platform has the ability to process various data types for dimensionality reduction. The dimensionality reduction algorithm and data parameters are shown in Table 2.

In Table 2, the PCA algorithm demonstrates superior dimensionality reduction performance in food nutrition data, with a variance retention rate of 99.1%. However, the three DDR methods are not ideal for dimensionality reduction of protein datasets. The t-SNE algorithm and LargeVis algorithm highlight the non-linear properties of the data. The LargeVis algorithm significantly improves time efficiency. The dimensionality reduction results of the protein structure and function dataset are shown in Figure 10.

Figure 10

Protein structure function dataset dimension reduction presentation. (a) OCA algorithm, (b) t-SNE algorithm, and (c) LargeVis algorithm.

Figure 10 shows the effects of three different algorithms on the dimensionality reduction of protein structure function datasets. The horizontal axis represents the normalized data distribution range, from 0 to 1. The vertical axis represents the difference degree, which is used to measure the structural difference between the data after dimensionality reduction and the original data. The smaller the value, the closer the dimensionality reduction effect is to the structure of the original data. The LargeVis algorithm performs better among these algorithms. Its dimensionality reduction effect is more complete and clear, with a difference of only 0.12, showing a higher accuracy. However, both the PCA and t-SNE algorithms have shortcomings in dimensionality reduction. In Figure 10(a), the dimensionality reduction of the PCA algorithm is not complete, with a difference of 0.53. In Figure 10(b), the dimensionality reduction result of the t-SNE algorithm is missing, with a difference of 0.34. Therefore, LargeVis’s superior performance in dimensionality reduction, especially in maintaining data integrity and clarity, is particularly evident. The results of the scatter matrix graph for group and categories dimension data are shown in Figure 11.

Figure 11

Scatter matrix graph under four-dimensional data. (a) Group the dimensionality reduction effect of the data set, (b) group visualization of the data set, (c) visualization of the calcium data set, and (d) dimensionality reduction effect of calcium data set.

In Figure 11, there is a small difference in dimensionality reduction and visualization effects between the group and categories dimensions of the data. In Figure 11(a) and (d), the group dataset shows a denser distribution density after dimensionality reduction, reaching 0.63. The scales dataset is relatively sparse after dimensionality reduction, but has rich information, with a distribution density of up to 0.78. In Figure 11(b) and (c), the visualization effect of the categories dataset is better than that of the group dataset, achieving a visualization accuracy of 89%. Therefore, the platform has good dimensionality reduction and visualization capabilities when processing different types of datasets. The performance evaluation comparison of dimensionality reduction algorithms is shown in Table 3.

As shown in Table 3, accuracy refers to the proportion of original label information retained in the data after dimensionality reduction, which is a reflection of the ability to maintain structure. The F1 score comprehensively measures the precision and recall of outlier detection. The higher the value, the better the detection effect. Efficiency is evaluated by the time (in seconds) required to process the MNIST dataset (containing 10,000 samples). The adaptability score is based on the algorithm’s generalization ability to different data types (linear/nonlinear) and sample sizes (1–100k), and is rated on a scale of 1–10. The proposed algorithm achieves an accuracy of 97.38%, significantly higher than LargeVis’s 94.87%, PaCMA’s 96.62%, ISOMAP’s 84.51%, PCA’s 74.32%, and t-SNE’s 81.76%. In terms of outlier detection capability, the proposed algorithm has an F1 score of 0.91, leading LargeVis with 0.84, t-SNE with 0.79, ISOMAP with 0.75, PCA with 0.69, and PaCMA with 0.88. In the efficiency evaluation, the proposed algorithm performs the best with a processing time of 63.81 s, outperforming LargeVis’s 92.45 s, PaCMA’s 76.34 s, t-SNE’s 297.1 s, ISOMAP’s 235.68 s, and PCA’s 118.2 s. In terms of adaptability score, the proposed algorithm ranks first with a score of 9.3, followed closely by LargeVis and PaCMA with scores of 8.8 and 7.6, respectively. ISOMAP, t-SNE, and PCA have adaptability scores of 8.5, 5.9, and 7.3, respectively. These results indicate that the proposed algorithm demonstrates excellent performance in multiple key performance indicators, particularly in terms of accuracy, outlier detection, and processing efficiency. To verify the performance of UMAP in dimensionality reduction tasks, this study conducts experimental comparisons among UMAP, and PCA, t-SNE, and LargeVis. By using the MNIST handwritten digit dataset, protein structure function dataset, and food nutrition component dataset, the accuracy, running time, and completeness of the data distribution after dimensionality reduction of each algorithm are evaluated. The experimental results are shown in Table 4.

Table 4 shows that UMAP outperforms t-SNE in terms of runtime and data distribution integrity after dimensionality reduction, and performs similarly to LargeVis, but slightly inferior to the optimization algorithm proposed in this study. By combining the characteristics of UMAP, this study further validates the flexibility and diversity of the platform. In the comparative experiment, a total of 30 users are recruited and used different platforms for data visualization tasks. Based on user feedback and experimental data, several indicators are evaluated, as shown in Table 5.

Table 5 shows that the research platform outperforms other tools in terms of user friendliness and interactivity ratings, and significantly reduces data processing time, indicating its advantages in practical use. In addition, in terms of visualization quality, this research platform has also received high ratings, further supporting its effectiveness in complex data visualization. To more comprehensively evaluate the performance of the proposed algorithm and other dimensionality reduction methods, indicators such as recall, precision, and ROC-AUC on the validation set are calculated, as shown in Table 6.

From Table 6, the proposed algorithm achieves a recall of 95.2% on the validation set, indicating that it has a high sensitivity in detecting true outliers. The precision of 96.4% further illustrates the accuracy of the algorithm in identifying outliers, with a very low false positive rate. The ROC-AUC value of 0.983 highlights the excellent ability of the algorithm in distinguishing normal data from abnormal data, which is a near-perfect performance. In contrast, LargeVis achieves a recall of 93.5% and a precision of 94.2% on the validation set, with a ROC-AUC value of 0.971. PaCMA achieves a recall of 94.0%, a precision of 95.0%, and an ROC-AUC value of 0.975. ISOMAP, PCA, and t-SNE also show good performance, with recall and precision ranging from 85.0 to 89.0%, and ROC-AUC values ranging from 0.920 to 0.950. Therefore, the proposed algorithm outperforms these methods in terms of recall, precision, and ROC-AUC, showing its superior performance in anomaly detection and data discrimination.

In this study, based on the evaluation of the performance of the dimensionality reduction algorithm, the sensitivity of parameter selection to the results is further analyzed. In particular, the number of neighbors in the LargeVis algorithm and the learning rate in the t-SNE algorithm are discussed in depth. The experimental results show that the increase in the number of neighbors in the LargeVis algorithm helps to better capture the local structure of the data. However, when the number exceeds 100, the performance improvement is limited, and the calculation time increases significantly, indicating that there is an optimal value for the number of neighbors. For the t-SNE algorithm, the adjustment of the learning rate has a significant impact on the convergence speed of the algorithm and the stability of the results. A moderate learning rate (about 20) can strike a balance between ensuring the convergence speed of the algorithm and the accuracy of the results. In summary, parameter selection has a significant impact on the performance of the algorithm, and reasonable parameter settings are crucial to maintaining algorithm performance and computational efficiency. Therefore, in practical applications, the parameters need to be carefully adjusted and optimized according to the specific dataset and application scenario.

Overall, domain impact analysis reveals the potential applications of visualization platforms in multiple fields such as market analysis, bioinformatics, and social network analysis. In market analysis, the platform helps businesses identify market trends and consumer preferences and optimize market strategies by processing consumer behavior and sales data. In the field of bioinformatics, the platform’s dimensionality reduction display of gene expression and protein interaction networks has promoted a deeper understanding of disease mechanisms and accelerated the discovery of biomarkers. In terms of social network analysis, platforms can reveal the characteristics of network structure and information dissemination paths, providing support for identifying key influential nodes and optimizing information dissemination strategies. Interdisciplinary applications have further expanded the practicality of the platform, such as combining market and social network analysis to provide a more comprehensive marketing perspective. Technological advancements, including algorithm optimization and enhanced computing power, have provided possibilities for processing large-scale datasets, while driving research and practice in related fields.