Research on image random matrix modeling and stylized rendering algorithm for painting color learning

1 Introduction

In recent years, realistic computer graphics technology has been relatively mature. Gradually, people no longer stick to the real simulation of the traditional real world, but focus on a wider field [1]. Under this background, non-photorealistic rendering (NPR) technology with artistic characteristics came into being and gained unprecedented development and attention. Putting the artistic temperament of NPR’s works in the first place does not demand the authenticity of the treatment effect [2]. Stylized rendering technology is one of the current research hotspots in computer graphics. It is mainly used to simulate the rendering effects of different artistic styles and also used to develop new rendering styles [3]. According to the survey, more than 20 billion photos have been uploaded to Instagram, a famous foreign social networking site. On Snapchat, another application with photo sharing as its main social function, more than 400 million photos are shared by people every day. The intervention of artificial intelligence has also reduced the difficulty and threshold for ordinary people to create artistic works. In the past, artistic creation required certain professional skills and knowledge reserves and often required long-term practice and exploration. However, with the continuous development of artificial intelligence technology, ordinary people can also generate works with specific artistic styles through simple operations and inputs. This allows more people to participate in artistic creation and enjoy the fun and sense of achievement of creation [4].

In multi-layer network architectures, each node symbolizes a feature or abstract concept [5]. The weights connecting these nodes illustrate their interrelations, embedding specific information about an image. For instance, in convolutional neural networks, convolutional layers capture localized image features through convolution operations, whereas fully connected layers integrate these features to represent the image’s overall content and style [6]. By training these models, deep learning techniques enable the extraction and interpretation of image content and style. This capability is particularly evident in style transfer tasks, where such models facilitate the imposition of one artistic style onto another, with adjustable parameters to fine-tune the resemblance in style and content between the original and generated images.

In contrast to 2D videos, 3D videos enhance the realism of visual effects and offer users the autonomy to select their viewing perspective, making it an interactive medium [7]. The applications of 3D video technology extend across digital television, distance learning, remote industrial operations, 3D conferencing systems, and virtual reality platforms [8]. The essence of 3D video technology relies on methods such as image-based modeling and image-based rendering. These methods compute detailed scene models and utilize a few captured viewpoint images, which are then processed through mapping rules for projection and sampling to achieve optimal rendering effects [9,10].

This study introduces a novel approach by constructing a matrix model specifically designed for artistic style rendering to facilitate non-realistic rendering of terrain scenes, diverging from the traditional focus on realistic visualization. It uniquely integrates image structure analysis, particularly emphasizing gradient fields and line features, into augmented reality and image stylization. This innovative application enhances both aesthetic appeal and functional utility, expanding beyond conventional image processing methods. By reviewing and advancing existing techniques for gradient field estimation and line feature extraction, this study not only fills existing gaps but also propels forward the theoretical and practical applications of image structure in digital media, virtual reality, and environmental modeling.

2 Related work

Liao et al. enhanced image stylization quality by analyzing structural information like occlusion relationships and scene hierarchy through user interaction [11]. Similarly, Qiao and Lu developed a method characterized by low computational complexity, multi-scale processing, and rotation invariance in 2002 [12]. To bolster texture extraction robustness, Jiang et al. utilized a multi-resolution analysis, starting with pyramid decomposition to capture images at varying resolutions and subsequently extracting intensity gradient features at each level to gather detailed texture information [13].

Deng et al. introduced a style transfer network capable of effectively executing style migrations in images. However, this network design starts from a noisy image for each stylized rendering, leading to significant computational overhead because the network cannot reuse the trained model for other images, resulting in high time and cost expenditures [14]. Meanwhile, Zhang addressed the issue of contour discontinuities in three-dimensional drawings by re-topologizing to enhance contour continuity [15].

Wei observed that artists often approximate correct contour lines using deliberately “incorrect” lines, which tend to be periodically distributed around the target lines. Their method suggests using multiple renderings to simulate this artistic technique by adding disorder to the lines, although it overlooks the presence of sporadic and brief lines, indicating potential areas for improvement [16]. Zhou et al. developed a hierarchical mapping concept using the non-recursive hierarchical iteration to address terrain fragmentation and enhance real-time rendering capabilities [17]. Chen et al. studied a visual feature model for fine style painting rendering based on computer-aided rendering [18]. This model also exhibits strong artistic and creative qualities in visual features. By using different rendering techniques and algorithms, unique styles and creative visual effects can be generated.

Sheng et al. crafted a painting tool within the gradient domain that allows for comprehensive image editing tasks such as illumination and contrast enhancements with minimal local interactions, diverging from the contour-based diffusion curve approach by enabling direct gradient manipulation on image boundaries [19]. Xu et al. utilized a pre-trained image recognition network to create a loss network, which then trained an image transformation network capable of real-time stylization during the testing phase, often completed in just seconds [20]. Based on the in-depth study of previous literature, this study makes some improvements on the basis of real-time image stylization algorithms. The main work includes the following two aspects:

1. In the real-time image stylization algorithm, although it only takes a few seconds to stylize the image in the testing stage, it often takes several hours to train the model of a certain style.

2. In the application of digital printing, the effect of real-time image stylization algorithm for the pure color background image is not satisfactory. For this problem, this study proposes a method based on foreground–background separation. First, the foreground and background of the image are separated, then the foreground and background of the image are stylized respectively, and finally, combined to obtain a stylized image that meets the production and printing requirements.

3 Methodology

3.1 Overview of image random matrix theory

Image random matrix theory can be used for big data analysis. It is a theoretical tool for processing complex systems and high-dimensional data. The single ring theorem (Ring Law) in random matrix theory has developed rapidly. In some studies about modeling with large-scale data sets, it is pointed out that the problem of simulating random matrix by using single ring theorem can not only accurately describe the distribution of the limit spectrum of the random matrix but also quantitatively analyze its calculation results conveniently, which is an important theoretical basis of this study. Consider the product matrix of l independent non-Hermitian matrices:

(1) Z = ∏ t = 1 L X , u l .

In the formula, X ˜ = { x ˜ i , j } N × T is a non-Hermitian random matrix. Each element in X ˜ satisfies the independent identically distributed condition and has randomness. Its expectation is 0 and the variance is 1, that is, E ( x ˜ i , j ) = 0 , E ( ∣ x ˜ i , j ∣ 2 ) = 1 is satisfied. C is a complex set, and the singular-value equivalent matrix of X ˜ is

(2) X u = X ˜ X ˜ U H .

Among them, U ∈ CN × N is the Harar unitary matrix. By making the variance 1/N, the matrix Z can be per-unitized as follows:

(3) z ˜ i = z i N σ ( z i ) , i = 1 , 2 , ⋯ N .

The linear eigenvalue statistic LSE represents the trace of a random matrix. The law of large number and the central limit theorem both show that the statistical characteristics of a random matrix element can be represented by its linear eigenvalue statistic, but its single eigenvalue cannot reflect this statistical characteristic because of its randomness. Figure 1 shows a 100 × 240 random matrix unicyclic theorem.

Figure 1

100 × 240 random matrix single ring theorem. Source: Created by the authors.

Because the adopted 3D model is a triangular mesh model, discontinuous color transition in the rendering results, the sampling unit selected in the coding part is the triangular ring at the vertex of the model. First, the visible viewpoint list of each vertex is calculated, and the triangle ring is sampled in the visible viewpoint image.

The projection matrix M is calculated. Matrix decomposition is adopted, as shown in equation (3), where g ( r , s ) k is called surface texture and h ( θ , φ ) k is called viewpoint texture:

(4) P v j ( r , s , θ , φ ) ≈ ∑ K = 1 K g k ( r , s ) ⋅ h k ( θ , φ ) = ∑ K = 1 K u k ν k T .

After the matrix decomposition is completed, some classical compression algorithms can be used to further compress the results, such as S3TC. First, the compressed code stream is preprocessed, and operations such as coordinate transformation, projection, and screen mapping are completed. Read the corresponding texture value from the surface texture, then calculate the texture coordinates of the viewpoint in the local coordinate system of three vertices, and read the viewpoint texture from the viewpoint texture h ν j ( θ n , φ n ) . The value of the pixel in the triangular surface can be obtained using formula (5):

(5) P Δ ( r m , s m , θ n , φ n ) = ∑ j = 1 3 g Δ ν j ( r m , s m ) ⋅ h v j ( θ n , φ n ) .

The execution process of this method is shown in Figure 2.

Figure 2

Flow chart of the image quality evaluation method. Source: Created by the authors.

4 Result analysis and discussion

The algorithm in this article is based on a PC platform (intel core (TM) i7-4770, 8gbram, nvidiageforcegtx770) and uses directxn to realize the soft shadow rendering algorithm based on GPU to construct shadow lines, and the range of soft shadow drawn is flexible and controllable. Compared with assarsson, it cannot real-time color the model with high complexity. This algorithm can be used to realize real-time binning rendering, and the effect is better, without the problem of soft shadow aliasing. Compared with the shadow volume improvement algorithms in recent years: PN algorithm and Sintor algorithm. The algorithm in this article not only has a stable dyeing efficiency but also can dye soft shadows with natural transition, and the drawn shadows are more realistic. Because the simplified model is used to generate the shadow lines, the efficiency of the algorithm does not decrease in a broken way when the model with high complexity is stained, as shown in Figure 3.

Figure 3

Line chart of frame rate change of three algorithms. Source: Created by the authors.

Comparison of dyeing efficiency of H-improved shadow volume improved the algorithms, as shown in Table 1.

The shading efficiency of H-improved shadow volume algorithms is related to the complexity of the model. However, because the algorithm in this article adds the steps of simplifying the model, the algorithm can still realize real-time processing when calculating the model with a high number of faces.

The background system will first specify the size of the process pool during initialization. In order to save overhead, not all processes will be created at this time. To execute the rendering task, return a waiting status code to the browser. After receiving the waiting status code, the browser will send a request to the background to ask whether the rendering is completed. Bootstrap is a popular open-source front-end framework that can help developers quickly build responsive, mobile-first websites and web applications. One of the advantages of using the Bootstrap framework is its powerful grid system, which makes page layout more flexible and adaptable to various device screen sizes. This means that the front-end pages of your rendering system can provide a consistent user experience on different devices, whether they are desktop computers, tablets, or mobile devices. In addition, Bootstrap also provides rich CSS and JavaScript components such as navigation bars, buttons, forms, and modal boxes, which can accelerate the development process and improve code maintainability. These components have been optimized and tested to work properly on various browsers and devices, reducing compatibility issues, and rendering buttons and displaying photos after rendering(Table 2).

As the files uploaded by users will have various formats and sizes, in order to ensure the normal operation of the system, we can pre-process the wrong input at the front end to avoid the incorrect input occupying the server network bandwidth and adding unnecessary pressure to the system. In the rendering system, we mainly focus on the file format and file size entered by the user. Since the thread creation and scheduling of OpenMP are randomly allocated by the operating system, uneven load will occur. In the parallelism of the vertex visibility calculation and the sampling module, the number of sampling points of the triangular ring of each vertex is different, and the amount of calculation carried by the two cores cannot be balanced, thus reducing the speedup ratio. When the iteration coefficient is 4, the terrain undulation and grid density are moderate, and the matrix model constructed by it is closest to the general matrix scene of nature, which can be used as the standard model of stylized rendering in this paper. Figure 4 shows a three-dimensional matrix modeling simulation diagram before rendering, in which the iteration coefficient is 4. From the figure, it can be seen that the overall fluctuation of the matrix is moderate, and the hierarchy is clear. It is the standard model for matrix scene rendering.

Figure 4

Iteration coefficient of matrix terrain frame. Source: Created by the authors.

The back-end adopts the now popular Python-based Django framework, which not only has high development efficiency but also greatly reduces it very low, which is convenient for adding or changing functions in the future. The front-end adopts the Bootstrap framework of Twitter, which makes the front-end interface more beautiful and supports mobile devices better. The deployment of the system does not require much configuration and supports multi-user simultaneous rendering. Generally speaking, the function and interface of the whole system are still a little rudimentary, but the expected goal of enabling ordinary users to quickly change the style of pictures has been achieved.

5 Conclusions

This article optimizes the process of uploading images to ensure that users can upload images faster and simpler. It provides a more intuitive display of results, such as using thumbnail or preview functions, allowing users to see rendering style results more clearly. At the same time, it provides more style options, such as color and line thickness, allowing users to adjust the rendering style more flexibly. Second, a local approximation method for discrete unsigned gradient fields is introduced, which can better preserve image features. Finally, a feature profile analysis method based on local fitting is proposed, which can dynamically distinguish sharp and smooth features. Furthermore, this article further demonstrates the improvement of the proposed image structure analysis method for stylized applications of various images and videos. By introducing the Chi-square equation to calculate the light and heavy distribution of lines, the problem of inflexible drawing lines by computer is solved, and the problem of poor simulation of line superposition effect by linear convolution noise algorithm is solved by using multiple rendering technologies. On the premise of knowing the 3D mesh model of the target and the surrounding viewpoint image, the triangle ring of the vertex is sampled to obtain the surface light field information of the target, and the virtual viewpoint is generated by the triangulation method, so as to achieve the purpose of observing and roaming the target under the free viewpoint. In order to accelerate the running speed of the program, the key parts of the algorithm are parallelized and improved. The experimental results show that the average speedup ratio of the parallel part reaches 1.70. In the image quality evaluation database, select a group of lost fine images, and then use this method! The objective evaluation results of fine image quality are in good agreement with the subjective evaluation.

To further enhance image and video stylization technologies, future research could focus on improving the adaptability and precision of the local approximation methods for discrete gradient fields. This would ensure better preservation and depiction of detailed image features across various media. Additionally, integrating advanced machine learning techniques to refine the feature profile analysis could provide more accurate and dynamic feature recognition, significantly enhancing visual quality and the overall user interaction experience with digital content.