Topic-Feature Lattices Construction and Visualization for Dynamic Topic Number

2.1 The Theory of ILDA

LDA is an unsupervised probabilistic model based on probabilistic latent semantic analysis (PLSA), which can implement implicit topic mining of documents^[11]. The LDA model is a three-layer Bayesian network, where documents can be viewed as discrete topic words, and different topics converge to a limited mixture of topic feature words with probability, which is shown as Figure 1. However, the hyper parameters α and β of this model need to be set in advance, and after many simultaneous, the number of topics K, which is manually set, is related to the granularity of text division. In extreme cases, an excessively large number of topics will merge too many divided text topics. The ILDA model^[12] generates an empty topic for integration, unable to obtain valid topic description information, which cannot meet the actual needs of topic division. Besides, the time-dependent relationship of the text realizes the topic classification under the dynamic number of topics. The model structure is shown in Figure 2.

Figure 1

LDA model structure

Figure 2

ILDA model structure

There are two main differences between the two models in Figures 1 and 2. First, on the basis of LDA, ILDA changes the value of the topic number K to a dynamic variable that can be arbitrarily selected in the interval [1,+]. Second, the document-topic distribution matrix θ in LDA is determined by the Dirichlet distribution of the hyper parameter α polynomial, and θ in ILDA is determined by the joint Dirichlet allocation process (DAP), regardless of the polynomial distribution of the hyper parameter α^[13]. DAP is a prior distribution based on random probability, which can be obtained by polynomial π. The polynomial π is a polynomial mixture that obeys the Griffiths-Engen-McCloskey (GEM) random measure distribution. The detailed calculation process can be found in [14]. The calculation of the base distribution O is shown in Equation (1)^[15]. The advantage of DAP is that its input is not a fixed number, but a discrete variable that changes dynamically. ILDA is a three-layer Bayesian network. By abstracting a document into a polynomial distribution containing K topics, and abstracting a topic into a polynomial distribution containing multiple feature words, it implements joint modeling of documents, feature words, and topics. At the same time, the number of topics depends on the random prior distribution of the hybrid model. It no longer requires that the topic priors of the document must obey the Dirichlet distribution, thereby reducing the sensitivity of the topic model to the number of topics and improving the ability to model large corpora.

2.2 The Theory of Formal Concept Analysis

FCA is a formal method that takes the formal context as its domain, which focuses on describing the hierarchical relationship between concepts^[16]. This theory takes the partial order relationship between formal concepts as the core, and realizes the semi-automatic identification of multi-level ordered concept nodes by establishing the mapping relationship between description objects and attributes^[17]. From the perspective of semantic relationship mining, the concept lattice construction process described by FCA theory can be regarded as the process of hierarchical relationship mining between topic nodes. Meanwhile, the association relationship between the topic concepts is obtained to enhance the semantic relationship between the feature words and the topic.

The mathematical foundation of FCA theory is lattice theory and order theory. The modeling process can be described as follows: First, based on the binary membership between objects and attributes, a ternary context is established including (objects, attributes, relationships). Afterwards, the formal context obtains formal concepts that satisfy the partial order relationship. Finally, a formal concept lattice is established based on whether there is an order relationship between the concepts. In the above process, concept nodes at different levels can reflect different generalization and instantiation relationships between objects, which provide new ideas for obtaining the semantic correlation between topics and feature words.

3.1 Model Construction based on TFL_DTN

The topic modeling of TFL_DTN can be divided into two sub-models: Self-adaptive topic analysis model (STAM) and Topic feature lattice construction model (TFLCM). First, the STAM model assumes that there is probability dependence between documents, topics, and feature words. Each document converges to a topic with a probability, and each topic extracts feature words with a certain probability, thereby forming a three-layer production probability distribution. Among them, the document is a topic polynomial distribution that obeys the Dirichlet distribution process, and the topic is also a feature polynomial that obeys the Dirichlet distribution, which is shared by the document set containing different mixed topic proportions and feature word weights. For the convenience of explanation, the meanings of the variables and parameters in the model are shown in Table 1. The topic analysis process of the STAM model is depicted as follows. First, we use Gibbs sampling to obtain the dynamic optimal number of topics and establish a document-topic probability matrix and topic-feature word probability matrix to extract topics and feature words respectively. Then, candidate feature words with top N high word frequencies from the document-feature word matrix can be selected, on the basis of extracting feature words with higher weights. Finally, the above steps are iterated to obtain the topic with feature words. The STAM model reconstructs the probabilistic dependency relationship between the topic and the feature words based on the ILDA model. In essence, the model does not change the generation of documents, topics and feature words, where the relationship still maps topics and feature words to the same semantic space through the probability selection model. Therefore, STAM can still be regarded as a three-layer Bayes network. The functional dependence of the variables and distribution matrix in the model is shown in Figure 3.

Figure 3

STAM model structure

The TFLCM model assumes that the probability value of the pair of document and topic has a positive correlation with the correlation strength of the pair of topic and feature word. The greater the probability that a document selects a topic is, and the greater the probability that a topic selects a feature word is. By setting the threshold, the strongly related topic features are filtered out and mapped into a context matrix of formal context, and the topic feature lattice is finally generated. The generation process of the TFLCM model is expressed as follows. First, the association probability with the highest probability value from the document-feature word probability matrix is extracted, as well as the topic association matrix, by calculating the feature word association strength under different topics in the document. Afterwards, the feature words with strong correlation are generated, and the correlation matrix of topic formal context is generated. Finally, the generated topic feature lattice is reduced through formal concept analysis. The transformation relationship among matrix variables in the model is shown in Figure 4. Based on the above analysis, the relevant definitions are given as follows.

Figure 4

Variable conversion relationships in TFLCM

Definition 1 (Document-Feature Word Matrix) For any document set D′={d1,d2,⋯,d_m} containing documents M and feature words V, the frequency vector of the feature word sequence contained in d_i can be computed respectively, then the document-feature word matrix for d_i can be represented as Dd={f1d,f2d,⋯,fmd}, where fjd=∑i=1Vtfwi.

Definition 2 (Document-Topic Probability Matrix) For any document d_i, i ∈ {1, 2, · · · ,M}, if topic probability vectors qkzabout topic z is generated, the sampling probability of topic z named as p(z_d|Q_d) can be obtained on the basis of the document-topic probability matrix Qd={q1z,q2z,⋯,qkz} of di.

Definition 3 (Topic-FeatureWord Probability Matrix) For any topic z_i, i ∈ {1, 2, · · · ,N}, if the feature word probability vectors pnwabout the feature word w is generated, the sampling probability p(w_d|P_z) of feature word w in the topic z_i can be obtained on the basis of the topic-feature word probability matrix Pz={p1w,p2w,⋯,pnw}.

Definition 4 (Feature Matrix of Feature Words) Let the dependence of the feature word’s probability w_i on the topic z_i under the number of topics K′ be skz,then the weight matrix of the feature word w_i is S_z(w) = H(z) − H(z|w), where H(z)=−∑i=1np(zd|Qd)logp(zd|Qd),H(z|w)=p(wd|Pz)H(z)+[1−p(wd|Pz)]H(z).

Definition 5 (Topic Association Matrix) Let the association set Ri={r1z,r2z,⋯,rkz}between topic z_i and feature words w_i satisfy the following constraints, then it is called the topic association matrix under the topic z_i. In particular, if riz≥rsz,where s=argmaxi=1⋯k−1(riz−ri+1z),R_i is called a strong association matrix (denoted as SR_i) of z_i, recorded as the feature set C_SRi of all topic associations that satisfy the constraints SR_i in the topic set. Constraint 1. riz=maxdi∈D′qik,qik∈Qd.Constraint 2. For any 1≤i≤j≤k,riz≤rjz.

Definition 6 (Topic Formal Context) Let the topic formal context be F = (T,W, I), where T = {t₁, t₂, · · · , t_K} represents the topic set and I ⊆ T ×W represents the feature word set. I ⊆ T ×W represents the mapping relationship between the topic and the feature set on condition that (t_i, w_j ) ∈ I, t_i ∈ C_SRi.

Definition 7 (Topic Feature Lattice) Let the topic formal context be F = (T,W, I), when A ⊆ T and B ⊆ W, for any two-tuple (A,B) satisfying A^∗ = B and B^∗ = A, (A,B) can be called a set of formal concepts on condition that when A₁⊆ A₂ or B₁⊆ B₂, there is a partial ordering relationship ≺ that makes (A₁,B₁) ≺ (A₂,B₂) be true, where * operation is defined as Equations (1) and (2). The partial order relationship set of all formal concepts in topic formal context F constitutes the topic feature lattice denoted as L(T,W, I).

To sum up, the topic in the TFL_DTN model is a latent variable that depends on a mixture of document-topic polynomials, and the feature words depend on significant variables of the multimodal mixture between (topic, feature word) and (feature word, feature word weight). The core idea of the model is described as follows. Firstly, potential semantic associations among variables can be established, through the probability dependence on documents, topics, feature words and feature word weights. Meanwhile, the Dirichlet stochastic process is viewed as the prior distribution of the Bayes network according to Gibbs sampling. What’s more, the sampling algorithm obtains the number of dynamic topics, and establishes a document-topic probability matrix, a topic-feature word probability matrix, and a feature word weight matrix. Finally, the TFL_DTN model calculates the topic association matrix to filter out strongly related topic features and maps them into a formal context association matrix. To enact this need, a binary partial order relationship between topics and feature words is established to generate a topic feature lattice. The overall structure of the TFL_DTN model is shown in Figure 5.

Figure 5

Overall structure of TFL_DTN model

3.2 Model Reasoning and Parameter Iteration

Since the derivation and parameter estimation of variables and distribution matrices in the TFL_DTN model are mainly handled by the STAM model, the TFLCM model mainly performs secondary filtering and correlation analysis of topic feature words. Therefore, this section mainly discusses the hidden variable z and the matrix P, Q parameter estimation. At the same time, the matrix relationship of the TFLCM model is transformed into the algorithm description in Subsection 3.3.

3.3 Algorithm Description

According to the description mentioned above, the parameter iterative process of the STAM model, as well as the matrix relationship conversion process of the TFLCM model can be described in Algorithm 1.

The proposed algorithm of TFL_DTN can be divided into two sub-models: STAM and TFLCM. STAM starts initializing the number of topic number and generating the matrix Q, P on the basis of Dirichlet distribution as shown from steps 1 to 6. The proposed algorithm then gets the document-feature word matrix by calculating vector of feature word frequency as shown from steps 7 to 10. The model iteratively samples the feature words, and calculates the feature word weight matrix under different topic numbers to obtain the topic-feature word probability matrix and document-topic probability matrix as shown from steps 11 to 21. Consequently, topics, feature words with weights are extracted on the basis of extracting feature words with higher weights as shown from steps 22 to 29. To build the topic feature lattice, TFLCM starts calculating topic association matrix by extracting the association probability with the highest probability value from the document-feature word probability matrix, as well as generating the correlation matrix of topic formal context as shown from steps 30 to 35.

4.1 Preprocessing

We randomly select 1,583,275 online review data of the 20 automobile brand forums from the two websites of Auto Home and Netease Auto, from August 1, 2019 to September 20, 2019. First, the initial document is segmented, and the standard document corpus is obtained by removing data such as stop words, special symbols, and useless tags. Then, the text is converted into a set of review phrases, and a document-word matrix is established. Afterwards, the TF-IDF vector can be calculated to obtain attribute feature words of comment data.

4.2 Results Analysis

4.2.1 Comparison of the Optimal Number of Topics

In order to verify the rationality of the number of dynamic topics in the STAM model, α = 0.1, β = 0.1, γ = 0.2, while the initial topic weight r₀ = 0.5, and the iteration threshold is 0.2. The number of topics in the Baseline method (LDA model) is set to K = 40, whose value is set manually in advance. The STAM model only specifies the number of algorithm iterations (200 and 280 respectively), and determines the number of topics in the document by week.

It can be seen in Figure 6 that although the content of the events described in the corpus is relatively fixed, the number of topics in different periods is dynamically changed, which reflects the correlation between the evolution of topics and the number of topics. In addition, the real data is summarized by the method of manual annotation, and the number of topics varies in the interval [15, 60], which is consistent with the experimental results of the STAM model. At the same time, there is no positive correlation between the document size and the number of topics, but related to the degree of clustering of the actual topics. For example, during the 200 iterations of the STAM model, the document set of the second week contains 847 texts, while the document set for Week 3 is composed of 561 texts. In contrast, the topic number of the former is only 32 but the one of the latter is 49.

Figure 6

Dynamic curve for topic number

In addition, in order to test the capabilities of topic prediction and text representation in STAM model, the perplexity of the above models in the corpus document is calculated. The smaller the value of the perplexity is, the stronger topic prediction capability for the document and it has. The calculation of perplexity is shown as Equation (12). And the experimental results are shown in Figure 7.

Figure 7

Perplexity curve

(12)perplexity=exp{−(∑d∈Dlnp(wd)∑d∈DNd)},

where, p(wd)=∏n=1Nn∑k=1Kp(wd|zk)p(zk|dm).

Figure 7 shows that the perplexity curve of the STAM model is lower than that of the Baseline method as a whole, and the perplexity of the dataset gradually decreases as the number of topics increases. Second, when the number of topics is K = 70, the changeable degree is small, which indicates that the topic distribution under this topic number tends to be stable and the model achieves optimal performance, while the STAM model achieves the best performance when the number of topics is K = 62, indicating that the model has relatively few topics. In that case, the ability to capture the correlation between topics under a dynamic number of topics is stronger, which reduces the model’s dependence on the number of topics K and improves the data representation ability for small sample data sets.

4.2.2 The Construction of Topic Feature Lattice

When the model’s iterative probability threshold is set to 0.01, the document-feature word matrix can be acquired in the corpus. At the same time, when a relatively stable state is reached in the STAM model, both the document-topic probability matrix and topic-feature word probability matrix are output. The top 10 feature words with higher probability are extracted, and their feature word weights are calculated separately. Due to the large number of topics, Table 3 lists only six relatively concentrated topics.

Table 3

Results of topic feature words (Partial)

Topic name	Topic feature words with probabilities (descending order)
Topic 22	brake 0.0923 / sideslip 0.0776 / blind zone 0.0681 / resonance 0.0635 / vision 0.0489 / vehicle warning 0.0437 / stability 0.0332 / weight 0.0332 / vehicle stall 0.0274 / loose parts 0.0165
Topic 8	fuel consumption 0.0851 / cost performance 0.0739 / price 0.0722 / comprehensive performance 0.0696 / per hundred kilometers 0.0696 / high speed 0.0584 / working condition 0.0492 / wind resistance 0.0477 / auto parts zero ratio 0.0368 / idle speed 0.0295
Topic 34	acceleration 0.0654 / maximum torque 0.0554 / power 0.0512 / vehicle climbing 0.0477 / idle speed 0.0461 / engine 0.0313 / turbine 0.0313 / transmission 0.0296 / performance 0.0296 / new energy 0.0212
Topic 68	peculiar smell 0.1284 / ride space 0.0937 / suspension system 0.0735 / NVH 0.0667 / Soundproof 0.0545 / vehicle seat 0.0479 / tire noise 0.0448 / assisted driving 0.0379 / seat ventilation 0.0345 / human-computer interaction 0.0307
Topic 71	maintenance 0.0762 / after-sales service 0.0754 / 4S 0.0694 / engine oil 0.0516 / vehicle failure rate 0.0507 / vehicle inspection 0.0472 / vehicle paint 0.0367 / tire 0.0286 / accessories 0.0286 / working hours 0.0104
Topic 79	vehicle stalled 0.1374 / jitter 0.1238 / steering 0.0863 / clutch 0.0794 / automatic 0.0794 / exhaust 0.0634 / vehicle 0.0432 / gear shift 0.0415 / brake 0.0364

Based on the identification results of topic feature words in Table 3, the content of the topic sets is analyzed manually, which is summarized as the following comment topics: Topic 1 (Topic 22) is security evaluation; Topic 2 (Topic 8) is economy evaluation; Topic 3 (Topic 34) is dynamic performance evaluation; 4 (Topic 68) is comfort evaluation; 5 (Topic 71) is service evaluation; Topic 6 (Topic 79) is manipulative evaluation. The top 10 associated topics of the reviews are listed in Table 4, in which the main characters of security-related topic are No. 13 and No. 22. The feature words contain the topic of the braking, sideslip, blind area, early warning etc. These words are highly associated with vehicle safety of the vehicles, which is strongly aligned with the classification results of manual annotation. In addition, according to algorithm 1, the document set is mined for strongly correlated topic features, and the relational matrix of formal context is established to construct the topic feature lattice. The corresponding part of Hasse structure of topic feature lattice is shown in Figure 8, which shows that the closer the concept is to the top-level root node, the more generalization features of topic words are, such as vehicle length, wheelbase, weight. The term specialization is usually more prominent, such as the acceleration, torque, vehicle power, and vehicle hill climbing associated with node Topic 34. The conclusions show that the topic feature lattice based on the TFLCM model can intuitively find the hierarchical relationships of different topic feature words, with a good modeling ability in obtaining generalization of topic words and semantic relationships.

Figure 8

Hasse diagram of the topic feature lattice (partial)

Table 4

Strongly related topic features (Partial)

Topic category	Strongly related topics (in descending order)
Security evaluation	Topic 22 / Topic 13 / Topic 46 / Topic 7 / Topic 21 / Topic 88 / Topic 74 / Topic 62 / Topic 107 / Topic 95
Economy evaluation	Topic 8 / Topic 11 / Topic 33 / Topic 40 / Topic 56 / Topic 75 / Topic 99 / Topic 7 / Topic 61 / Topic 115
Dynamic performance evaluation	Topic 34 / Topic 124 / Topic 81 / Topic 73 / Topic 84 / Topic 113 / Topic 18 / Topic 22 / Topic 51 / Topic 92
Comfort evaluation	Topic 68 / Topic 16 / Topic 53 / Topic 63 / Topic 77 / Topic 127 / Topic 137 / Topic 12 / Topic 64 / Topic 76
Service evaluation	Topic 71 / Topic 5 / Topic 107 / Topic 143 / Topic 19 / Topic 67 / Topic 55 / Topic 112 / Topic 17 / Topic 35
Manipulative evaluation	Topic 79 / Topic 24 / Topic 61 / Topic 19 / Topic 107 / Topic 46 / Topic 93 / Topic 40 / Topic 122 / Topic 51

4.3 Discussion

In order to verify the rationality of the TFL_DTN model, the accuracy rate, recall rate, F1 value, and mean absolute error (MAE) are selected as the evaluation indicators. Meanwhile, a comparison experiment is performed with the TFIDF algorithm^[18], the TDFCA algorithm^[15], and the ILDA algorithm^[12] on the same data set. Tables 5 and 6 are the comparison results of the evaluation indexes of the above algorithms. The results show that the prediction performance of the TFL_DTN model is significantly better than the other methods on the six review topics. The accuracy, recall and F1 value of the measured data can be maintained around 0.65, and the MAE value can be below 0.85. The reason is that the TFL_DTN model combines the probabilistic relationship and partial order relationship between topic feature words and topics, which not only effectively reduce the dimensionality, but also improve the topic awareness of the document in the changing topic word K.