Spectral Graph-based Features for Recognition of Handwritten Characters: A Case Study on Handwritten Devanagari Numerals

4.1 Image Pre-processing

Image pre-processing deals with reducing variations on scanned images of handwritten numerals caused by noise. In this study, scanned numeral images are first filtered by difference of Gaussian filtering, then normalisation is applied to handle variability in size, and later numeral images are binarised. Finally, numeral images are skeletonised by a 3 × 3 thinning operator [30].

4.2 Graph Representation

There exist various graph representations [18]; however, we selected interest point graph representation as it preserves inherent structural characteristics of numeral images. It identifies the points in an image where the signal information is rich such as junction points, start and end points, and corner points of circular primitive parts of numerals. Various approaches are proposed for giving interest point graph representations. In this paper, interest point graph representation was inspired by [28, 52]. However, in contrast with [28], the edges in the representation are added based on [52]. Additionally, the orientation point is further added. Figure 4 shows some extracted sample numeral graphs and interest points in each numeral graph.

Figure 4:

Snapshot of Underlying Graphs Obtained from Handwritten Devanagari Numerals with Interest Points (Numerals 0–9).

4.3 Feature Extraction

Weighted graphs include more discriminating information than unweighted such as stretching of the graph [18]. In order to give weights to numeral graphs, edges are labelled with the most well-known and intuitive weighting function w: E(G) → R⁺, which assigns Euclidean distance to each edge in G. Euclidean distance is computed from respective 2D coordinates of nodes incident with each edge e in E (shown in Figure 5A). The motivation behind using such a weighting function is twofold; first, it is computationally simple and, secondly, the distance between any two objects (in this study, nodes) remains unaffected with the inclusion of more objects (nodes) in the analysis [24]. However, there is an arsenal of weighting functions described in the literature [27]; one can use any one of them. As stated earlier, SGE is described in terms of matrices associated with graphs. Selection and extraction of matrices which preserve the underlining structure or topology of the numeral graphs are indispensable. In consideration to this fact, we selected the weighted adjacency matrix (WA(G)), weighted Laplacian matrix (WL(G)) and distance matrix (Dist(G)). These matrices exhibit different topological information (global or local) of graphs which can be crucial for the characterisation of numeral graphs. The adjacency matrix consists of a length of edges, and it is unique for each graph (up to permutation rows and columns) that leads to isomorphism, invariance of graphs. A total number of connected components and spanning trees for a given graph is given by the Laplacian matrix. A number of spanning trees t(G), in a connected graph, is a well-known invariant and leads to many more discriminating properties of the graph. The distance matrix gives the mutual pairwise distance between each node; the matrix thus formed is different for graphs having equal order [1, 17, 53, 57]. Matrix decomposition follows the subsequent representation of these matrices in terms of eigenvalues (with their multiplicities) called spectral decomposition or eigendecomposition of graphs. Let M be some matrix representation of graph G (WA(G), WL(G) and Dist(G)); then the spectral decomposition (or eigendecomposition) is M = ΦΛΦ^T where Λ=diag(λ1,λ2,λ3,…,λ|V|) is the ordered eigenvalues of a diagonal matrix and Φ = (Φ1,Φ2,Φ3,…,Φ|V|) is the ordered eigenvectors as columns in a matrix M. Then the spectrum (eigendecomposition) of M is the set of eigenvalues {λ1,λ2,λ3,…,λ|V|}. For the eigenvalues {λ1,λ2,λ3,…,λ|V|} and the corresponding eigenvectors (Φ1,Φ2,Φ3,…,Φ|V|) Equation (1) holds. The advantage of using a spectrum in characterising a graph is that eigendecomposition of various matrices associated with graphs can be quickly computed (computation of a spectrum from a matrix requires O(n³) operations, where ‘n’ is the order of the graph). Furthermore, the spectral parameters of a graph illustrate/specify various discriminating properties, which otherwise are exponentially computed (chromatic number, sub-graph isomorphism, perturbation of graph, number of paths of length ‘K’ between two nodes, number of connected components in a graph, etc.). Thus, exploiting the spectrum for the graph characterisation is clearly beneficial.

For an illustration of eigendecomposition, let WA(G) = M be the matrix representation of a graph G described in Section 3.

Figure 5:

Illustration of Assigning Weights to Numeral Graphs: (A) Each Node Labelled with 2D Coordinates; (B) Each Edge in the Numeral Graph Labelled (Weighted) with the Euclidean Distance Between Two Adjacent Nodes.

Equation (1) can also be written as

where ‘I’ is the identity matrix, Φ is a special vector (eigenvector) that is in the same direction as MΦ. After multiplying Φ with M, the vector MΦ is a number λ times the actual Φ, called an eigenvalue of M. That means, upon linear transformation M on Φ, λ is an amount of how much vector Φ is elongated or shrunk, reversed or unchanged, which is described by an eigenvalue.

Eigendecomposition of the weighted adjacency matrix WA(G) can be carried out as follows:

Then solving the equation −λ + 140λ³ + 378λ² − 1445λ + 344, we arrive at ordered (dominant) eigenvalues:

Similarly, eigendecomposition is carried out for the weighted Laplacian matrix WL(G) and distance matrix Dist(G). Thereafter, we arrive at feature matrices consisting of ordered (dominant) eigenvalues (spectrum) of WA(G), WL(G), Dist(G), respectively. Furthermore, these features (spectrum) are first inspected individually for characterisation potential, and later they are fused together at decision level (or classifier level fusion) to characterise the numeral graphs.

4.4 Adequacy of the Features

Spectrum inherits different properties (global and local) from their respective graph-associated matrices which make them ideal candidates for recognition purposes; a thorough study can be found in [14, 19, 20, 21]. However, few important properties which are concerned with this study are described as follows:

1. Spectrum is real if the associated graph matrix is real and symmetric. Since, the spectral decomposition map graphs in a coordinate system, any classification or clustering procedures can be used.

2. Spectrum is invariant with respect to labelling of a graph (isomorphic graphs) if sorted either in ascending or descending order because swapping of two columns has no effect on values. Therefore, different orders of the graphs have no influence.

3. Since each eigenvalue contains information about all nodes in a graph so it is possible to use only a certain subset of them. Therefore, it is not mandatory to use all eigenvalues. Imbalanced (short) spectra can be balanced with padding zero values.

4. For disconnected graph G spectrum is the union of the spectra of different components in G.

5.1 Dataset Description and Experimental Setup

For experimentation, we used an isolated handwritten Devanagari numeral dataset from CVPR Unit, ISI Kolkata. It consists of 22,556 samples written by 1049 persons. A total of 368 mail pieces, 274 job application forms, and specially designed forms were used. In a dataset, numerals are with different writing styles, size and stroke widths. The dataset also comprises certain samples that cannot be recognised by humans also. We divided the entire dataset of labelled numeral images into three disjoint sets, namely, training, validation and test sets. The validation set is used to tune/optimise the meta-parameters of the classifier and proposed method. However, the original dataset is divided into training and testing ratios, but the authors of the dataset have stated in [10] that depending upon the requirement, the dataset can be partitioned into training, validation and test sets. Hence, we divided the dataset into two standard ratios of 60:20:20 and 50:25:25 [38] of training, validation and test sets. Figure 1 shows some numeral samples of the dataset. The complete description of the dataset can be found in [9].

Due to its robustness, which is validated from numerous fields of pattern recognition, we employed multi-class support vector machines (SVM) in association with a kernel called Gaussian kernel (also called the radial basis function, RBF-kernel) [25, 34]. There are two possible ways of classification in multi-class SVM: one-vs.-one classifier (IV1) and one-vs.-all classifier (IVA). We have used the one-vs.-one method, as it is insensitive towards an imbalanced dataset. In this method, training is done with all pairs of two-class SVMs (e.g. for 3-class problem, 1 − 3, 2 − 3, 1 − 2), also called pairwise decomposition. All possible pairwise classifiers (n(n − 1)/2) are evaluated and decision for unseen observation is made by majority vote. During training RBF-based SVMs have to optimise two meta-parameters (namely C and Υ, representing classification cost and non-linear function, respectively), empirically on the dataset. To arrive at optimised parameters, values for C and Υ are varied from 0.001 to 10,000 on a logarithmic scale (base-2) (i.e. 0.001, 0.01, …). Each SVM is trained for every possible pair (C, Υ) on the training set and the recognition accuracy is tested on the validation set. Values leading to the best recognition accuracy are then used with an independent test set (Table 1).

Each spectrum (spectra of WA(G), WL(G) and Dist(G)) is investigated individually for recognition potential. From now on, we refer to the spectra of WA(G), WL(G) and Dist(G) as (feature type) FT₁, FT₂ and FT₃, respectively.

The individual recognition results from each feature type are then compared. In order to improve the accuracy of individual classifiers, multi-classifier system (MCS) [34] or classifier fusion is employed. Classifier fusion combines their results by using various combining strategies; however, we used Bayesian fusion (described in Subsection 5.2). It is worth underlining that in MCS, individual classifiers should be accurate and diverse [34]. As stated earlier, the accuracy of SVMs is experimentally validated in a number of practical recognition problems; diversity means each classifier should make different errors or their decision boundaries should be different. In this study, diversity is achieved by using different feature types (as discussed in Subsection 4.3) of the numeral graphs.

5.2 Fusion Technique

We used the Bayesian combination rule (also known as Bayesian belief integration) as a combined technique. It is based on the concept of conditional probability. To compute the conditional probabilities of each classifier for all classes, the confusion matrix has to be calculated first. Let C^l be the confusion matrix for each classifier e_l, with l = 1, … , L, where L is the total number of classifiers used (in this study L=3).

where i, j = 1, … , N, N is the number of classes, and C_i,j in C^l is the total number of samples in which classifier e_l predicted class label j whereas the actual label was i. By using information present in the confusion matrix, the probability that the test sample ‘x’ corresponds to class ‘i’ if the classifier e_l predicts class j can be calculated as follows:

The probability matrix P^l for each classifier e_l is

Based on P^l for each classifier a combined estimate value, b(i) for each class ‘i’ is calculated for each sample ‘x’ in the test set.

For a test sample, ‘x’ classifier e_l predicts class label j_l. To make a decision, one of the class maximum values in b(i) is used.

5.3 Experimental Results

Several experiments were carried out for all three feature types (FT₁, FT₂ and FT₃) and subsequently repeated for 50 random trials of training, validation and testing in the ratios of 60:20:20 and 50:25:25, respectively. In each trial, the performance of the proposed method is assessed by the recognition rate in terms of F-measure, and the average F-measure is computed from all trials. Table 1 gives the class wise performance in terms of F-measure (for both the ratios belonging to all the feature types) and also presents validated meta-values for the RBF-kernel. Figure 6 shows confusion matrices obtained for optimised parameters of the classifier (for each feature type: FT₁, FT₂ and FT₃).

Figure 6:

Confusion Matrices for Each Feature Type (FT₁, FT₂ and FT₃) for Both Divisions, respectively.

The performance of any recognition method is assessed in terms of precision, recall, and F-measure described as follows:

Measures ‘Precision’, ‘Recall’ and ‘F-measure’ are based on correct positive, false negative, false positive, and correct negative for overall samples of the test set.

Table 2 presents the average F-measure computed from all trails. Individually, these feature types (FT₁, FT₂ and FT₃) generate 75–85% average recognition rate. Since FT₃ comprises all the pairwise distances, the shape of the numeral graph is not preserved. Numeral graphs with an equal number of vertices |V| are only distinct in pairwise distances of the vertices but equal in a number of non-zero entries. Perhaps, this could be the reason for its (FT₃) lowest recognition result (75–76%). FT₁ and FT₂ preserve the exact shape of the numeral graphs such as the presence of edges and also their weights; hence they generate over 80% average recognition rates. Since each graph-associated matrix contains non-overlapping information, therefore by combining the classifiers at the decision level greater recognition rates can be achieved. With classifier fusion at the decision level, we achieved the maximum average recognition rate (fusion is carried out individually for each trial and then average recognition accuracy is recorded) of 93.73%, as shown in Table 3. Therefore, by decision fusion at the classifier level recognition rate is increased (FT₁, FT₂ and FT₃) by 7.9%. The numerals which have the same underlying graph structure (more or less) build the misclassified pairs such as Devanagari zero and Devanagari one (as can be observed from Table 1, confusion matrices and Figure 7). Furthermore, the invariance property of the spectrum also adds to the confusion. It can be understood by observing the shape of the Devanagari numeral three and Devanagari numeral six (as shown in Figure 7, just mirror images of each other). As, we sorted the spectrum, therefore, their spectra are more or less equal. In consideration of these facts, recognition performance is encouraging.

Figure 7:

Few Confusing Pairs Such as (A) Devanagari Zero and Devanagari One (More or Less Same Graph Representation) and (B) Devanagari Three and Devanagari Six (Just Mirror Images of Each Other).

It should be noted that each spectrum was sorted in descending order. In order to choose ‘n’ largest eigenvalues for each feature type FT₁, FT₂ and FT₃, we conducted experiments for various values ‘n’ on the validation set. We observe that only the small value of ‘n’ has significant development (n = 3). But when we increase the value of ‘n’ we do not observe much significant development in recognition performance. Thus, in experimentation, we considered the value of ‘n’ equal to 3 for every feature type (FT₁, FT₂ and FT₃). The results obtained after fusion with varying ‘n’ are shown in Table 4.

5.4 Comparative Study

We compared our model with the paper, in which graph representation is used on the same dataset. From the literature, we observe that the authors in [8] achieved a recognition accuracy of 95.85% (in terms of F-measure) by using graph representation and Lipchitz embedding. Lipchitz embedding is based on transforming a graph into ‘n’ distances to already set aside ‘n’ m-dimensional reference sets of graphs, as shown in Figure 8. Each ‘di’ in the feature vector F = (d₁, d₂, … , d_n) is obtained by taking the minimum distance between the input graph ‘g’ and graphs present in each reference set, that is, d_i = min(R₁, R₂, … , R_m), where R₁, R₂, … , R_m are the individual graphs belonging to each reference set. Consequently, a graph ‘g’ is converted to the n-dimensional vector space Rⁿ by computing the graph edit distance (GED) of ‘g’ to all of the ‘n’ reference sets (each m-dimensional). However, transforming numeral graphs into vector spaces by computation of dissimilarities from ‘n’ m-dimensional selected reference sets (carefully selected set of graphs) is time-consuming. The input graph ‘g’ is matched with every single graph in the reference set that requires time complexity in cubic order with respect to the order of the graph (thus inappropriate for graphs having large orders). Furthermore, GED depends on optimisation of various factors, namely, insertion, deletion and substitution cost of nodes and edges. Recognition performance is greatly influenced by the number and dimension of reference sets. Moreover, the type of the graphs selected from the dataset for each reference set also has a great impact on the recognition performance. Our model transforms numeral graphs into vector space by eigendecomposition (or spectrum of a numeral graph as a feature vector) to avoid computationally expensive pairwise graph matching. Besides being powerful in characterising small graphs, they are easy to compute (computational complexity is O(n³), where ‘n’ is the number of nodes present in a graph) and include information about the structure (shape) of the graphs. Furthermore, most misclassification occurs in our model due to the invariance property of the spectrum. Thus the efficacy of the proposed method can easily be justified. Since our model gives graph representation, it is not directly comparable with conventional feature representation models.

Figure 8:

Illustration of the Compared Model.