A Genetic Algorithm Approach for Discovering Tuned Fuzzy Classification Rules with Intra- and Inter-Class Exceptions

1 Introduction

Fuzzy rule-based systems (FRBSs) are linguistic rule-based systems that have been conveniently applied in classification problems [10, 21]. FRBSs are known for their competence to deal with noisy, imprecise, or incomplete information, which is often present in any real-world application. The knowledge base (KB) of FRBSs consists of a rule base (RB) and a data base (DB). RB contains fuzzy classification rules (FCRs) and DB incorporates the linguistic labels and definitions of associated fuzzy sets. The automated learning of RB is accountable for discovering a set of FCRs, and the interpretability or semantic power of DB can be enhanced by tuning it with respect to the type of membership functions (MFs), their parameters, and number of fuzzy labels.

Accuracy and interpretability are the eminent criteria for evaluating the performance of FCRs. Accuracy is the capability of FCRs to closely represent the real-world knowledge and is measured as the ratio of correctly classified instances to the total number of instances. Interpretability of the rule is the capacity to express the behavior of a real system in a comprehensible way. This is related to several factors like the model structure, the number of fuzzy rules, the number of linguistic labels, and the shape of the related fuzzy sets [28]. It is ideal to satisfy both criteria to a high degree; however, with accuracy and interpretability being the conflicting criteria, it is generally not possible. That is why researchers focus on obtaining the best trade-off between accuracy and interpretability [9, 15, 22, 34].

Even an expert of a problem domain cannot exactly identify the most appropriate MFs and number of linguistic terms to reflect a system’s true behavior. Knowing the appropriate fuzzy sets and fuzzy labels is a determining factor in automating fuzzy rule extraction processes. The accuracy and interpretability of the discovered rules are adversely affected due to the inflexibility in the concept of type of fuzzy sets and linguistic terms, which imposes hard restrictions on fuzzy rules [28]. A way to get the best trade-off between the accuracy and interpretability is to tune the fuzzy sets and number of linguistic labels (DB) in advance according to the distribution of data in the training data set available for a system.

Accuracy and interpretability are certainly important, but they are not enough to measure the real interestingness of an RB. A rule is said to be interesting if it represents a knowledge that is not only previously unknown to the users but also contradicts the original beliefs of the users [31, 32]. Although interestingness is a subjective issue, unexpected and rare knowledge is usually considered interesting. Discovery of exceptions is interesting because exceptions are rare conditions that contradict the prior knowledge about the domain, add curiosity, and improve the quality of decision making in those rare circumstances where default rules cease to work. Exceptions have low support, and therefore, it is not possible to discover exceptions using the typical rule discovery methods, as these are accountable for the generality of discovered knowledge, whereas exceptions make rules more specific.

This article proposes a three-phased design to discover fine-tuned FCRs augmented with two kinds of exceptions, namely intra- and inter-class exceptions [39]. Intra-class exceptions identify the unique and interesting features of an object within the class to which it belongs. For example, within the class of birds, there could be unique birds that have exceptional features, i.e. penguin, kiwi, and ostrich are rare non-flying birds. Inter-class exceptions are the rare features that change the class of an object. For instance, all animals that can fly are birds, whereas an animal like bat falls into the category of mammals that can fly as well as give milk. The proposed system discovers the rules that have the flexibility of fuzzy logic and perform well on the metrics of accuracy, interpretability, and interestingness. The first phase is a pre-processing step in which the type of fuzzy sets and number of linguistic labels are tuned by taking into account the mapping between the predicting attributes and the class variable. An overall appropriate DB that reflects the true mapping of the attribute values and the class is selected from a number of combinations of fuzzy sets and number of fuzzy labels. Tuning the parameters of the MFs is another dimension of this work, which can be addressed in the future. The second phase employs a genetic algorithm (GA) to evolve the FCRs. The third phase appends the exceptions to the fuzzy rules.

2 Fuzzy Classification Rules

FRBSs need to learn and update their RB (FCRs) to automate and predict the behavior of a system. An FCR is usually represented in the following form:

If (X1 is A1k) ∧ (X2k is A2k) ∧ … ∧ (Xn is Ank), then Dm.

This rule takes the fuzzy values instead of crisp values for the antecedent parts, the consequent parts, or both. Here x_i is an n-dimensional attribute vector and A_1k, A_2k, …, A_nk are the fuzzy sets that comprise the antecedent part on unit interval [0…1]. Here k varies for an attribute x_i over the number of fuzzy linguistic labels like (small, small–medium, medium, medium–large, and large). Attributes may have either the same value of k or different values. D_m is the class label and comprises the consequent part of the fuzzy rule. The following rule is an example of FCRs:

If(Math_score is high)∧(English_score is high), then Good_student.

This fuzzy approach of rule is important for a human expert to comprehend the classifier decision, for example, in medical diagnosis or safety critical applications. Fuzzy logic allows to use linguistic interpretations in a mathematical framework, it provides a usual means for constructing fuzzy rule-based classification systems that are closer to the human decision-making process [42]. The above rule is definitely a better and natural representation of the way human beings comprehend and reason rather than a crisp rule.

3 Fuzzy Rule Performance Measures

FRBSs are evaluated on the basis of accuracy, interpretability, and interestingness. The measure of accuracy is straightforward, and most of the initial research in FRBSs is focused on improving the accuracy of the models. Interpretability is subjective, rather difficult to define, and recently, the interest of researchers has grown manifold to build interpretable models. According to the double-axis taxonomy proposed in [9], complexity and semantics are two important criteria in measuring the interpretability of an FRBS. Complexity-based interpretability describes the complexity of the obtained RB in terms of the number of rules and the number of conditions per rule. An RB is considered more interpretable if it contains fewer number of rules with fewer number of conditions per rule provided that the performance of RB is preserved to a satisfactory level [23]. Semantic interpretability is related to the DB, and it involves adjusting the shape and parameters of MFs and number of linguistic terms. The use of inappropriately complex/simple fuzzy partitions and large/small number of linguistic labels may deteriorate semantic interpretability. To obtain interpretable linguistic models, the MFs of the associated fuzzy sets must be defined to have the following three properties: (1) The MFs should be complete to cover the universe of discourse of a variable so that every data point belongs to at least one of the fuzzy sets; (2) the MFs should represent a linguistic term with a clear semantic meaning and should be easily distinguishable from the remaining MFs of the corresponding variable; (3) the sum of membership values over a universe of discourse should be near to 1 to guarantee a uniform distribution of the meanings among the elements [28].

Further, the issue of interestingness has also drawn a significant attention from researchers. An accurate rule sometimes represents very obvious and non-interesting facts. Therefore, the rule should be discovered to cover previously unknown facts and exceptional behavior of a system.

4 Rules with Exceptions

Exceptions are often ignored as machine noise in classical rule mining algorithms. Several works dealing with discovery of exceptions exist in literature. A censored production rule (CPR), proposed by Michalski and Winston [22], is a common rule augmented with censor/exception conditions. A CPR can be written in the form

If (x1 op v1j) ∧ (x2 op v2j) ∧ … (xn op vnj), then Dm, unless (x3 op v3j) ∨ (x4 op v4j); γ1:γ2P = (x1 op v1j) ∧ (x2 op v2j) ∧ … ∧ (xn op vnj)C = (x3 op v3j) ∨ (x4 op v4j).

In this rule, P, the premise part, is a conjunction of attribute–value pairs. C may contain a single exception or it may be a disjunction of exceptions. It is imperative to specify that the attributes present in the premise and exception part are mutually exclusive. The “if P then D” part of a CPR holds frequently and the censor part C holds rarely. Therefore, the “unless” operator performs akin to an XOR operator between decision and exception, but it has an expositive aspect that is not commutative. To capture this asymmetry precisely, two parameters associated with this kind of rule structure are defined below:

γ1 = Prob⋅[D|P] = Ω(PD)Ω(P) = |P ∧ D||P|.γ2 = Prob⋅[C|P] = Ω(PC)Ω(P) = |P ∧ C||P|.

In the above equations, Ω(PD) is a subset of events for which both P and D hold, Ω(PC) is subset of events for which both P and C hold, and ΩP is the subset of events for which only P holds. The main constraints on rules with exceptions are that γ₁+γ₂≤1 and γ₁>>γ₂. The following rule is an example of a CPR:

If(X = bird), then fly, unless (X = kiwi OR X = ostrich).

Another popular and renowned formation for discovery of exceptions has been given in [34] in the form of a rule pair as

If Pμ, then D (strong rule).If Pμ ∧ C, then D′ (exception).

In the above-mentioned rule pair, P_μ and C are conjunction of attribute values and D and D′ are decision attribute values. The strong rule is a rule of high generality having high recall and precision. The exceptional feature C affixed with P_μ is accountable to change the divergence of the class D to D′. The second rule in the rule pair covers a few objects in the data set and has low support and high confidence. A rule triplet – consisting of strong rule, reference rule, and exception – has also been suggested in [37].

5 Related Work

A lot of work has been done in attempt to achieve a satisfactory trade-off between accuracy and interpretability. Ishibuchi and colleagues [9, 22, 34] have proposed a GA for rule selection problem that tries to maximize accuracy and minimize the number of rules. In addition, multi-objective evolutionary algorithms have also been presented for rule learning problems. These approaches discover non-dominated Pareto optimal solutions by considering accuracy and interpretability (in terms of number of rules and number of antecedent conditions per rule) as the optimization criteria [15, 16].

FCRs have demonstrated their ability in a wide spectrum of applications in the domain of control [24], modeling [25], and data mining problems [17, 20]. Therefore, there have been many attempts to discover FCRs from real valued data sets [5, 8, 11, 14, 19, 18, 21]. Most of these approaches proposed in the literature for automated discovery of FCRs use pre-defined fuzzy MFs and number of linguistic labels [6, 7, 13, 38]. These may prevent the fuzzy linguistic model from achieving the desired trade-off between interpretability and accuracy.

Although interest has grown in tuning the MFs to help discovering highly accurate fuzzy models without a lot of compromise on the interpretability front, we have come across a limited number of attempts in this domain. Mendes et al. [21] have suggested a co-evolutionary approach to simultaneously evolve a population of fuzzy rules and MFs to get a final outcome of a fuzzy rule set and their associated membership definitions that are well adapted to each other. The limitation of this work is that it pre-defines a trapezoidal MF with three linguistic labels for fuzzification of all the attributes of a data set and only adjusts the parameters of the pre-defined MF through mutation. Another co-evolutionary approach to simultaneously evolve the rules and MFs is suggested in [26, 27]. The limitation of these co-evolutionary approaches is that these have to work with really large and complex search space, have long running time for bulky data sets in terms of the number of either instances or attributes, and are more prone to premature convergence. Proposals for genetic tuning of linguistic systems as a post-processing scheme have been suggested in [1, 2, 28, 29].

Silberschatz and Tuzhilin [31, 32] have addressed the issue of interestingness. They have considered the exceptions as interesting pieces of knowledge that challenge the generally held beliefs. Suzuki and colleagues [12, 33, 35, 36] have done extensive work in the domain of exception discovery. They have classified exceptions in several categories and discovered exceptions in the form of rule pairs and rule triplets. They have addressed the task of dependence modeling, and their algorithm discovered a large number of exceptions, making the discovered knowledge unsuitable for human insight and analysis. A classification algorithm based on evolutionary approach for discovering comprehensible rules with exceptions in the form of CPRs is presented in [3, 30]. A post-processing scheme has been suggested to organize and summarize the rules in the form of “rule+exceptions” framework [40]. A genetic programming-based intelligent miner has also been proposed to mine rules with fuzzy hierarchies with exceptions at every level [4]. Vashishtha et al. have discovered classification rules with intra- and inter-class exceptions. However, their algorithm is designed to work with data sets containing discretized/nominal attributes only [39].

In this article, we extend the idea given in [39] to discover an interesting linguistic RB in the form of FCRs with intra- and inter-class exceptions. We also promulgate tuning of DB in a pre-processing phase to get more accurate and interpretable RB without having to deal with large search spaces, as in the case of the co-evolutionary approaches.

6 The Proposed Fuzzy Tuning and Evolutionary Design

This section describes the proposed design for discovery of FCRs with intra- and inter-class exceptions. The proposed algorithm consists of three phases. The first phase of the algorithm is a pre-processing phase that tunes MFs and number of linguistic terms for the fuzzification process of attributes. The second phase discovers a fuzzy rule set, and in the third phase, intra- and inter-class exceptions are appended to the rules. We have employed a crowding GA to discover FCRs in the intermediate phase of the design. This strategy maintains diversity in the population and avoids the convergence of a traditional GA to the single best solution [39]. The detailed descriptions of each phase are given next.

6.1 Phase 1: Pre-Processing

6.1.1 Normalization

Different data sets have data falling into dissimilar ranges. To simplify the fuzzification process, we applied normalization to place the data in same range (0–1). The following formula is employed for normalization:

A′  =  A − min(A)max(A) − min(A).

6.1.2 Fuzzy Tuning and Fuzzification

Several authors have used pre-defined MFs (i.e. triangular or trapezium) and fixed number of linguistic terms (i.e. small, medium, large in case of three fuzzy partitions) for fuzzifying the data sets. As single pre-defined MF and fixed number of linguistic terms may not produce suitable fuzzy partitions for classification, such approaches compromise the predictive accuracy of the rule set discovered in the subsequent phase. Therefore, we have adjusted the type and number of fuzzy partitions by taking into account the data distribution of attributes with respect to the class attribute. We have considered two fuzzy MFs (triangular and trapezium), and the linguistic terms can vary from two to five. Because the search space is not large, an exhaustive search algorithm, given in Figure 1, is used to find the best combination of MF and the number of linguistic terms for each attribute of a data set. The exhaustive search algorithm computes the gain ratio for all the eight combinations of an attribute and returns the combination for which the fuzzy gain ratio is maximal. Finally, the fuzzification process is carried out using different combinations of MFs and number of linguistic terms for different attributes.

Figure 1:

Fuzzy Tuning and Fuzzification.

The formulas applied for computing membership degrees using triangular and trapezoidal fuzzy sets are given as

f(x; a, b, c) = max(min(x − ab − a, c − xc − b), a)   f(x; p, q, r, s) = max(min(x − pq − p, 1, r − xs − d), p).

Parameters a and c locate the feet of the triangle and b locates the peak for triangular function, whereas parameters p and s locate the feet of the trapezoid and parameters q and r locate the shoulders.

The gain ratio for A^k, the k^th attribute is computed as

GR(Ak) = IG(Ak)SplitInfo(Ak)  SplitInfo(Ak) = − ∑j = 1v|Ajk||Ak|log2(|Ajk||Ak|).

Here, |Ajk| is the cardinality of fuzzy set with respect to the value of the α cut parameter (0.5≤α≤1). |A^k| is the cardinality for all fuzzy set associated with attribute A^k.

IG(Ak) = E − E(Ak)E(Ak) = ∑j = 1v|Ajk||Ak| × H(Ajk)   H(Ajk) = − ∑C = 1mpj(Ajk)log2pj(Ajk),

where pj(Ajk) is the relative frequency of the j^th subset of attribute A^k with respect to class C: (1≤C≤m) and defined as

pj(Ajk) = |Ajk∩C||Ajk|.

For illustration, let us consider an example data set given in Table 1. The data set is already in the normal form. The tuning algorithm given in Figure 1 computes the gain ratio for various combinations of the MFs and the number of linguistic terms, and the results obtained are listed in Table 2. The fuzzified data set (on the basis of obtained combinations of the MFs and number of linguistic terms for various attributes) is given in Table 3.

Table 1

The Example Data Set.

Att1	Att2	Att3	Att4	Att5	Class
0.222	0.11	0.05	0.01	0.55	1
0.1667	0.24	0.04	0.3	0.57	1
0.1111	0.35	0.08	0.2	0.53	1
0.0833	0.92	0.05	0.05	0.59	1
0.1944	0.28	0.8	0.6	0.8	1
0.3055	0.3	0.56	0.8	0.88	1
0.0833	0.4	0.9	0.9	0.87	1
0.1944	0.7	0.52	0.55	0.89	1
0.5666	0.6	0.05	1.0	0.54	2
0.5833	0.8	0.05	0.01	0.56	2
0.7222	0.81	0.08	0.02	0.49	2
0.3333	0.45	0.89	0.03	0.48	2
0.6111	0.56	0.05	0.04	0.59	2
0.3889	0.12	0.32	0.6	0.92	2
0.5555	0.81	0.9	0.61	1.0	2
0.1667	0.67	0.42	0.6	0.89	2
0.6388	0.81	0.0133	0.52	0.87	2
0.53	0.73	0.45	0	0.88	2
0.1944	0.49	0.015	0.56	0.54	2
0.4444	0.63	0.59	0.1	0.9	2
0.7778	0.56	0.01	0.1	0.54	3
0.75	0.9	0.05	0.03	0.55	3
0.9	0.81	0.18	0.08	0.59	3
0.9166	0.9	0.14	0.14	0.6	3
0.8	0.59	0.09	0.89	0.61	3
0.8333	0.92	0.57	0.55	0.99	3
0.9	0.83	0.54	0.54	0.98	3
0.8055	1.0	0.52	0.56	0.89	3
0.94	0.89	0.14	0.89	0.99	3
0.7	0.16	0.55	0.6	0.98	3
0.0222	0.87	0.98	0.58	0.04	3
0.0111	0.95	0.95	0.09	0.06	3

Table 2

Gain Ratios for Various Combinations of MFs and Number of Linguistic Terms.

Linguistic terms	Attribute1		Attribute2		Attribute3		Attribute4		Attribute5
	Triag.	Trap.	Triag.	Trap.	Triag.	Trap.	Triag.	Trap.	Triag.	Tarp.
S, L	0.299	0.376	0.279	0.135	0.063	0.015	0.0	0.169	0.130	0.015
S, M, L	0.399	0.537	0.182	0.163	0.009	0.009	0.066	0.071	0.119	0.119
S, SM, ML, L	0.298	0.298	0.271	0.289	0.090	0.080	0.096	0.129	0.121	0.112
S, SM, M, ML, L	0.362	0.362	0.243	0.243	0.071	0.071	0.127	0.127	0.140	0.139
Tuned MFs and number of linguistic terms

Triang., Triangular; Trap., trapezoidal.

Table 3

Fuzzified Example Data Set.

Sr. no.	Attribute1			Attribute2				Attribute3				Attribute4		Attribute5					Class
	S	M	L	S	SM	ML	L	S	SM	ML	L	S	L	S	SM	M	ML	L	D1	D2	D3
1	1	0	0	0.9	0.1	0	0	0.8	0.2	0	0	1	0	0	0	0.8	0.2	0	1	0	0
2	1	0	0	0	1	0	0	0.84	0.16	0	0	1	0	0	0	0.72	0.28	0	1	0	0
3	1	0	0	0	1	0	0	0.68	0.32	0	0	1	0	0	0	0.88	0.12	0	1	0	0
4	1	0	0	0	0	0	1	0.8	0.2	0	0	1	0	0	0	0.64	0.36	0	1	0	0
5	1	0	0	0	1	0	0	0	0	0.8	0.2	0.6	0.4	0	0	0	0.8	0.2	1	0	0
6	0.56	0.44	0	0	1	0	0	0	0.38	0.62	0	0	1	0	0	0	0.48	0.52	1	0	0
7	1	0	0	0	1	0	0	0	0	0.4	0.6	0	1	0	0	0	0.52	0.48	1	0	0
8	1	0	0	0	0	1	0	0	0.46	0.54	0	0.8	0.2	0	0	0	0.44	0.56	1	0	0
9	0	1	0	0	0	1	0	0.8	0.2	0	0	1	0	0	0	0.84	0.16	0	0	1	0
10	0	1	0	0	0	1	0	0.8	0.2	0	0	1	0	0	0	0.76	0.24	0	0	1	0
11	0	0.22	0.78	0	0	0.9	0.1	0.68	0.32	0	0	1	0	0	0.04	0.96	0	0	0	1	0
12	0.33	0.67	0	0	1	0	0	0	0	0.44	0.56	1	0	0	0.08	0.92	0	0	0	1	0
13	0	1	0	0	0	1	0	0.8	0.2	0	0	1	0	0	0	0.64	0.36	0	0	1	0
14	0	1	0	0.8	0.2	0	0	0	0.86	0.14	0	0.6	0.4	0	0	0	0.32	0.68	0	1	0
15	0	1	0	0	0	0.9	0.1	0	0	0.4	0.6	0.56	0.44	0	0	0	0	1	0	1	0
16	1	0	0	0	0	1	0	0	0.66	0.34	0	0.6	0.4	0	0	0	0.44	0.56	0	1	0
17	0	0.89	0.11	0	0	0.9	0.1	0.94	0.05	0	0	0.92	0.08	0	0	0	0.52	0.48	0	1	0
18	0	1	0	0	0	1	0	0	0.6	0.4	0	0.68	0.32	0	0	0	0.48	0.52	0	1	0
19	1	0	0	0	0.6	0.4	0	0.94	0.06	0	0	0.76	0.24	0	0	0.84	0.16	0	0	1	0
20	0	1	0	0	0	1	0	0	0.32	0.68	0	1	0	0	0	0	0.4	0.6	0	1	0
21	0	0	1	0	0	1	0	0.96	0.04	0	0	1	0	0	0	0.84	0.16	0	0	0	1
22	0	0	1	0	0	0	1	0.8	0.2	0	0	1	0	0	0	0.8	0.2	0	0	0	1
23	0	0	1	0	0	1	0.1	0.28	0.72	0	0	1	0	0	0	0.64	0.36	0	0	0	1
24	0	0	1	0	0	0	1	0.44	0.56	0	0	1	0	0	0	0.6	0.4	0	0	0	1
25	0	0	1	0	0	0.7	0	0.64	0.36	0	0	0	1	0	0	0.56	0.44	0	0	0	1
26	0	0	1	0	0	0	1	0	0.36	0.64	0	0.8	0.2	0	0	0	0.04	0.96	0	0	1
27	0	0	1	0	0	0.1	0.3	0	0.42	0.58	0	0.84	0.16	0	0	0	0.05	0.92	0	0	1
28	0	0	1	0	0	0	1	0	0.46	0.54	0	0.76	0.24	0	0	0	0.44	0.56	0	0	1
29	0	0	1	0	0	0.3	0.9	0.44	0.56	0	0	0	1	0	0	0	0.04	0.96	0	0	1
30	0	0.4	0.6	0.4	0.6	0	0	0	0.4	0.6	0	0.6	0.4	0	0	0	0.08	0.92	0	0	1
31	1	0	0	0	0	0.3	0.7	0	0	0.08	0.92	0.68	0.32	0.84	0.16	0	0	0	0	0	1
32	1	0	0	0	0	0	1	0	0	0.2	0.8	1	0	0.76	0.24	0	0	0	0	0	1

6.2 Phase 2: GA Design to Discover FCRs

6.2.1 Initializing Population

In this step, GA is applied to generate an initial population of FCRs. Each chromosome in the population represents a single rule following the Michigan approach. Each fuzzy rule is denoted by a binary string. An n bit string block signifies n successive linguistic fuzzy variables like “small”, “small–medium”, “medium”, “medium–large”, and “large”. The presence of a linguistic value in the antecedent part is represented by 1 bit, whereas a 0 bit marks the absence of any value. A “don’t care” state indicates the non-existence of an attribute in the rule and is encoded by all 1s or all 0s. The consequent part contains the number of bits equal to the number of class labels, i.e. for a three-class classification problem, the consequent part will have three bits. To assign the chromosome to only one of the class, only one of the bits is set to 1 in the consequent part. Figure 2 shows the encoding scheme and its mapping to the corresponding rules. The encoding scheme maps the chromosomes to rules in the CNF form where there is a conjunction between different attributes and a disjunction within the different values of the same attribute.

Figure 2:

Encoding Scheme.

6.2.2 Fitness Evaluation

In the proposed approach, the precision, recall, and simplicity of a fuzzy rule is considered for computing the fitness of an individual. To compute the precision and recall, we require a mechanism to measure the degree of match between a fuzzy rule and an object from a data set. The process of computing the degree of match between a fuzzy rule r and an object of DB u is based on the max_min operators of the Mamdani model and is the same as that used in [25]. The degree of match for an individual attribute A_i between rule r and object u is given by

mAi(r, u) = {1 if s(Ai) = #,max (min (s(Tik), μTik(μ)) otherwise.

In the above equation, s(A_i) is the binary string for i^th attribute, # is the “don’t care” state, s(Tik) is the k^th bit of the i^th attribute, and μTik(μ) us the membership degree of the corresponding term in k^th fuzzy set. The degree of match between premise P of rule r and object u of the data set is given by

mP(r, u) = mini(mAi(r, u)).

The conclusion of rule r and the class of object u can be directly matched, as there is no need to convert this part of the rule into the fuzzy form. To make the matching process clear, we illustrate the process in Figure 3. From the illustration given in Figure 3, we can say that the premise part of rule r and instance u matches with a degree of 0.7. This degree of match is compared to significance level α. If the value of mP(r, u) is greater than the α cut value and the conclusion part of rule r(c) matches with class label of instance u(c), we can count this as one of the true-positive TP cases.

Figure 3:

The Matching Process.

Now we can define precision and recall of a fuzzy rule at significance level α as follows [41]:

Precision(γ1) = ∑u ∈ Ufα(mP(r, u)) ∧ (r(c) = u(c))∑u ∈ Ufα(mP(r, u)),Recall(γ2) = ∑u ∈ Ufα(mP(r, u)) ∧ (r(c) = u(c))|r(c)|.

Here, f_α is the α cut function with a value >0.5.

#n=No. of conditions present in the antecedent of rule R_i.

Thereafter, fitness has been calculated according to following formula:

Fitness(Ri) = (γ1×γ2)#n

6.2.3 Genetic Operators

New individuals are created from existing ones by employing three primary genetic operators: selection, crossover, and mutation. The essential part of the selection process is to select stochastically better-fit individuals from one population and to create a new population for the next generation. In the present design, we have applied the roulette wheel method for selection. From the selected pair of individuals, two new individuals are generated using one point crossover operator. The crossover is applied with a constraint that the crossover point takes place only at the start of the string blocks. For example, two parents are generating two offspring through crossover at the beginning of fourth blocks:

Parent1: 100: 0000: 1100: 00: 1111: 100Parent2: 001: 1100: 1111: 01: 0000: 001Child1: 100: 0000: 1100: 01: 0000: 100Child2: 001: 1100: 1111: 00: 1111: 001

Afterward, a mutation operator generates a new individual by mutating one of the binary string blocks in the chromosome. Mutation is applied with equal probability on each string block. For example, a mutation on the second block may generate new chromosomes as given below:

Parent:(001: 0000: 0110: 01: 00001: 001)Mutated: (001: 0100: 0110: 01: 00001: 001)

The phenotype interpretation of the crossover and mutation operators is given in Figure 4A and B.

Figure 4:

Genetic Operators. (A) Crossover. (B) Mutation. *# Represents the “don’t care” state.

6.2.4 Computing Crowding Similarity

As we are interested in discovering a rule set and not the single best rule, a GA with crowding technique is employed to maintain the diversity of rules in the population. The crowing GA takes two additional parameters – crowding factor and crowding subpopulation size. A subpopulation of rules of crowding subpopulation size is sampled from the GA rule population through uniform sampling. The similarity of an offspring rule is computed with the worst rule from the subpopulation. This process is repeated for crowing factor times. The offspring rule replaces the worst performing but the most similar rule. The similarity of two rules is measured using the following formula:

Similarity_count = |Iworst ∩ Ioffspring|Iworst ∪ Ioffspring.

Here, I_worst and I_offspring are the examples covered by the worst rule of the subpopulation and the offspring rule, respectively. The more common is the data space covered by the worst and the offspring rules, more similar they are. The crowding algorithm is given in Figure 5. The GA to evolve FCRs is given in Figure 6.

Figure 5:

Crowding Algorithm.

Figure 6:

Algorithm for Evolving FCRs.

6.3 Phase 3: Discovery of Exceptions – Intra- and Inter-Class

A framework for discovery of intra- and inter-class exceptions has been given by Vashishtha et al. [39]. In this section, we extend the framework for discovery of FCRs with intra- and inter-class exceptions. An FCR usually takes the symbolic form

If (x1 is Aik) ∧ (x2 is A2k) ∧ … ∧ (xn is Ank), then Dm; γ1:γ2.

Here, x₁, x₂,…, x_n are linguistic variables and A_1k, A_2k,…, A_nk are linguistic terms. The linguistic variables can take values from different numbers of linguistic terms. These linguistic terms may be defined using different MFs. The rules discovered by any machine learning algorithm must have high generalization power with high precision (γ₁) and recall (γ₂). The definitions of precision and recall for fuzzy rules have already been given in Section 3.

A discovered rule is added to the final pool of the rules only if the precision and recall of the rule exceed the user-defined threshold values for precision (t_p) and recall (t_r). We are calling such rules as default rules. The default rules discovered from the example data set are given in Table 4. The values for t_p and t_r are kept at 0.6 and 0.5, respectively.

Table 4

FCRs Discovered from the Example Data Set.

Rule	FCRs	Precision (γ1)	Recall (γ2)	Accuracy (%)
R1	(A1=small)→D1	0.67	1.0	84.3
R2	(A1=medium)→D2	1	0.75
R3	(A1=large)→D3	0.9	0.83

However, such default rules do not deal with rare situations. The rules with high generality can be augmented with an exception part, making them more accurate and interesting. The outline for discovering intra- and inter-class exceptions for fuzzy rules is given below.

6.3.1 Intra-Class Exceptions

Intra-class exceptions extract the unique or rare features of an object within the class. An FCR augmented with intra-class exceptions is represented as follows:

If (x1 is Aik) ∧ (x2 is A2k) ∧ … ∧ (xn is Ank), then Dmϖ(x4 is A4k) ∧ (x5 is A5k); γ1:γ2:γ3:γ4P = (x1 is Aik) ∧ (x2 is A2k) ∧ … ∧ (xn is Ank)E = (x4 is A4k) ∧ (x5 is A5k).

In the above rule, P signifies the premise part and E represents the exception part. Note that the conditions in the premise and exception parts are fuzzy constructs and need to be mutually exclusive. The symbol ϖ indicates the “with” operator, which is used for augmenting intra-class exceptions to the default rule. The rule states that a few objects of class D_m are very unique and of special interest, with the features x₄ and x₅ taking some rare values. We need to define two extra parameters to capture intra-class exceptions as below:

γ3 = |fα(P ∧ E ∧ Dm)||fα(P ∧ Dm)|,γ4 =  |fα(P ∧ E ∧ Dm)||fα(P ∧ E)|.

The parameter γ₃ calculates the support of an intra-class exception with respect to a default rule. A minimum threshold value t_E for γ₃ needs to be set by a user, and it should be much less compared to the value of γ₂, i.e. γ₃≤t_E and t_E <<γ₂. The parameter γ₄ signifies the precision of a rule with intra-class exception, and the value of γ₄ should always be equal to 1, i.e. γ₄=1. This constraint ensures the uniqueness of the rare object in a data set, i.e. P∧E holds in no other class than D_m. f_α signifies the α-cut function to compute γ₃ and γ₄, and the matching of the fuzzy rule and an object from the data set takes place the same way as described in Section 6.2.2. An FCR discovered with intra-class exceptions for rule R2 in Table 4 is given as

If(A1 = Medium) → Class D2 ϖ(A2 = small):γ1 (1.0):γ2 (0.75):γ3 (0.11):γ4 (1.0).

6.3.2 Inter-Class Exceptions

Inter-class exceptions are able to switch the class of an object. The following representation is used for a rule with its inter-class exceptions:

If (x1 is A1k) ∧ (x2 is A2k) ∧ … ∧ (xn is Ank), then Dm ⊕ (x3 = A3k) ∨ (x6 = A6k) (Dj); γ1:γ2:γ5:γ6.

In the above notation, D_m is the default class predicted by the premise being true. The symbol ⊕ is used as the “unless” operator, which is used for the augmentation of inter-class exceptions to the default rule. A rule with inter-class exceptions is interpreted as the following: if P holds, then the class of an object under consideration is D_m, unless any of the rare circumstances/conditions (E_is) hold true. In case any of the exceptions hold, the class of the object changes to class D_j. The rule is an affirmation, for example, the majority of the highly intelligent students perform well in examinations unless one is very sick. To capture such exceptions (E_is), two additional parameters, γ₅ and γ₆, with some constraints are defined as

γ5 = |fα(P ∧ E ∧ Dj)||fα(P)|   γ6 = |fα(P ∧ E ∧ dj)||fα(P ∧ E)|,

where

γ5 = 1 − γ1; γ6 = 1; γ1 < 1.0.

The two inter-class exceptions discovered for rule 1 of the example data set are given as

If(A1 = small) → D1 ⊕ (A5 = small):(D3) ∨ (A3 = SmallMedium):(D2):γ1(0.67):γ2(1.0):γ5(0.0833, 0.167):γ6.(1.0)

The above rule is interpreted as the following: if A₁ is small, then the class of the object is D₁, unless A₅ is small (the class changes to D₃) or unless A₃ is small_medium (the class changes to D₂). This rule classifies the object into the right category in rare and exceptional circumstances. The discovery of inter-class exceptions can bring a significant improvement in the accuracy of data sets where a number of exceptional conditions are spread over several small disjuncts of a data set. In the case of our example data set, the default rule set classifies 27 instances correctly out of 32 instances, giving a predictive accuracy of 84.3. The rule set augmented with inter-class exception classifies three more instances correctly, improving the accuracy up to 93.7.

The detailed algorithm for the third phase to discover exceptions is given in Figure 7. In this phase, the FCRs discovered in the previous phase are mutated over a number of generations to capture the exceptions. As the attributes occurring in the premise and exception parts are mutually exclusive, the mutation operator in this phase is restricted only to the blocks of attributes that do not figure in the premise part of the pre-discovered FCRs.

The attribute value pairs occurring in the exception part are classified as intra- and inter-class based on the framework given above. If the value of γ₁ computes to <1, i.e. t_p ≤γ₁<1, there is a possibility of inter-class exceptions, whereas if γ₁=1, then there is a likelihood of intra-class exceptions.

Figure 7:

Algorithm to Discover Intra- and Inter-Class Exceptions in the Second Phase.

The flowchart of the proposed system for discovery of FCRs with exceptions is given in Figure 8.

Figure 8:

The Flowchart of the Overall System.

7 Experimental Design and Results

We have already demonstrated the proposed approach on an example data set containing 32 instances, five attributes, and three classes. The approach is further validated on the six data sets obtained from UCI machine learning repository. Table 5 describes the data sets with respect to number of instances, number of attributes, and number of classes.

Each data set was fuzzified on the basis of the optimal combination of fuzzy MF and number of linguistic variables obtained in the pre-processing phase. Subsequently, a crowding GA was used to discover FCRs with the following parameters:

In the third phase of design, mutation has been applied only to the exceptional part of the default rules (excluding the bits corresponding to the attributes appearing in the rule part). The mutation probability for this phase is kept at 0.2 and crossover was not required for this phase. The final rules augmented with intra- and inter-class exceptions for the iris data set are shown in Table 6.

Intra-class exceptions merely discover the unique and interesting objects of a class, but do not improve the accuracy of a rule. Inter-class exceptions, although, may occur in a small and insignificant portion of a data set, yet these improve the performance (accuracy) of a default rule. The improvement in accuracy may be significant in cases there are many exceptional conditions spread over several small disjuncts in a data set. The proposed approach has discovered three FCRs with two intra-class and one inter-class exceptions. The accuracy of FCRs with exceptions increases from 95.07% to 96.13%.

The proposed approach improves the complexity-based interpretability (in terms of the number of rules) because it discovers fewer number of rules. It makes the general rules more specific by adding intra- or inter-class exceptions to the rule. This may slightly increase the number of conditions per rule; however, the rules become more correct and semantically more intuitive. Discovering FCRs augmented with exceptions, we not only capture the general behavior but also confine the rare and interesting facts about the workings of a system.

Results obtained for all the six data sets are summarized in Tables 7 and 8. The predictive accuracy of the FCRs discovered with pre-tuned optimal combinations of MFs and number of linguistic terms is compared to the predictive accuracy of FCRs obtained using pre-defined sets of MFs and number of linguistic terms for fuzzification process. It is clear from these tables that FCRs discovered with pre-tuned MFs and number of linguistic terms give better accuracy. The numbers of rules discovered are also minimal across all the data sets. Thus, the tuning in the pre-processing phase results into the best accuracy and complexity trade-off. The predictive accuracies and number of rules given in these tables are averaged over 20 runs of the GA. The summary of FCRs discovered with exceptions is given in Table 9.

We could discover exceptions in Iris, Breast Cancer, Seed, and Glass data sets, whereas no exceptions could be captured in the Wine and Sensor_reading_4 data sets. There is a slight improvement in predictive accuracy wherever inter-class exceptions were discovered. Moreover, the rules with exceptions are interesting. An RB containing FCRs with exceptions makes a system more fool-proof and infallible in rare and exceptional circumstances.

8 Conclusion

In this article, we have proposed a scheme for discovering an accurate, interpretable, and interesting KB for FRBSs. The DB is tuned in pre-processing phase to have appropriate MFs and number of linguistic labels. Inclusion of appropriate MFs and a number of linguistic terms in DB is the fundamental requirement to discover accurate and interpretable RBs. We have also proposed a procedure for discovering FCRs augmented with intra- and inter-class exceptions. Intra-class exceptions confine the unique and interesting facts about an object, whereas inter-class exceptions hold the rare features that change the class of an object. Discovery of an RB in the proposed form makes it more complete and captures the exceptional behavior of a system pertaining to small disjuncts. The proposed discovery may also prove to be useful for fuzzy control applications in predicting the behavior of a system in rare circumstances. As for future work, the proposed system needs to be applied and tested on the domain of medical diagnosis and fuzzy controllers. In this work, we have only tuned the type of MFs and number of linguistic labels to be used for attributes of a data set. Tuning of the parameter values defining the MFs may further increase the accuracy and interpretability of the KB of FRBSs.