Can Fuzzy Relational Calculus Bring Complex Issues in Selection of Examiners into Focus?

1 Introduction

It is crucial to identify unbiased experts with similar perceptions in their judgment. Several researchers have studied this important aspect from various viewpoints: Einhorn [11] argued that a consensus between experts is a necessary condition for expertise. Hoffmann [19] used a simple model designed by McGrew [31]. He inferred that consensus in the relevant expert community is an indicator if the following two necessary conditions are fulfilled: reliability and statistical independence. There is no epistemic means to determine the general reliability of genuine moral experts [19]. Due to variability in experts’ judgments, it is essential to scrutinize multiple experts to obtain the cluster having a similarity in perception with increased validity and reliability in evaluation tasks [1, 5, 9–12, 15, 18, 28, 29, 34, 35, 41, 42, 44, 47]. Obtaining a single distribution of elicited information that comprises several experts’ beliefs is desirable [8, 33]. The correlation between the experts’ judgments should be greater, perhaps much greater, when they are judging the same trait than when they are judging different traits [42]. Goldman [17] discussed about how laypersons should evaluate the testimony of experts and decide which of two or more rival experts is most credible. Goldman’s definition of identification criteria for experts [16, 17] has been improved by Scholz [40]. Only the expert has epistemic access to the knowledge of the domain D of expertise (Goldman calls it esoteric knowledge). By contrast, the layperson only has access to exoteric knowledge, i.e. knowledge outside the domain D (Ref. [17], p. 94). The crucial question is: how can a layperson (merely with the help of the exoteric knowledge) identify an expert without having the relevant esoteric knowledge and the cognitive abilities? Ashton [1] suggested an approach to measure the validity of an experts group while adding a new expert in the group. The studies by Ashton and Ashton [2], Libby and Blashfield [27], Makridakis and Winkler [30], and Winkler and Makridakis [49] follow factorial experimental design for experts selection using group validity. This experimental design has more computational time complexity. Hierarchical clustering can be obtained using fuzzy relational calculus, with a factorial experimental design [24, 25, 43, 48, 51].

After the emergence of the fuzzy set theory in the simple task of looking at relations as fuzzy sets in the universe, as accomplished in a celebrated paper by Zadeh [51], he introduced the concept of fuzzy relation, defined as the notion of equivalence, and gave the concept of fuzzy ordering. Several researchers have made seminal contributions and extended the concept of fuzzy relations. Identification of similar experts/examiners in performance evaluation relates to mostly fuzzy classification based on fuzzy similarity relation. The mathematical concept of fuzzy equivalence relation is the basis for fuzzy classification. Although in the literature [3, 4, 6, 7, 20, 22, 26, 32, 38, 45, 46], various approaches for the subjective evaluation of students’ answer script are proposed using fuzzy sets and logic, there are no references for how to evaluate/classify the expertise of experts before they will evaluate the students’ answer scripts. How many teachers are almost similar in their subjective judgment while evaluating students’ answer script? In other words, how many teachers are reliable for a particular evaluation task with a defined possibility level (α-level cut)? The authors have made an attempt to address this issue using fuzzy sets as a basis (part A) with a focus on fuzzy relational calculus to identify similar experts [39]. The study in part A relates to the agreement of teachers based on fuzzy sets at a defined α-level cut using fuzzy similarity measures. It is essential to confirm the similarity among teachers based on actual evaluation of the object (students) as an additional measure.

The paper is organized as follows: Section 2 refers to mathematical preliminaries, while an approach for the selection of teachers for students’ evaluation, based on fuzzy sets theoretic operations and fuzzy relational calculus, is described in Section 3. The case study for selecting fuzzy similarity-based examiner/teacher is covered in Section 4. The results and discussion of the case study are discussed in Section 5. The conclusion and future scope for research are integral parts of Section 6.

2 Preliminaries and Notations

This section briefly describes fuzzy relation/fuzzy relational calculus [52] operations used in the paper.

Definition 2.1: Let U be a universe set. A fuzzy set A of U is defined with a membership μ_A (x)→[0, 1], where μ_A (x), ∀x∈U indicates the degree of x in A [50, 51].

Definition 2.2: Let R be a fuzzy relation on X×Y, i.e. R={((x, y), f_R (x, y))|(x, y)∈X×Y}, the α-cut matrix R_α is denoted by

Rα = {((x, y), fR(x, y))|fR(x, y) = 1, if_fR(x, y) ≥ α;fR(x, y) = 0; if_fR(x, y) < α, (x, y) ∈ X × Y, α ∈ [0, 1]}.

Definition 2.3: Let R⊂X×Y and S⊂Y×Z be fuzzy relations; the max-min composition R∘S is defined by

R ∘ S = {((x, z), maxy{minx,z{fR(x, y), fS(y, z)}})|x ∈ X, y ∈ Y, z ∈ Z}.

Definition 2.4: A fuzzy relation R on X×X is called a fuzzy equivalence relation if the following three conditions held [37, 50, 51]:

1. R is reflexive if f_R (x, x)=1, ∀x∈X.

2. R is symmetric if f_R (x, y)=f_R (y, x), ∀x, y∈X.

3. R is transitive if R⁽²⁾=(R∘R)⊂R, or more explicitly

   fR(x, z) ≥ maxy{{minx,z{fR(x, y), fR(y, z)}}, ∀x, y, z ∈ X.

Definition 2.5: A fuzzy relation R on X×Y is called is a fuzzy compatible or tolerance or proximity relation if it satisfies reflexive and symmetric conditions.

Definition 2.6: The transitive closure, R_T , of a fuzzy relation R is defined as the relation that is transitive, contain R, and has the smallest possible membership grades.

Definition 2.7: Let R be a fuzzy compatibility relation on a finite universal set X with |X|=n, then the max-min transitive closure of R in the relation is defined as the relation R⁽ⁿ⁻¹⁾ [21, 25].

Algorithm A: Find the transitive closure R_T of fuzzy compatibility relation [25].

1. Calculate R⁽²⁾ if R⁽²⁾⊂R or R⁽²⁾=R, then transitive closure R_T =R and stop. Otherwise, k=2, go to step 2.

2. If 2^k≥n−1, then R_T =R⁽ⁿ⁻¹⁾ and stop. Otherwise, calculate R(2k) = R(2k − 1) ∘ R(2k − 1), if R(2k) = R(2k − 1), then transitive closure RT = R(2k) and stop. Otherwise, go to step 3.

3. k=k+1, go to step 2.

Definition 2.8: A partition of S means a family of disjoint subsets, say {S₁, S₂, …, S_n }, such that the union of these subsets coincides with the entire set S. In other words, S₁∪S₂∪…∪S_n and S_i ∩S_j =ϕ, ∀i≠j.

Definition 2.9: The normalized Euclidean distance [23, 26]:

where n is the cardinality of the universe of discourse, A and B are fuzzy sets, and d is a distance measure.

Definition 2.10: Fleiss κ [13, 14] is a statistical measure of inter-rater reliability. κ can be defined as

The factor 1−P̅_e gives the degree of agreement that is attainable above chance, and P̅−P̅_e gives the degree of agreement actually achieved above chance. If the raters agree strongly, then κ=1. If there is no agreement among the raters, then κ<0.

Definition 2.11: The cosine amplitude method is manipulated on a collection of n data samples. The collected data set is represented as X={x₁, x₂, …, x_n }. Each of the n samples is represented as a vector with an m dimension, x_i ={x_i1, x_i2, …, x_im }. The position of each datum in space is represented by m feature values. The relation value r_ij reflects a similarity relationship between x_i and x_j data. For n data samples, the size of the relation matrix will be n×n. The relation matrix always obeys the rules of being reflexive and symmetric, and so it is a tolerance relation. All r_ij values are always in the interval of [0, 1] in this method, and they are calculated through the following equation:

If x_i and x_j are very similar to each other, r_ij becomes close to one. Unlike this situation, if they are very dissimilar to each other, r_ij becomes close to zero.

3 The Proposed Approach

Fuzzy sets as a basis (Part A) have been used to identify similar experts by the authors as a first step in the experts selection process [39]. The study in part A relates to agreement among experts at a defined α-cut level in construction of fuzzy sets. However, it is also equally important to identify the pairwise similarity among experts when they actually evaluate the features of the object. This has prompted the authors to apply one of the facets of other fuzzy relational calculus, which is explained below. Further screening of experts can be done using “Objects Features as a Basis (Part B),” which is proposed below.

4 Case Study

The case study relates the selection of examiners/teachers for evaluating the answer script of students having a similar evaluation perception. The examination answer script samples in subject Marathi language were obtained from 237 secondary school students from three different institutions in Mumbai, India, during academic year 2013–2014. Each student wrote a 10–12-page solution with respect to 12 subjective questions. The questions selected in the question paper can be used to assess students’ writing ability. Answers were evaluated using a 10-point rubric for our study from the Secondary School (SSC) Board, Maharashtra. Also, 20 subject matter experts (teachers) from different schools were identified for the answer scripts evaluation. All 20 evaluators belonging to the “Marathi” department of the SSC board evaluated every 237 answer scripts. This experiment helped us observe the nature of consistency among all experts in the evaluation process. Experts were trained before any evaluation began with respect to the assessment rubric scale and evaluation process. Before the evaluation process begins, each evaluator expressed their evaluation judgment based on their perception for each linguistic variable. Five satisfaction levels, g_i , to award marks scored for the 12 questions, are selected, like very poor, poor, average, good, and very good. Then, all the 20 teachers constructed fuzzy sets for each linguistic variable. The evaluation process for the pilot study occurred in May 2014. All 20 evaluators met together in the central assessment examination room in Marathi Vidyalaya, Mumbai (India), school campus for evaluation. Each teacher was given an answer script to grade. Each teacher awarded a score to every question in numeric value based on the weightage of the question, as shown in Table 1. All teachers reported the students’ performance on a separate result sheet. The evaluators made no marking on the answer script. In the evaluation process, the students’ identities were concealed using masks and identified using unique alphanumeric codes to preserve anonymity. Let

• (Student)_k denote the kth student, where 1≤k≤237;

• T_j denote the jth teacher, where 1≤j≤20;

• Q_i denote the ith question, where 1≤i≤12;

• m(Q_jkn ) denote the marks awarded to question n, by teacher j, of student k; and

• W_i denote the weightage of question i, as shown in Table 1.

The typical computations to follow using the cosine amplitude similarity measure among 20 experts for (Student)₁₂ are summarized below. The typical evaluation score sheet of student roll number 12 evaluated by 20 experts is shown in Table 2, which is normalized as shown in Table 3. A column-wise normalized data sheet is shown in Table 4. These normalized data of students’ marks corresponding to each teacher column is a vector that is used in pairwise cosine amplitude similarity measure computations among all teacher vectors. The similarity index value is in the range [0, 1], where 0 indicates absolute dissimilarity and 1 indicates absolute similarity among teachers in the context of evaluation judgment.

5 Results and Discussion

The procedure detailed in Section 3 was followed, and the results obtained are discussed below:

The relation in Table 3 is reflexive, symmetric but is not transitive and is a fuzzy tolerance relation between 20 experts for student 12. The fuzzy tolerance relation shown in Table 5 is transformed to a fuzzy equivalence relation, as shown in Table 6.

It is necessary to convert the fuzzy equivalence relation presented in Table 6 to a crisp value known as the defuzzification process, as shown in Table 7 at possibility α-value 0.98. Figure 1 shows the partitioning of 20 teachers at various intervals of α-cut for student S₁₂. Cluster {1,2,3,4,5,7,8,9,10,11,12,17,18,19} having the maximum number of experts as shown in Table 8 at possibility α-value 0.98 for student ID 12 is selected.

Figure 1:

Dendrogram on Fuzzy Equivalence Relation [S′12]20 × 20 for Partitioning of Teachers on Different Intervals of α-Cut Using the Cosine-Amplitude Method.

The detailed computational procedure given in Section 3 is performed on all 237 students in order to identify similar experts at the defined α-cut level (α=0.98). The frequency distribution of occurrence for each 20 experts in context with all 237 students is shown in Table 9. The final ranking of all 20 experts at α-cut (α=0.98) using the cosine-amplitude method for all 237 students, ranging from highest to lowest, is given below. A histogram of all teachers is shown in Figure 2.

Figure 2:

Histogram of 20 Experts for Belonging to Partition Having Maximum Cardinality at α-Cut (α=0.98) for All 237 Students.

(T10 < T1 < T2 < T8 < T15 < T14 < T13 < T11 < T18 < T12 < T3 < T17 < T4 < T20 < T5 < T7 < T16 < T19 < T6 < T9)

Educational administrators/policy makers select 11 out of 20 experts using a benchmark of 70% over the experts’ occurrence frequency for all 237 students.

<T10, T1, T2, T8, T15, T14, T13, T11, T18, T12, T3>

are selected for evaluating the students’ academic performance in using the Marathi subject in Maharashtra State Board of Secondary School Certificate Exam, India.

The inter-rater reliability (Fleiss κ coefficient) among all 20 experts, selected 11 experts, and rejected 9 experts are computed for the dataset of 237 students’ evaluation score sheet, as shown in Table 10. The Fleiss κ coefficient for the selected 11 experts is 0.41, indicating moderate agreement among all 11 experts according to Fleiss’ guidelines to interpret the κ statistics. It can be also inferred that for all the 20 teachers as well as rejected 9 teachers, the computed κ coefficient is lower than that of the selected 11 teachers. This additional information will help the decision maker for the final selection of 11 experts. In our view, we can consider the perception of all the 11 experts for aggregating into multiexpert knowledgebase to obtain a fair result of evaluation. Eleven out of 20 experts are selected for future evaluation of the students’ performance of Marathi subject in Maharashtra State Secondary Examination Board, Mumbai, India.

In the recent past, researchers have used various statistical and fuzzy logic-based methods in computing similarity between the experts with no general agreement. In summary, development of the state of the art, especially in expert selection, is a distant dream. In the absence of a standard universal approach, it was considered appropriate to use the authenticated method of κ coefficient for the validation of similar experts having high inter-rater agreement. The κ coefficient is invariably used in medical imaging and other related fields. The expert identification problem to justifiably distinguish between a reliable and an unreliable expert by lay people is highly debated in recent social epistemology [19]. The method for the selection of similar experts presented in this paper by a layperson with only esoteric knowledge can be used in several areas of science and technology in general, and selection of similar examiners in the education system in particular. The software implementation of this approach is done in MATLAB R2008a and Microsoft Access 2010.

6 Concluding Remarks

The fair and unbiased evaluation of students’ academic performance depends on the reliability in experts’ judgments. A combination of the epistemic uncertainty in experts’ subjective judgment and aleatory uncertainty in object’s features is the basis of the formalism presented in this paper. The authors have outline the formalism for the selection of experts based on their subjective judgment using fuzzy relational calculus. The identification of similar experts/examiners in performance evaluation relates to mostly fuzzy classification based on fuzzy similarity relation. The authors propose to extend this concept to large data sets in order to ensure its credibility. The method is somewhat like hierarchical clustering in multivariate data analysis. Are experts similar in their thinking? This is one of the questions that need to be answered while implementing a fuzzy inference system. The authors have looked into this issue and will incorporate the knowledgebase of the identified 11 experts in the fuzzy expert system to infer the performance of students in linguistic terms with a degree of certainty.