IUPAC/CITAC guide: interlaboratory comparison of categorical characteristics of a substance, material, or object (IUPAC Technical Report)

1.1 Scope and field of application

This Guide is developed for harmonization of interlaboratory comparisons of categorical characteristics of a substance, material, or object. It will be helpful also for a correct interpretation of categorical data on properties of substances, materials and objects, the validation of corresponding methods of characterization (e.g., methods of sensory analysis), the development of reference materials with categorical properties, and similar tasks.

The document is intended for quality control, measurement and testing chemical analytical laboratories, metrologists and analytical chemists, specialists involved in the laboratory accreditation activity, laboratory customers, quality managers, and regulators.

1.2 Terms and definitions

Terms and definitions used in this Guide are sourced from the International Vocabulary of Metrology (VIM), ²³ the ISO Vocabulary of Statistics, ^{54 , 55 , 56} the ISO Quality Measurement Systems – Fundamentals and Vocabulary, ⁵⁷ and the IUPAC Compendium of Terminology in Analytical Chemistry (The Orange Book). ⁵⁸ A definition is given as a phrase that can be substituted in a sentence for the term, following ISO practice. ⁵⁹

The most relevant terms and definitions relating to the categorical characteristics of a substance, material, or object applied in this Guide are given below.

1.3 Symbols

α

risk to reject null hypothesis H 0 when it is true (probability of a Type I error)

β

risk of failure to reject the null hypothesis H 0 when in fact it is not true (probability of a Type II error)

β l

risk of a failure to reject null hypothesis H 0 when in fact it is not true, related to factor Xl, l = 1 or 2

γ_{k 0}

intercept (cutoff point) for category k in the ordinal logistic regression model

γ ₁ to γ _m

logistic regression coefficients (slopes) of components contents c ₁ to c _m

C ˆ B

between(inter)-laboratory component of the sample total variation V ˆ T

C ˆ X l B

component of variation C ˆ B caused by factor Xl, l = 1 or 2

c 1 to c m

measured values of contents of chemical components

d f B

number of degrees of freedom of variation C ˆ B

d f T

number of degrees of freedom of variation V ˆ T

d f W

number of degrees of freedom of variation V ˆ W

d f X l

number of degrees of freedom of variation C ˆ X l B , l = 1 or 2

d f l

number of degrees of freedom of chi-square distribution χ d f l 2 , l = 1 or 2

E

expected value

F k

cumulative theoretical probability of ordinal responses up to the k -th category

F ˆ i j k

sample (observed) cumulative relative frequency of ordinal responses up to category k at the i-th level of factor X1 and j-th level of factor X2

F ˆ i . k

sample cumulative relative frequency of ordinal responses up to category k at level i of factor X 1 ; a dot in a subscript symbol means the index of summation (for averaging) of the corresponding frequencies, e.g., j in F ˆ i . k

F ˆ . j k

sample cumulative relative frequency of ordinal responses up to category k at level j of factor X 2

F ˆ . . k

sample total cumulative relative frequency of ordinal responses up to category k

H 0

null hypothesis

H 1

alternative hypothesis

η ₁ to η _m

logistic regression coefficients (slopes) of components contents c ₁ to c _m , equivalent of −γ ₁ to −γ _m

i

index of a level of factor X1, i = 1, 2, …, I

I

number of levels of factor X1

(i, j)

cell in a cross balanced design

j

index of a level of factor X2, j = 1, 2, …, J

J

number of levels of factor X2

k

index of a category of the responses, k = 1, 2, …, K

K

number of categories of the responses

λ

parameter of non-centrality of a distribution

l

index of a factor Xl, l = 1 or 2

L

estimated likelihood

m

number of considered chemical components

M _full

full regression model with predictors

M _intercept

regression model without predictors, i.e., containing only the intercept

n

number of replicate responses

n

vector of response frequencies by categories n 1 , n 2 , … , n K

n i j k

number (frequency) of responses of category k at i-th level of factor X1 and j-th level of factor X2

n k

frequency of responses of category k

N

total number of responses

P

probability; used also with subscripts related to a property, e.g., properties p1 to p5

P ˆ

sample estimate of P

P joint

probability of a joint event (intersection)

P ˆ joint

sample estimate of P joint

P _l

power of a test for assessing interlaboratory consensus, equal to probability 1 − β l , l = 1 or 2

p

vector of the response probabilities by categories p 1 , p 2 , … , p K

p k

probability of a response of category k

p ˆ

vector of the sample probabilities (relative frequencies) of responses by categories p ˆ . . 1 , p ˆ . . 2 , … , p ˆ . . K

p ˆ i j k

sample probability (relative frequency) of nominal responses of category k at i-th level of factor X1 and j-th level of factor X2

p ˆ i . k

sample probability (relative frequency) of nominal responses of category k at i-th level of factor X1

p ˆ . j k

sample probability (relative frequency) of nominal responses of category k at j-th level of factor X2

p ˆ . . k

sample probability (total relative frequency) of nominal responses of category k

pseudo-R ²

McFadden’s statistics for evaluation of goodness-of-fit of logit models

q

index of a category of the responses (like k), q = 1, 2, …, K

Q _index

quality index

Q index exc

index of a product having imagined excellent quality

S I ˆ X l

significance index (test statistics) for evaluation of effect of factor Xl, l = 1 or 2

S I X l crit

critical value of significance index S I ˆ X l under null hypothesis H 0 , l = 1 or 2

S I ˆ X l MC

significance index generated with the MC method, l = 1 or 2

S I ˆ X l , λ

significance index, shifted/modified under alternative hypothesis H 1 with parameter of non-centrality λ, l = 1 or 2

S I ˆ X l , λ MC

significance index, shifted/modified under alternative hypothesis H 1 , generated with MC method, l = 1 or 2

w

effect of the statistical sample size for the chi-square test

Xl

factor 1 or factor 2 (l = 1 or 2)

x l

critical value of chi-square distribution χ d f l 2 , l = 1 or 2

χ d f l 2

chi-square distribution with d f l degrees of freedom, l = 1 or 2

χ d f l , λ 2

non-central chi-square distribution shifted under alternative hypothesis H 1 with parameter of non-centrality λ, degrees of freedom d f l , and l = 1 or 2

V A R

variance

V ˆ T

sample total variation of the response variable Y , normalized on [0, 1] interval

V ˆ W

within(intra)-laboratory component of V ˆ T caused by factor X2 and/or “residual” variation due to unknown reason(s)

Y

random quantity on a categorical scale, Y = n ; used also with subscripts related to a property, e.g., properties p1 to p5

∩

intersection of events

1.4 Abbreviations

ANOVA

Analysis of Variance

CATANOVA

Categorical (nominal) Analysis of Variation

CDF

Cumulative Distribution Function

CITAC

Cooperation on International Traceability in Analytical Chemistry

exp

exponential function

IBM SPSS

Statistical Package for the Social Sciences of the International Business Machines Corporation

IEC

International Electrotechnical Commission

ISO

International Organization for Standardization

IUPAC

International Union of Pure and Applied Chemistry

logit

logarithmic odds of ordinal responses

MASS

Modern Applied Statistics with S programming environment

MC

Monte Carlo

ORDANOVA

Ordinal Analysis of Variation

PDF

Probability Density Function

PMF

Probability Mass Function

VIM

International Vocabulary of Metrology

2.1 Test items

Choice and preparation of test items having homogeneity and stability of the properties of interest fit for purpose of a planned interlaboratory comparison is a task of the comparison provider. ¹ When test items are consumer products (e.g., samples of a packaged sausage) from different producers, purchased simultaneously from a market for comparison, ⁶⁴ they shall be examined before their expiration dates.

2.2 Responses

2.3 Cross-balanced design

A design of an interlaboratory comparison without replication at any (i, j) cell, when I J = N , is the simplest cross-balanced design. It is shown in Table 1, where n i j k denotes the number of responses obtained in the i-th laboratory at the j-th condition, related to a k-th category. No interaction between the two factors can be analyzed when all n i j k = 1 .

In general, a cross-balanced design may contain (i, j) cells with the same number n > 1 of replicate responses and the total number of responses N = n I J . The design with replication allows testing of the interaction between the factors. ^{25 , 47}

Mathematical issues of random effects of factors for a nominal scale were described recently in ref. ⁶⁶ .

3.1 Total variation

Treating N responses as a statistical sample, and the number of responses n i j k as a random variable, then p ˆ i j k = n i j k / N and F ˆ i j k = ∑ q = 1 k p ˆ i j q denote the sample (observed) relative frequency of responses belonging to the k -th category and the sample cumulative relative frequency of responses up to the k -th category in cell i , j , respectively. The sample total cumulative relative frequency of all responses belonging to the k -th category is denoted by

Here, F ˆ i . k = 1 J ∑ j = 1 J F ˆ i j k i = 1 , 2 , … , I ;   k = 1 , 2 , … , K and F ˆ . j k = 1 I ∑ i = 1 I F ˆ i j k j = 1 , 2 , … , J ;   k = 1 , 2 , … , K denote the sample cumulative relative frequency of responses up to the k -th category at level i of factor X 1 and at level j of factor X 2 , respectively. Dots in a subscript symbol mean the indices of summation (for averaging) of the corresponding frequencies, e.g., i and j in F ˆ . . k .

The observed (sample) total variation of the response variable Y , normalized on the [0, 1] interval, is estimated in two-way ORDANOVA for ordinal variables ⁴⁷ as

with degrees of freedom d f T = N − 1 . A similar estimate in two-way CATANOVA for nominal variables ⁶⁷ is

where p ˆ . . k = n . . k / N is the sample proportion (relative frequency) of data belonging to the k -th category and ∑ k = 1 K p ˆ . . k = 1 .

3.2 Decomposition of the total variation

In the model without replication, the total sample variation V ˆ T is partitioned into the between(inter)-laboratory component C ˆ B and the within(intra)-laboratory component V ˆ W , caused by the second factor and/or “residual” variation due to unknown reason(s). For ordinal data, ⁴⁷ this is

where

and

The degrees of freedom of the variation components are d f B = I − 1 + J − 1 and d f W = I − 1 J − 1 , respectively.

The individual effects of factors X 1 and X 2 can be estimated using the next decomposition of the variation C ˆ B :

where

with degrees of freedom d f X 1 = I − 1 and d f X 2 = J − 1 , respectively.

A similar decomposition for nominal data ⁶⁷ leads to

Such decomposition may include a component related to the possible interaction between the two factors. In addition, decomposition by response categories was discussed in papers. ^{25 , 26 , 27} Note also that the sample estimators by Eqs. (3–10) are biased from the corresponding population variations. ^{42 , 68}

3.3 Null and alternative hypotheses

The null hypothesis H 0 of homogeneity of the responses states that the probability of classifying the responses as belonging to the k -th category does not depend on the levels of the first factor (levels i ) nor on those of the second factor (levels j ), i.e., p i j k = p k for all i = 1 , 2 , … , I and j = 1 , 2 , … , J . Under this hypothesis, the following relations are applicable for both nominal and ordinal data:

where E is the expected value. The numerator of the last term in Eq. (11) is the population total variation V T corresponding to the probability vector p = p 1 , p 2 , … , p K . The alternative hypotheses H 1 are that one or both the studied factors influence the probability vector p , i.e.,

To test the statistical significance of both the factor effects, the following significance indices (test statistics) have been defined: ⁴⁷

3.4 Hypothesis testing for nominal variables

Distributions of the statistics d f l   S I ˆ X l , l = 1, 2, for nominal variables are asymptotically approximated by the chi-square distributions χ d f l 2 ²⁵ with d f 1 = K − 1 I − 1 and d f 2 = K − 1 J − 1 , respectively. They have the following expectations and variances:

This approximation allows the application of a chi-square test for testing the null and alternative hypotheses. ⁶⁹ The null hypothesis H 0 regarding the equivalence of the levels of factor X 1   p i . k = p k , i.e., insignificance of the effect of factor X 1 on the response variable Y , is rejected when d f 1   S I ˆ X 1 exceeds the critical value x 1 of the chi-square distribution χ d f 1 2 at the 1 − α   100   % level of confidence, i.e., when the probability P d f 1   S I ˆ X 1 > x 1 = α . It is the probability of a Type I error, which may be interpreted as the α -risk of a false decision that a consensus of laboratories is absent, when it is actually achieved. Similarly, the H 0 regarding the levels of factor X 2   p . j k = p k is rejected when d f 2   S I ˆ X 2 exceeds the critical value x 2 of the chi-square distribution χ d f 2 2 . This also means that the null hypothesis H 0 related to factor X l is rejected when S I ˆ X l exceeds x l / d f l at the 1 − α   100   % level of confidence. The α -risk here is the probability of a false decision of the significance of the influence of factor X l on the responses, when it is insignificant.

The alternative hypothesis H 1 by Eq. (11) corresponds to the shifted/modified distribution of the statistics d f l   S I ˆ X l which would be valid under the null hypothesis H 0 . The modified distribution is denoted further as d f l   S I ˆ X l , λ , where λ is the parameter of non-centrality, i.e., the shift in the distribution. The following expectations and variances related to the modified distribution are

and

This modified distribution is approximated by the noncentral chi-square distribution χ d f l , λ 2 ⁷⁰ . The values are calculated as λ = w 2 N , where w is the effect of the statistical sample size on the chi-square test. Conventionally, a value of w = 0.1 is considered as a small effect, w = 0.3 – a medium effect, and w = 0.5 – a large effect. ⁷¹ As the sample size N = I J is equal for both factors X 1 and X 2 , the same λ is applicable.

Then, values of the power of the homogeneity test of the responses at different levels of the factor X 1 (levels i ) and factor X 2 (levels j ) can be calculated as the power P _l , l = 1 or 2, of the corresponding chi-square test: ^{72 , 73}

where CDF means cumulative distribution function, and β l denotes the probability of a Type II error. It may be interpreted in the case of factor X 1 as the β -risk of a false decision of a consensus of the laboratories, when the consensus was not achieved. If the influence of factor X 2 is tested, this is the β -risk of a false decision of the factor insignificance, when it is significant.

An example of the power values versus numbers of the factor levels and categories, calculated and plotted in R programming environment, ^{48 , 74} is available in Annex A, Example 1. An evaluation of the consensus (the power and β -risk at the set α -risk) of laboratories which participated in a comparison of weld imperfections that are nominal weld characteristics, is in Annex A, Example 2.

3.5 Hypothesis testing for ordinal variables

Testing the null hypothesis H 0 on the effect significance for ordinal variables also requires knowledge of an asymptotical distribution for the indices S I ˆ X 1 and S I ˆ X 2 by Eq. (13), in order to calculate the critical values of the indices S I X 1 crit and S I X 2 crit at the 1 − α   100   % level of confidence.

A calculation tool using random MC draws from a multinomial distribution – an Excel spreadsheet with macros ⁷⁵ – calculates (from the empirical data) the sample vector of relative frequencies p ˆ = p ˆ . . 1 , p ˆ . . 2 , … , p ˆ . . K , as well as the variation components ( C ˆ X 1 B , C ˆ X 2 B , V ˆ W , V ˆ T ) and the values of the indices S I ˆ X 1 and S I ˆ X 2 . At each iteration, the calculator performs random draws from the multinomial distribution with K categories and the vector of relative frequencies p ˆ , and stores the calculated values of the significance indices. Finally, for each significance index, an empirical CDF is constructed and relative frequency (%) plots of the simulated values (empirical distributions of S I ˆ X l MC , l = 1 , 2 ) are displayed. The critical values S I X l crit for the significance indices, as an equivalent of x l / d f l for nominal variables, are recovered as the points, where the 1 − α   100   % level of confidence of the empirical CDF is achieved. The null hypothesis H 0 is rejected when the significance index S I ˆ X l MC exceeds the critical value S I X l crit at the 1 − α   100   % level of confidence.

The alternative hypothesis H 1 is represented by the shifted/modified empirical distribution of the significance index S I ˆ X l , λ MC = 1 + λ / d f l S I ˆ X l MC . Thus, the power value P l of the criterion for testing homogeneity of the responses at different levels of the factor X l is P l = 1 − CDF S I ˆ X l , λ MC S I X l c r i t .

The tool was applied, for example, for evaluation of the consensus of laboratories which participated in a comparison of intensity of odors (ordinal characteristics) of different drinking water samples. ²⁶ This is Example 3 in Annex A.

4.1 Sensory evaluation and chemical composition

Categorical characteristics of a substance, material, or object may be correlated with its quantitative characteristics, such as contents of main chemical components and impurities. For example, relationships between sensory evaluation and the chemical composition of meat and meat products have been a subject of research. ^{76 , 77 , 78} Regression analysis is the known tool for studying and modeling such relationships. However, as in applications of ANOVA, the additivity assumption should not be violated when applying regression analysis. ⁷⁹ This is possible with multinomial ordered logistic regression (ordered logit), quite commonly applied in medicine, ⁸⁰ financial activity, ⁸¹ in a study of consumer purchasing behavior, ⁸² and in other fields.

4.2 Multinomial ordered logistic regression

The ordered logit model is based on the following concepts. When Y is an ordinal outcome with K categories, the cumulative probability of the responses of categories q, which are less than or equal to a category k, is P q ≤ k . The odds of the responses being less than or equal to a category k, are defined as P q ≤ k / P q > k , when k = 1 , … , K − 1 . For k = K , the denominator is zero and the odds cannot be defined. The log odds, called logit, is defined as logit P q ≤ k = log P q ≤ k / P q > k . The ordinal logistic regression model is parameterized as

where γ _k0 is the intercept (cutoff point) for category k; c ₁ to c _m are the measured component contents (mass fractions), i.e., the observable continuous variables; γ ₁ to γ _m are the corresponding regression coefficients (slopes), constant across categories. Note that this model is based on the parallel regression (proportional odds) assumption: the logit dependences on the compositions are parallel hyperplanes for different categories k and, hence, the intercepts are different for each category but the slopes are constant across categories. The odds of being less than or equal to category k are

Calculation of the model parameters in R programming environment, including their confidence intervals and goodness-of-fit measures for the model, is described, for example, at the webpage. ⁸³ The following notation is used in R:

where η _i = −γ _i for all the regression coefficients of components contents c ₁ to c _m . Inverting Eq. (20), the probability of getting a response of a certain category k or below can be obtained ⁸⁴ as

The R function “polr” of the MASS package ⁸⁵ is used to fit multinomial ordered logistic models to the experimental data, while the function “predictorEffects” of the Effects package ⁸⁶ is helpful for calculating and plotting probabilities of getting a response equal to a certain category k.

The goodness-of-fit of a model can be evaluated by calculating several pseudo-R statistics, ⁸⁷ which estimate the variability in the outcome of the fitted model. For example, McFadden’s pseudo-R ² is defined as

where M _full is a full model with predictors; M _intercept is the model without predictors, i.e., containing only the intercept; and L is the estimated likelihood.

When the M _full model does not predict the outcome better than the M _intercept model, its ln L(M _full) is not much larger than ln L(M _intercept), hence the corresponding ratio is close to 1 and the McFadden’s pseudo-R ² is close to 0: the model has poor predictive value. Conversely, when the M _full model is good, its ln L(M _full) is close to zero since the likelihood value for each observation is close to 1, and McFadden’s pseudo-R ² is close to 1, indicating successful predictive ability. When comparing two models on the same data, McFadden’s pseudo-R ² would be higher for the model with the greater likelihood.

The R function “PseudoR2” of the DescTools package ⁸⁸ is applicable to the corresponding calculations. Note that correlations between contents of the chemical components may affect the regression coefficients and p values, but they do not influence the predictions, precision of predictions, and the goodness-of-fit statistics. ⁸⁹

An example of multinomial ordered logistic regression of sensory responses to the quality of a sausage from different producers versus the chemical composition of this sausage, is available in Annex A, Example 4.

5.1 The index for independent responses

The probability mass function P by Eq. (1) of a multinomial random variable Y , characterized by a vector p of response probabilities, is the probability that the event Y = n occurs. For example, five such multinomial variables, each corresponding to a product quality property, will be used further with corresponding subscripts from p1 to p5 in the symbols of variables and parameters related to these quality properties.

The probability P _joint of the joint (intersection) event, consisting in the expert responses to the five properties simultaneously, is: ^{96 , 97}

where ∩ is the symbol of intersection of events.

The Venn diagram of the joint event is shown in Fig. 1.

Fig. 1:

Venn diagram of the joint event. The event Y _p1 = n _p1, when the probabilities of the responses to property 1 by categories are as in the specific vector p _p1, is shown with a semi-transparent brown ellipse 1; similar color ellipses indicate 2 (green) − the event Y _p2 = n _p2 for property 2; 3 (violet) − the event Y _p3 = n _p3 for property 3; 4 (blue) − the event Y _p4 = n _p4 for property 4; and 5 (yellow) − the event Y _p5 = n _p5 for property 5. The joint (intersection) event, consisting of the corresponding responses to the five properties simultaneously, is highlighted by the central red non-transparent shape. ²⁸

When responses to these five quality properties are independent, the probability of the joint event (joint probability) is the product:

Treating N responses to each quality property as a separate statistical sample, and corresponding frequencies n k as random variables, the probability vector p = p 1 , p 2 , … , p K is estimated for the quality property as a vector of relative frequencies p ˆ = p ˆ 1 , p ˆ 2 , … , p ˆ K , where p ˆ k = n k / N . Then, an estimate of P p 1 by Eq. (1) is P ˆ p 1 , similarly for other probabilities, while the estimate of the joint probability by Eq. (24) is P ˆ joint = P ˆ p 1 · P ˆ p 2 · P ˆ p 3 · P ˆ p 4 · P ˆ p 5 . The estimate P ˆ joint is the probability that the product will have these, and no other, quality characteristics, expressed as the sensory property values on their ordinal scales.

The product quality index Q _index can be formulated as the negative common logarithm (−log₁₀ or − lg) of the estimate of the joint probability:

When the probability estimates P ˆ of the five quality properties tend to 1, the quality index approaches its minimum value Q index = 0. In any other case Q index > 0. A greater Q index value means smaller values of the joint probability P joint and its estimate P ˆ joint , i.e., a greater chance that the product quality properties will differ from the claimed ones. In this sense, greater Q index values are worse, and the estimation of Q index can be compared with, for example, counting a number of defects. Being the negative logarithm by Eq. (25), the quality index is related to the entropy (a measure of uncertainty ^{98 , 99 , 100} ) of the probability distribution of that product quality properties.

Note that there are no assumptions concerning the contribution of each quality property to the quality index, either being equal or different: the formulated quality index is not a kind of geometric or weighted mean of the property values with probabilities as the weights. Determining the quality characteristics and their categories (the ranges on the ordinal scale), as well as the relevance of these characteristics to consumers, are not the task of this Guide. They are a part of the product specifications in a standard or another regulatory document related to the product.

5.2 The index for responses which might not be independent

When responses to two or more quality properties might not be independent, the probability of the joint (intersection) event P _joint can be represented numerically by a Gaussian copula-based procedure. ^{101 , 102} This procedure is used for generating samples from a discrete multivariate random variable with the prescribed experimental marginal cumulative distributions F ˆ k _ p 1 , F ˆ k _ p 2 , F ˆ k _ p 3 , F ˆ k _ p 4 , F ˆ k _ p 5 and an empirical correlation matrix of those quality properties. When a large number of those multivariate samples, each of size N, are generated, an estimate P ˆ _joint of P _joint can be calculated as the relative frequency of realization of the intersection event Y p 1 = n p 1 ∩ Y p 2 = n p 2 ∩ Y p 3 = n p 3 ∩ Y p 4 = n p 4 ∩ Y p 5 = n p 5 . Then, the quality index Q index = − lg P ˆ joint is obtained.

5.3 Variability of the quality index

A representative dataset of responses to each property is necessary for an estimation of the vector of relative frequencies and PMFs by Eq. (1). It may be a dataset containing results of examination of samples from one batch. The quality index characterizes this batch in such a case. When the dataset is accumulated by examination of batches during a specified term, the quality index characterizes the product (and production in the specified term) in general. The variability of the Q index value for N > 100 is decreasing and may be considered as negligible at the quality index evaluation.

However, it is important that the Q index value depends on the definition of the joint probability P _joint and its estimate P ˆ _joint. For example, the quality index may also be formulated as the negative common logarithm of the joint probability of the product with some specified (preferred) categories of its quality properties. In particular, in the case of an “ideal” product with the excellent (highest) category of each examined property, the joint probability means the probability that the product will be made at the same production conditions as other products of the same kind during the studied term. Then, the corresponding quality index Q index exc is the negative common logarithm of the joint probability of the event when the vector of frequencies of responses of the highest category is obtained for each of the properties.

Such quality indices are discussed in Annex A, Example 5, using the described approach for analysis of a dataset related to a sausage from two producers. ²⁸

6.1 Algorithms for data treatment

When a property of samples of a product of different producers is examined in one laboratory/institution by a group of experienced experts, and the chemical composition of each sample is known from a certificate provided by its producer, a flow chart of data treatment is shown in Fig. 2. It starts from calculation of the frequencies of expert responses (of different categories), and evaluation of the total variation. The next step is decomposition of the total variation into components with the purpose to assess the effects of two factors influencing the variation – “producer” (X1) and “expert” (X2).

Fig. 2:

Flow chart of the data treatment for interlaboratory comparison of categorical characteristics of a substance, material, or object. ²⁷

The components of variation obtained are used for testing the hypotheses on homogeneity of the producers (i.e., the responses to their product quality properties) and homogeneity of the experts (their responses to a property of the same product sample). When responses of different experts are inhomogeneous, and/or the responses to different producers are homogeneous, the analysis is ended. Otherwise, it is assumed that the difference between responses to the product quality of different producers is caused by the differences in the product chemical composition. This hypothesis is tested with multinomial ordered logistic regression analysis. If any of the regression coefficients are statistically significant, probabilities of obtaining a response related to a specific category for different components of the chemical composition are calculated. The last step is plotting such probabilities for visualization of the results and their discussion like in Annex A, Example 4.

Another algorithm is necessary, when the chemical composition of each sample of a product under examination is known and correlations between the responses to different properties of the product are considered. Such an algorithm may start from testing the homogeneity of the datasets of chemical composition that able to influence the responses. For samples, for which the hypothesis on homogeneity of the chemical composition is not rejected, ORDANOVA is implemented. Then only testing correlation of the responses to the different quality properties can be performed. If correlation is not statistically significant, a quality index of the product can be calculated by Eq. (25). Otherwise, the quality index is calculated numerically by a Gaussian copula-based procedure as explained in Sec. 5.2 and Annex A, Example 5.

6.2 Limitations

This Guide discusses the risks of false decisions on consensus as probabilities, not considering the severity of their consequences: quality loss, aesthetic and taste worsening in a product, financial loss, etc. There are also typical limitations of the applied methods of mathematical statistics: the use of any model is a simplified reflection of reality; adequacy of the treatment of a dataset of item-to-item (batch-to-batch) and/or expert-to-expert responses; the goodness-of-fit of experimental and theoretical distributions, etc.