Home Medicine Analytical performance and diagnostic accuracy of six different faecal calprotectin assays in inflammatory bowel disease
Article Open Access

Analytical performance and diagnostic accuracy of six different faecal calprotectin assays in inflammatory bowel disease

  • Matthijs Oyaert EMAIL logo , An Boel , Julie Jacobs , Stefanie Van den Bremt , Maxime De Sloovere , Hilde Vanpoucke and Lieve Van Hoovels
Published/Copyright: February 21, 2017
Download Article (PDF)
Clinical Chemistry and Laboratory Medicine (CCLM)
From the journal Clinical Chemistry and Laboratory Medicine (CCLM) Volume 55 Issue 10

Abstract

Background:

We evaluated the analytical performance of six different faecal calprotectin immunoassays together with their diagnostic accuracy in the discrimination between functional and organic bowel disorders.

Methods:

The faecal samples were obtained from inflammatory bowel disease patients (n=27) at the time of diagnosis [Crohn’s disease (n=15), colitis ulcerosa (n=12)], gastroenterologic disease control patients (n=52) and rheumatologic disease control patients (n=26). All individuals included in the study underwent a concurrent ileocolonoscopy. Analytical performance (imprecision, accuracy, carry-over, correlation and agreement) and diagnostic accuracy (sensitivity, specificity, likelihood ratios) of the different assays were evaluated.

Results:

All methods demonstrated good analytical performance, but within-run and total imprecision varied depending on the assay methodology used. Using Passing Bablok and Bland-Altman analyses, low quantitative agreement was observed between the assays. All assays showed excellent diagnostic accuracy, with areas under the receiver operating characteristic curves (ROC) ranging from 0.974 to 0.998. The AUCs were not significantly different between assays (p>0.05). Diagnostic sensitivity at the cut-off at a fixed specificity of 75% ranged from 95.2% to 100%. Introduction of multiple result intervals increased the clinical interpretation of all the assays.

Conclusions:

Analytical and diagnostic performance of the evaluated faecal calprotectin assays is good, but numerical values differ substantially between the assays necessitating the use of different clinical cut-offs. Introduction of multiple result intervals aids in clinical decision-making.

Keywords: faecal calprotectin; immunoassays; inflammatory bowel disease

Introduction

Inflammatory bowel diseases (IBD) are chronic diseases that result from the inflammation of the intestinal wall and include Crohn’s disease (CD) and ulcerative colitis (UC). In IBD, activated leukocytes infiltrate the mucosa and appear in faeces due to shedding in the intestinal lumen [1], [2]. According to the World Gastroenterology Organization, the diagnosis of IBD relies on a combination of physical examination, patient history as well as a number of diagnostic tests including laboratory analyses, stool examination, endoscopy, biopsy and imaging studies [3].

Among the various (non-invasive) biomarkers that have been proposed over the past few years, faecal calprotectin has gained an important role [4], [5]. Calprotectin is a calcium binding protein present in the neutrophil cytoplasm, representing over 60% of cytosolic proteins. Consequently, faecal calprotectin concentration may be related to inflammation of the bowel mucosa in IBD. Despite some analytical and pre-analytical drawbacks [6], [7], [8], [9], [10], encouraging data have recently been published about the clinical efficacy of this biomarker for monitoring disease activity, response to treatment and relapse [11], [12]. In addition, faecal calprotectin is resistant to bacterial degradation, being stable in stool for up to 3 days at room temperature, which adds more benefits to its use as a laboratory marker [13].

Over the past years, different assays for the detection of faecal calprotectin have been introduced including point-of-care methods [7], [14], [15], [16], [17], [18], [19], [20], [21]. These methods have the advantage of being easy to use and allowing analysis as soon as the sample is received in the laboratory. In addition, results are rapidly available, which allows quick clinical decision making. However, as laboratories are being consolidated, more automated assays such as enzyme-linked immunoassays (ELISA) [7], [22], [23], chemiluminescence immunoassays (CLIA), fluoro enzyme immunoassays (FEIA) and particle enhanced turbidimetric immunoassays (PETIA) for the measurement of faecal calprotectin have been introduced [10].

The aim of this study was to evaluate six different automated faecal calprotectin immunoassays for the diagnosis of IBD using the results of ileocolonoscopy as the criterion standard. Analytical and diagnostic performances were compared for the different tests.

Materials and methods

Patient population

The total population consisted of 105 patients (median age, 35 years; range, 14–94 years; 44 males, 61 females) with suspicion of IBD for whom a stool sample for faecal calprotectin analysis was sent to the clinical laboratory. Of those 105 patients, 86 consecutive patients were recruited over a 6-month period at the OLV Hospital Aalst. Based on the endoscopic results (see further), these patients were categorised into 21 IBD patients, 39 gastroenterologic disease control patients (GCG) and 26 rheumatologic disease control patients (RCG). To enhance the diagnostic power of the likelihood calculations, 19 diagnostic stool samples from AZ Delta Roeselare were added from six IBD patients and 13 gastroenterologic disease control patients. A summary of the patient characteristics with the faecal calprotectin concentrations for the different assays is presented in Table 1.

Table 1:

Overview of the patient characteristics and faecal calprotectin concentrations (μg/g) obtained with the different methods evaluated.a

Final DiagnosisnMen/womenAge (range)bEliA Calprotectin 2Inova QUANTA Flash®Diasorin CalprotectinBühlmann fCAL TurboEuroimmun CalprotectinOrgentec Calprotectin
IBD
 Crohn’s disease1510/536 (14–62)1020.0 (460.5–3168.0)451.4 (291.7–2588.7)278.0 (179.5–1520.0)799.1 (483.7–1491.6)985.5 (542.6–1829.6)1835.2 (1168.5–7275.0)
 Colitis ulcerosa127/535 (20–76)1167.0 (641.0–2734.0)619.5 (396.7–1136.4)560.5 (320.8–1025.3)1058.0 (708.1–1844.1)1555.9 (659.5–1985.3)2539.0 (890.2–5174.3)
Non-IBD
 Gastroc2310/1353 (26–68)98.0 (19.5–206.5)61.6 (<16.1–140.6)60.8 (10.9–86.8)105.9 (<20–222.8)137.2 (14.6–205.1)141.7 (21.9–237.0)
 Rheumatod154/1140 (17–5964.0 (24.0–114.0)37.9 (<16.1–109.0)36.6 (15.9–57.7)71.9 (16.2–83.8)78.0 (33.2–218.0)99.4 (44.1–364.2)
 IBSe4013/2729 (14–67)14.5 (5.4–30.3)<16.1 (<16.1)6.1 (<5–17.3)<20 (<20–29.9)15.6 (3.3–42.5)23.0 (13.0–41.9)

  1. aData are presented as median (IQR). bAge is presented as median in years. Values in brackets indicate minimum and maximum ages. cIncluding oesophagitis, erosive gastritis, gastric ulcers, diverticulitis, microscopic colitis, colorectal cancer, hyperplastic polyps, adenomatous polyps. dIncluding spondylo-arthritis, (undifferentiated) arthritis. eIncluding patients from the gastrologic and rheumatologic disease control group with a final diagnosis of IBS. IBS, irritable bowel syndrome.

The main reasons for performing faecal calprotectin determination were diarrhoea, mucous or bloody stools, weight loss and abdominal pain and cramping. Diagnostic work-up included physical examination and case history, endoscopic and histologic analysis, radiologic work-up and laboratory tests. To allow accurate diagnosis, all included patients underwent ileocolonoscopy. Patients who had previously been diagnosed with IBD or had not received ileocolonoscopy were excluded. Other exclusion criteria included unclear diagnosis (e.g. indeterminate colitis), inability to collect enough faecal sample and age younger than 14 years. Senior gastroenterologists performed all endoscopies and findings were documented in a computer-based database. The final diagnosis of IBD (i.e. CD and UC) was independently made by a pathologist or gastroenterologist who was blinded for calprotectin results.

Faecal calprotectin was determined on a single stool sample in all patients. Samples were transported to the laboratory at room temperature and stored at –20 °C prior to extraction. For all assays, all analyses were performed within the same freeze-thaw cycle of the faeces sample, except for the Inova assay, for which the samples underwent one additional freeze-thaw cycle [17].

The study was performed with full respect for individuals’ rights to confidentiality and in accordance with procedures supervised by Local Authorities responsible for Ethical Research (Belgian registration number of ethical approval B126201526847).

Measurement of faecal calprotectin

Faecal calprotectin extraction:

To prevent pre-analytical variation [6], [10], all samples were weighted and extracted with the Smart Prep Extraction Device (Roche Diagnostics, Mannheim, Germany) in accordance with the manufacturer’s instructions but using the different company specific faecal calprotectin buffers. Extractions were performed in different batches, immediately followed by calprotectin analyses.

Faecal calprotectin tests:

Measurement of faecal calprotectin was performed using six automated immunoassays: Thermo Fisher EliA Calprotectin 2 assay on the Phadia 250 (Thermo Fisher Scientific, Uppsala, Sweden); Diasorin Calprotectin assay on the Liaison (Diasorin S.P.A., Saluggia, Italy), Inova QUANTA Flash Calprotectin (research use only) on the Inova BIO-FLASH instrument (Inova Diagnostics, San Diego, CA, USA); Bühlmann fCAL Turbo (Bühlmann Laboratories AG, Schönenbuch, Switzerland) on the Roche Cobas c501 (Roche Diagnostics, Mannheim, Germany), Euroimmun Calprotectin assay (Euroimmun; Lübeck, Germany), on an automated ELISA instrument (QUANTA-Lyser 2; Inova) and Orgentec Calprotectin assay on the Alegria (Orgentec Diagnostika, Mainz, Germany). All assays were performed following manufacturer’s instructions. An overview of the different methods is presented in Table 2.

Table 2:

Faecal calprotectin assay characteristics.

ManufacturerAntibodyFaecal extraction dilutionaNo. of calibratorsMeasuring principleProposed cut-off, μg/gMeasuring range, μg/gLOQLinearity (R2)Number of tests per kit
CaptureDetection
EliA Calprotectin 2Monoclonal, nativeMonoclonal, mouse1:756FEIA503.8–6000.03.8NR64
Diasorin CalprotectinMonoclonal, recombinantMonoclonal, recombinant1:502CLIA505.0–8000.0b5.00.996100
Inova Quanta Flash®Polyclonal, nativeMonoclonal, native1:503CLIA5016.1–3500.016.10.990100
Bühlmann fCAL TurboNAPolyclonal avian1:506PETIA5020.0–8000.0b20.0NR200
Euroimmun CalprotectinMonoclonal, nativeMonoclonal, native1:506ELISA506.5–2100.06.50.98096
Orgentec CalprotectinPolyclonalcMonoclonalc1:50NRELISA505.2–1000.05.2NR24

  1. aUsing the Roche Smart Prep Extraction Device. bAfter 1:10 (Diasorin) and 1:4 (Bühlmann) dilution, concentrations up to 8000.0 μg/g faecal calprotectin can be obtained. cOrigin not reported. FEIA, fluoro-enzyme immunoassay; CLIA, chemiluminescence immunoassay; PETIA, particle enhanced turbidimetric immunoassay; ELISA, enzyme-linked immunosorbent assay; LOQ, limit of quantification; NA, not applicable; NR, not reported.

Comparison of immunoassays

Analytical performance:

The imprecision was assessed by duplicate testing of at least five aliquots of extracted patient sample and internal quality control samples (one sample with a high calprotectin value and one with a low calprotectin value), according to the CLSI EP5-A2 guideline [24]. To determine the amount of carry-over, a sample with a high calprotectin concentration and one with a low concentration (i.e. blank extraction buffer) were analysed in the sequence HHBBB, where H is the sample with the high concentration and B is the blank [25]. Accuracy was evaluated from eQC samples (n=4, 2 positive, 2 negative) obtained from the INSTAND® external quality control program for faecal makers. For analytical method comparison, Bland-Altman plots, Passing-Bablok regression analysis and Pearson correlation coefficients were determined for all assays.

Diagnostic performance:

Based on the results of the consecutive sample collection, receiver operating characteristics (ROC) analyses were carried out to determine the diagnostic performance of each test. Deduced from the results of the ROC-analyses, cut-off values with the highest sensitivity and specificity have been calculated for diagnosing IBD. In addition, using the results of the whole patient sample population (n=105), the calculation of positive likelihood ratio’s (LR) for IBD was performed for four result intervals for each assay (<upper limit of normal (ULN), 1–3×ULN, 3–10×ULN and >10×ULN). Comparisons were made using non-parametric tests (Kruskall-Wallis for unpaired samples). A p-value <0.05 was considered statistically significant. All statistical analyses were performed using MEDCALC® (software version 15.6.1, Ostend, Belgium).

Results

Analytical performance

Imprecision, carry-over and accuracy

Within-run imprecision varied between 0.6% and 19.7%. Total imprecision results ranged from 1.5% to 23.3% (Table 3). The lowest imprecision was obtained with the CLIA methods and the highest with the ELISA assays. No significant carry-over was detected (<0.03% for all methods). The results of the eQC-samples were qualitatively correctly interpreted (i.e. positive or negative) by all the assays using the manufacturer’s cut-offs (Table 2). However, important differences in the numerical values were obtained with the different assays.

Table 3:

Intra-run and total imprecision for faecal calprotectin determinations with the tested assays.

Intra-run imprecisionaTotal imprecisiona
Kit-controlsbPatient samplescKit-controlsbPatient samplesc
LowHighLowHighLowHighLowHigh
EliA Calprotectin 29.18.25.78.010.09.15.78.0
Diasorin Calprotectin3.34.32.52.13.37.24.72.8
Inova QUANTA Flash®3.60.91.70.63.71.91.73.7
Bühlmann fCAL Turbo5.61.44.12.55.61.58.42.5
Orgentec Calprotectind––13.29.7––13.29.7
Euroimmun Calprotectin10.619.716.27.710.623.316.218.1

  1. aThe results are presented as %CV (coefficient of variation). bLow and high kit-controls ranged from 20 to 60 μg/g and 130–270 μg/g. Kit-controls for the EliA and Inova Quanta Flash assays were analysed 10 times in duplo, for the Diasorin, fCAL Turbo and Euroimmun assays, five in duplo determinations were performed. cLow and high patient-sample controls ranged from 15 to 60 μg/g and 100 to 450 μg/g, respectively. The patient-sample controls were measured five times in duplo for all assays. dAs internal kit-controls are reported as positive/negative, no data for intra-run and total imprecision for kit-controls are available for this assay.

Method comparison

The range of the faecal calprotectin concentrations and median faecal calprotectin concentrations in diagnostic IBD samples varied substantially between the measurement methods (Table 1). Quantitative data distribution showed that calprotectin concentrations in diagnostic samples of IBD patients measured with the Diasorin method were significantly lower (p<0.05) compared to the concentrations obtained with the Euroimmun, Orgentec and Phadia assays. The highest values were measured with the Orgentec assay, which were found significantly higher (p<0.05) compared to the other assays (Figure 1).

Figure 1: Box-Whisker plots representing faecal calprotectin concentrations among the different assays for the IBD populations.The values falling outside the Box-Whisker plot are outliers.
Figure 1:

Box-Whisker plots representing faecal calprotectin concentrations among the different assays for the IBD populations.

The values falling outside the Box-Whisker plot are outliers.

For the total population (i.e. disease+disease control group), significantly lower (p<0.05) concentrations were obtained with the Diasorin assay compared to the Euroimmun, fCAL Turbo, Orgentec and Phadia assay. The Orgentec assay measured significantly higher (p<0.05) compared to the Diasorin and Inova assay. For the other assays, no significant differences in faecal calprotectin concentration were found.

Using Pearson correlation, we assessed linear agreement among the different assays. The correlation coefficients (r) for all possible correlations are presented in Table 4. The lowest correlation coefficients were observed between the Euroimmun assay and the Inova, Orgentec, fCAL Turbo and Diasorin assays (ranging from 0.60 to 0.75). For all other assays, we observed a very good correlation (mean r, 0.89; median, 0.90; range, 0.81–0.95) (Table 4). A high correlation does not mean that two methods have a good agreement. We therefore performed Passing-Bablok regression and Bland-Altman analyses (Table 4). These analyses showed low agreement between assays. Additionally, Bland-Altman analysis, which was performed by plotting the difference between observations against the mean, revealed that the difference between assays increased for increasing means (heteroscedasticity). The best agreement was observed between the results obtained with the Inova and Diasorin methods.

Table 4:

Faecal calprotectin ELISAs: results of Pearson correlation (r), Bland-Altman analysis (mean differences [μg/g]), and Passing-Bablok Regression analysis.

Comparison method
Inova QUANTA Flash®Diasorin CalprotectinEuroimmun CalprotectinBühlmann fCAL TurboOrgentec Calprotectin
Diasorin Calprotectin
 r0.95 (0.92–0.96)
 Bias79.3 (−518.1 to 676.7)
 Intercept1.64 (−0.03 to 5.27)
 Slope1.25 (1.09–1.39)
Euroimmun Calprotectin
 r0.75 (0.64–0.82)0.74 (0.64–0.82)
 Bias−129.0 (−1588.6 to 1330.7)−181.5 (−1567.7 to 1204.7)
 Intercept5.54 (−1.41 to 6.69)1.12 (0.20–1.29)
 Slope0.49 (0.40–0.64)0.42 (0.37–0.45)
Bühlmann fCAL Turbo
 r0.90 (0.86–0.94)0.95 (0.93–0.97)0.70 (0.59–0.79)
 Bias–136.6 (–1437.8 to 1164.5)–189.0 (–1367.4 to 989.3)–7.6 (–1861.7 to 1846.7)
 Intercept1.73 (0.82–2.81)–2.57 (–3.77 to 0.02)–6.77 (–13.2 to 1.49)
 Slope0.63 (0.52–0.72)0.56 (0.51–0.63)1.35 (1.11–1.65)
Orgentec Calprotectin
 r0.85 (0.78–0.90)0.85 (0.79–0.90)0.60 (0.46–0.70)0.90 (0.86–0.93)
 Bias−887.0 (−6124.0 to 4350.4)−896.8 (−6019.3 to 4225.6)−715.3 (−6131.2 to 4700.5)−707.8 (−4924.3 to 3508.7)
 Intercept3.20 (0.85–5.50)1.87 (0.19–2.70)1.68 (−0.72 to 5.84)5.83 (1.87–8.10)
 Slope0.29 (0.25–0.36)0.24 (0.21–0.29)0.62 (0.53–0.71)0.37 (0.33–0.50)
EliA Calprotectin 2
 r0.95 (0.92–0.97)0.90 (0.86–0.93)0.82 (0.75–0.88)0.87 (0.81–0.91)0.81 (0.74–0.87)
 Bias−209.1 (−1313.3 to 895.1)−256.7 (−1572.1 to 1058.8)−75.2 (−1518.4 to 1368.0)−67.6 (−1389.1 to 1253.8)640.1 (−3934.8 to 5215.1)
 Intercept3.40 (−0.59 to 5.40)1.15 (−0.07 to 1.71)1.27 (−3.06 to 2.93)4.00 (1.43–6.63)−0.67 (−7.36 to 5.22)
 Slope0.58 (0.48–0.66)0.45 (0.41–0.48)1.07 (1.00–1.22)0.80 (0.66–0.96)1.72 (1.53–2.02)

  1. The 95% confidence intervals are indicated in brackets.

Diagnostic performance of the faecal calprotectin assays

Among the study population, the prevalence of IBD was 25.7% (27/105). Sensitivities and specificities were calculated for each faecal calprotectin method (Table 5). The areas under the curves (AUC) were 0.974 (0.915–0.996), 0.993 (0.946–1.000), 0.978 (0.916–0.998), 0.988 (0.936–1.000), 0.991 (0.942–1.000), and 0.998 (0.954–1.000) for the Phadia, Diasorin, Inova, Bühlmann, Euroimmun and Orgentec assay, respectively. The AUCs were not statistically significantly different between the assays (p>0.05). At the cut-off value proposed by the manufacturer (i.e. 50 μg/g for all assays), sensitivity for all assays was 100%, but the specificity was much lower (Table 5). No significant differences in specificity were obtained between the assays, except for the Phadia and Inova assay (p<0.05). Based on the ROC curve analysis, cut-offs at fixed specificity of 75% and 95% differed between the assays, ranging from 41.3 to 142.9 μg/g and 110.0 to 411.9 μg/g faecal calprotectin, respectively (Table 5). Sensitivities at these cut-off levels ranged from 95.2% to 100% and 84.2% to 100%, but no significant differences in sensitivity were obtained between the assays.

Table 5:

Sensitivities and specificities of the faecal calprotectin test at the the cut-off with maximal sensitivity and specificity, the manufacturer’s cut-off, and cut-off with fixed specificity of 75% and 95%.

Assay
EliA Calprotectin 2Inova QUANTA Flash®Diasorin CalprotectinBühlmann fCAL TurboEuroimmun CalprotectinOrgentec Calprotectin
ROC’s cut-off (μg/g) at maximal sensitivity and specificity376115111285371477
AUC0.974 (0.915–0.996)0.978 (0.916–0.998)0.993 (0.946–1.000)0.988 (0.936–1.000)0.991 (0.942–1.000)0.998 (0.954–1.000)
SE0.01640.01290.00620.01090.00700.0025
Sensitivity (95% CI)85.7 (63.7–97.0)100 (82.4–100)100 (83.9–100)95.2 (76.2–99.9)95.2 (76.2–99.9)100 (83.9–100)
Specificity (95% CI)100 (94.5–100)86.4 (75.3–92.9)96.9 (89.3–99.6)95.4 (87.1–99.0)96.9 (89.3–99.6)98.5 (91.7–100)
Manufacturer’s cut-off, μg/g505050505050
Sensitivity (95% CI)100 (83.9–100)100 (82.4–100)100 (83.9–100)100 (83.9–100)100 (83.9–100)100 (83.9–100)
Specificity (95% CI)66.2 (53.4–77.4)72.9 (59.7–83.6)78.5 (66.5–87.7)66.2 (53.4–77.4)58.4 (45.6–70.6)58.6 (45.6–70.6)
Cut-off (μg/g) at fixed specificity of 75%102.061.641.371.9125142.9
Sensitivity (95% CI)95.2 (76.2–99.9)100 (82.4–100)100 (83.9–100)100 (83.9–100)100 (83.9–100)100 (83.9–100)
Cut-off (μg/g) at fixed specificity of 95%250.0208.7110.0284.5345.4411.9
Sensitivity (95% CI)85.7 (63.7–97.0)84.2 (60.4–96.6)100 (83.9–100)95.2 (76.2–99.9)95.2 (76.2–99.9)100 (83.9–100)

  1. Only the consecutive population (n=86 patients) were selected for ROC curve analysis. CI, confidence interval; SE, standard error.

Calculation of likelihoods and associated LRs for chosen result intervals clearly illustrated that the LR increases significantly when the faecal calprotectin concentration was higher than the cut-off value (Table 6). Consequently, diagnostic information is lost when results are interpreted as positive/negative only. Bayes theorem was used to calculate post-test probabilities as a function of pre-test probability.

Table 6:

Likelihood ratios for various result intervals for the different test.

MethodLikelihood ratios (95% CI)
Negative1-3xULN3-10xULN>10xULN
EliA

Calprotectin 2		0.00

(0.00–0.44)		0.41

(0.10–1.70)		1.4

(0.6–3.5)		55

(8–391)
Diasorin

Calprotectin		0.00

(0.00–0.38)		1.2

(0.6–2.7)		10.1

(2.2–45.7)		+∞

(5 to +∞)
Inova QUANTA

Flash®		0.00

(0.00–0.46)		0.43

(0.10–1.78)		2.8

(1.3–6.3)		+∞

(5 to +∞)
Bühlmann fCAL

Turbo		0.00

(0.00–0.47)		0.14

(0.02–1.03)		1.9

(0.8–4.9)		59

(8–410)
Euroimmun

Calprotectin		0.00

(0.00–0.50)		0.21

(0.03–1.50)		0.76

(0.30–1.80)		+∞

(7.6 to +∞)
Orgentec

Calprotectin		0.00

(0.00–0.50)		0.00

(0.00–1.67)		0.19

(0.03–1.39)		19

(7–49)

  1. The values are presented as likelihood ratios with the 95% confidence interval (in parentheses); ULN, upper limit of normal; CI, confidence interval.

By defining result intervals, IBD was ruled out in 51 (64.6%), 60 (75.9%), 48 (67.6%), 48 (60.8%), 45 (57.0%) and 45 (57.0%) of the non-IBD patients with the Phadia, Diasorin, Inova, Bühlmann, Euroimmun and Orgentec assay (<50 μg/g), respectively. Furthermore, calprotectin values between 1 and 3 times and 3 and 10 times the ULN have no clinical usefulness and cannot be used to exclude or to confirm the diagnosis of IBD, except for the 3–10 times ULN for the Diasorin assay, where the diagnosis of IBD was confirmed in seven of the IBD patients (25.9%) [LR=10.1 (95% CI 2.2–45.7)]. The diagnosis was confirmed in 19 (70.4%), 13 (48.1%), 14 (56.0%), 20 (74.1%), 21 (77.8%) and 26 (96.3%) of the IBD patients when using a cut-off of >500 μg/g faecal calprotectin with the Phadia, Diasorin, Inova, Bühlmann, Euroimmun and Orgentec assays, respectively.

Discussion

In this study, we evaluated and compared six current available faecal calprotectin assays and investigated whether quantitative results of different assays were comparable.

Generally, all evaluated faecal calprotectin assays demonstrated good analytical performance. Within-run and total imprecision were found to be higher for the ELISA/FEIA techniques (Euroimmun, Orgentec and Phadia) compared to the CLIA/PETIA assays (Diasorin, Inova and Bühlmann fCAL Turbo). It should be mentioned that the protocol for Euroimmun calprotectin ELISA was applied on Inova QUANTA-Lyser 2, which is a non-validated instrument for this method. This could contribute to the higher imprecision results found for this assay.

Qualitative correlations between the methods from the different manufacturers were found to be good, but quantitative agreement was poor, which means that the result of one method cannot be replaced by the result of another (Table 4). Therefore, faecal calprotectin concentrations are not interchangeable. These results are in line with a study from the UK National External Quality Assessment Service that revealed up to 3.8-fold differences between methods from different manufacturers [6]. This suggests that, among other possible reasons, the antibodies used in the different assays are directed against different complexes of the faecal calprotectin protein. In addition, the methods evaluated use different antibodies (monoclonal vs. polyclonal) of different origins (recombinant vs. s. native) with different immunoassay techniques (ELISA vs. PETIA vs. CLIA vs. FEIA) (Table 2). With the introduction of more assays, efforts to harmonise faecal calprotectin assays would be of particular interest [26]. Our study and other studies identify the need for both a standardised calprotectin reference material against which these assays can be calibrated, as well as an in-house clinical evaluation. By comparing faecal calprotectin concentrations with endoscopy and histology, the diagnostic accuracy of the assays can be verified and if necessary, be used to establish local cut-off values according to the intended use of the assay (i.e. diagnostic vs. disease monitoring).

Along with non-standardisation between the different faecal calprotectin assays, pre-analytical issues of faecal calprotectin analyses can contribute to the (often large) variation in faecal calprotectin concentrations between different assays [6], [7], [10], [26]. In this study, all faecal samples where appropriately homogenised and weighed before extraction. Extraction was performed using the same extraction device (i.e. the Roche Smart Prep Extraction device) for all assays. By using this device, we excluded variation in faecal calprotectin concentrations due to liquid or hard faecal samples, which can be found when using other extraction devices [6]. Moreover, by adding the appropriate amount of extraction buffer based on the weighed faecal sample, we are convinced that the initial dilution was correct. These pre-analytical confounders need to be taken into account when laboratories are planning to determine faecal calprotectin concentrations.

Overall, the diagnostic performance of the six evaluated faecal calprotectin assays was excellent, with only minor differences in sensitivity and specificity. The diagnostic accuracy at the cut-off with the highest sum of sensitivity and specificity was high and not statistically different between the assays evaluated, signifying that the diagnostic performance of the various assays was comparable. Cut-off values calculated from ROC analyses for detection of IBD were considerably higher for the Orgentec assay (477 μg/g) than for the other assays, compared to the recommended cut-offs by all manufacturers (all 50 μg/g faeces). The sensitivity as well as specificity of all assays was comparable when using the optimal cut-offs. On the contrary, we found that all assays had a sensitivity of 100% when the cut-off of the manufacturer was used (i.e. 50 μg/g) but specificity at this cut-off value differed between the assays (ranging from 58.4% to 78.5%). This, together with the fact that there is not yet a standardised method for faecal calprotectin measurement, indicates that recommended cut-off values would have to be determined depending on the clinical setting they are used in [10].

There is a close relationship between spondylarthropathies (SpA) and IBD. Clinically, 5%–10% of all patients with ankylosing spondylitis have concurrent IBD. Even in the absence of relevant gastrointestinal symptoms, microscopic gut inflammation was observed in more than 40% of SpA patients [27]. The presence of (microscopic) gut inflammation appears relevant for prognosis and therapeutic decision making as SpA patients with a chronic type of gut inflammation have a less favourable disease course. Therefore, faecal calprotectin analysis is part of the clinical diagnostic work-up of SpA patients [28]. In our study, 26 consecutive rheumatologic patients (24.8%) were included. However, a recalculation of the diagnostic performance of the different assays for IBD by excluding this control group revealed no significant differences in optimal cut-off values and performance characteristics of the different tests (data not shown).

A drawback of the use of a single cut-off value is that this diagnostic information is lost when results are interpreted as positive/negative only. For example, a calprotectin result higher than 477 μg/g (Orgentec) does not mean the patient has IBD; neither does a result lower than 477 μg/g rules out the diagnosis of IBD, but the probability is different. To tackle this interpretative ambiguity, we defined result intervals and, thus, take full advantage of the quantitative test data. In addition, LRs allow us to integrate (through Bayesian calculations) a physician’s judgment about pre-test probability of the disease in the patients’ cohort under estimation. For all assays, we were able to define intervals with associated LRs that increase when the results deviate quantitatively more from normal. Our approach results in a clear gain of diagnostic power of individual laboratory tests as compared with classic dichotomous interpretation [29].

Interpretation of laboratory results must be done in the clinical context and in function of the pre-test probability [29]. For example, in our study, we determined that in patients with diarrhoea, mucous or bloody stools, weight loss and abdominal pain and cramping, the pre-test probability for IBD is 25.7%. The post-test probability for IBD in these patients will be 31.0%, 77.4%, 49.3%, 39.4%, 19.6% and 19.6% for patients with a faecal calprotectin concentration between 3 and 10 times the ULN for the Phadia, Diasorin, Inova, Bühlmann, Euroimmun and Orgentec assay, respectively. The post-test probability for patients with a faecal calprotectin concentration greater than 10-fold the ULN will be 94.9%, 99.4%, 93.1%, 95.1%, 99.6% and 86.8% for the Phadia, Diasorin, Inova, Bühlmann, Euroimmun and Orgentec assay, respectively. These findings suggest that patients with a high pre-test probability and levels of faecal calprotectin greater than 10-fold the cut-off value have a high probability for having IBD, aiding in clinical decision making. However, these findings depend on the clinical setting and the assay used for faecal calprotectin determination and therefore need to be confirmed in larger cohorts of patients.

The strength of our study lies in the fact that we included six different assays, of which the Inova QUANTA Flash® and EliA Calprotectin 2 are evaluated for the first time. In addition, we defined two well-described disease control populations, thereby reflecting the real clinical use of this test. However, our study also has two important limitations. Compared to other studies, we included a relative low number of samples. Second, the sensitivity of all assays at their recommended cut-off is 100%, which is in contrast to other studies. Including more IBD patients of which at least some have faecal calprotectin concentrations lower than 50 μg/g would make the test population more suitable for comparison. The only differences we found are in specificity, but there are definitely also differences in sensitivity between the assays.

In conclusion, we demonstrated good analytical performance of all assays tested. Variation in the faecal calprotectin concentrations between the different methods does not allow to use these assays interchangeably. Diagnostic performance is high but depends on the clinical setting in which the test is used. Use of LRs for different result intervals aids in the clinical interpretation of faecal calprotectin results, thereby improving the clinical interpretation for the diagnosis of IBD as compared with conventional dichotomous test interpretation.

Acknowledgments

We thank Thermo Fisher Scientific, Diasorin, Inova, Bühlmann, Euroimmun and Launch Diagnostics for the donation of the assays. We are grateful to the laboratory technicians for their most valuable efforts.

  1. Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

  2. Research funding: None declared.

  3. Employment or leadership: None declared.

  4. Honorarium: None declared.

  5. Competing interests: The funding organization(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication.

References

1. Silicia B, Saro C. Inflammatory bowel diseases: a disease(s) of modern times? Is incidence still increasing. World J Gastroenterol 2008;14:5491–8.10.3748/wjg.14.5491Search in Google Scholar PubMed PubMed Central

2. Soon I, Molodecky N, Rabi D, Ghali W, Barkema H, Kaplan G, et al. Increasing incidence and prevalence of the inflammatory bowel diseases with time, based on systematic review. Gastroenterol 2012;143:46–54.10.1053/j.gastro.2011.10.001Search in Google Scholar PubMed

3. World Gastroenterology Organisation (WGO). World Gastroenterology Organisation Global Guideline. Inflammatory Bowel disease, a global perspective. Munich, Germany: WGO, 2009.Search in Google Scholar

4. Van Rheenen PF, Van de Vijver E, Fdler V. Faecal calprotectin for screening of patients with suspected inflammatory bowel disease: diagnostic meta-analysis. Br Med J 2010; 341:c3369.10.1136/bmj.c3369Search in Google Scholar PubMed PubMed Central

5. Menees SB, Powel C, Kurlander J, Goel A, Chey WD. A meta-analysis of the utility of C-reactive protein, erythrocyte sedimentation rate, fecal calprotectin, and fecal lactoferrin to exclude inflammatory bowel disease in adults with IBS. Am J Gastroenterol 2015;110:444–54.10.1038/ajg.2015.6Search in Google Scholar PubMed

6. Whitehead SJ, Franch J, Brookes MJ, Ford C, Gama R. Between-assay variability of faecal calprotectin enzyme-linked immunosorbent assay kits. Ann Clin Biochem 2013;50:53–61.10.1258/acb.2012.011272Search in Google Scholar PubMed

7. Oyaert M, Trouvé C, Baert F, De Smet D, Langlois M, Vanpoucke H. Comparison of two immunoassays for measurement of faecal calprotectin in detection of inflammatory bowel disease: (pre)-analytical and diagnostic performance characteristics. Clin Chem Lab Med 2014;52:391–7.10.1515/cclm-2013-0699Search in Google Scholar PubMed

8. Dumoulin EN, Van Biervliet S, Langlois MR, Delanghe JR. Proteolysis is a confounding factor in the interpretation of faecal calprotectin. Clin Chem Lab Med 2015;53:65–71.10.1515/cclm-2014-0568Search in Google Scholar PubMed

9. Kolho KL, Turner D, Veereman-Wauters G, Sladek M, de Ridder L, Shaoul R, et al. Rapid test for fecal calprotectin levels in children with Crohn’s disease. J Pediatr Gastroenterol Nutr 2012;55:436–9.10.1097/MPG.0b013e318253cff1Search in Google Scholar PubMed

10. Delefortrie Q, Schatt P, Grimmelprez A, Gohy P, Deltour P, Collard G, et al. Comparison of the Liaison® Calprotectin kit with a well-established point-of-care test (Quantum Blue – Bühlmann-Alere® in terms of analytical performances and ability to detect relapses amongst a Crohn population in follow-up. Clin Biochem 2016;49:268–73.10.1016/j.clinbiochem.2015.10.010Search in Google Scholar PubMed

11. Basso D, Zambon CF, Plebani M. Inflammatory bowel diseases: where from pathogenesis to laboratory testing. Clin Chem Lab Med 2014;52:471–81.10.1515/cclm-2013-0588Search in Google Scholar PubMed

12. Tonutti E, Agostinis P, Bizzaro N. Inflammatory Bowel Diseases: where are we and where should we go. Clin Chem Lab Med 2014;52:463–5.10.1515/cclm-2014-0146Search in Google Scholar PubMed

13. Lasson A, Strid H, Ohman L, Isaksson S, Olsson M, Rydström B, et al. Fecal calprotectin one year after ileocaecal resection for crohn’s disease- a comparison with findings at oleocolonoscopy. J Crohns Colitis 2014;8:789–95.10.1016/j.crohns.2013.12.015Search in Google Scholar PubMed

14. Hessels J, Douw G, Yildrim DD, Meerman G, van Herwaarden MA, van den Bergh FA. Evaluation of Prevent ID and Quantum Blue rapid tests for fecal calprotectin. Clin Chem Lab Med 2012;50:1079–82.10.1515/cclm-2011-0855Search in Google Scholar PubMed

15. Dolci A, Panteghini M. Comparative study of a new quantitative rapid test with an established ELISA method for faecal calprotectin. Clin Chim Acta 2012;413:350–1.10.1016/j.cca.2011.09.030Search in Google Scholar PubMed

16. Wassell J, Wallage M, Brewer E. Evaluation of the quantum blue rapid test for faecal calprotectin. Ann Clin Biochem 2012;49:55–8.10.1258/acb.2011.011106Search in Google Scholar PubMed

17. Coorevits L, Baert FJ, Vanpoucke HJ. Faecal calprotectin: comparative study of the Quantum Blue rapid test and an established ELISA method. Clin Chem Lab Med 2013;51:825–31.10.1515/cclm-2012-0386Search in Google Scholar PubMed

18. Benahmed NA, Manéné D, Barbot-Trystram L, Kapel N. Evaluation of Calfast® immunochromatographic quantitative assay for the measurement of calprotectin in faeces. Clin Chem Lab Med 2014;52:e143–5.10.1515/cclm-2014-0017Search in Google Scholar PubMed

19. Vestergaard T, Nielsen SL, Dahlerup JF, Hornung N. Fecal calprotectin: assessment of a rapid test. Scand J Clin Lab Invest 2008;68:343–7.10.1080/00365510701576198Search in Google Scholar PubMed

20. Burri E, Manz M, Rothen C, Rossi L, Beglinger C, Lehmann FS. Monoclonal antibody testing for fecal calprotectin is superior to polyclonal testing of fecal calprotectin and lactoferrin to identify organic intestinal disease in patients with abdominal discomfort. Clin Chim Acta 2013;416:41–7.10.1016/j.cca.2012.11.008Search in Google Scholar PubMed

21. Sydora M, Sydora B, Fedorak R. Validation of a point-of-care desk top device to quantitate fecal calprotectin and distinguish inflammatory bowel disease from irritable bowel syndrome. J Crohns Colitis 2012;6:207–14.10.1016/j.crohns.2011.08.008Search in Google Scholar PubMed

22. Labaere D, Smismans A, Van Olmen A, Christiaens P, D’Haens G, Moons V, et al. Comparison of six different calprotectin assays for the assessment of inflammatory bowel disease. United European Gastroenterol J 2014;2:30–7.10.1177/2050640613518201Search in Google Scholar PubMed PubMed Central

23. Nilsen T, Sunde K, Hansson L, Havelka AM, Larsson A. A novel turbidimetric immunoassay for fecal calprotectin optimized for routine chemistry analyzers. J Clin Lab Anal 2016. DOI:10.1002/jcla.22061.Search in Google Scholar PubMed PubMed Central

24. CLSI. Evaluation of Precision Performance of Quantitative Measurement Methods; Approved Guideline – Second Edition. CLSI Document EP05-A2. Wayne, PA: Clinical Laboratory Standards Institute, 2004.Search in Google Scholar

25. Haeckel R. Recommendations for definition and determination of carry-over effects. J Automat Chem 1988;10:181–3.10.1155/S1463924688000380Search in Google Scholar PubMed PubMed Central

26. Sun S, Cavey T, Peltier L, Bendavid C, Bouguen G. Letter: wide variation in faecal calprotectin values according to the assay. Aliment Pharmacol Ther 2016;43:166–80.10.1111/apt.13455Search in Google Scholar PubMed

27. Cypers H, Varkas G, Beeckman S, Debusschere K, Vogl T, Roth J, et al. Elevated calprotectin levels reveal bowel inflammation in spondyloarthritis. Ann Rheum Dis 2015;75:1357–62.10.1136/annrheumdis-2015-208025Search in Google Scholar PubMed PubMed Central

28. Wendling D, Vuitton L, Koch S, Prati C. Spondyloarthritis and the gut: a new look. Joint Bone Spine 2015;82:77–9.10.1016/j.jbspin.2014.12.004Search in Google Scholar PubMed

29. Bossuyt X. Clinical performance characteristics of a laboratory test. A practical approach in the autoimmune laboratory. Autoimmun Rev 2009;8:543–8.10.1016/j.autrev.2009.01.013Search in Google Scholar PubMed

Received: 2016-11-4
Accepted: 2016-12-29
Published Online: 2017-2-21
Published in Print: 2017-8-28

©2017, Matthijs Oyaert et al., published by De Gruyter.

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Articles in the same Issue

  1. Frontmatter
  2. Editorials
  3. Reporting LDL-cholesterol levels in the era of intensive lipid management: a clarion call
  4. The challenges of genetic risk scores for the prediction of coronary heart disease
  5. Reviews
  6. Advanced lipoprotein testing for cardiovascular diseases risk assessment: a review of the novel approaches in lipoprotein profiling
  7. A review of the challenge in measuring and standardizing BCR-ABL1
  8. Mini Review
  9. Challenges in the analysis of epigenetic biomarkers in clinical samples
  10. Opinion Paper
  11. Defining a roadmap for harmonizing quality indicators in Laboratory Medicine: a consensus statement on behalf of the IFCC Working Group “Laboratory Error and Patient Safety” and EFLM Task and Finish Group “Performance specifications for the extra-analytical phases”
  12. Genetics and Molecular Diagnostics
  13. Assessment of EGFR mutation status using cell-free DNA from bronchoalveolar lavage fluid
  14. General Clinical Chemistry and Laboratory Medicine
  15. A survey of patients’ views from eight European countries of interpretive support from Specialists in Laboratory Medicine
  16. Verification of examination procedures in clinical laboratory for imprecision, trueness and diagnostic accuracy according to ISO 15189:2012: a pragmatic approach
  17. Expressing analytical performance from multi-sample evaluation in laboratory EQA
  18. A candidate reference method for serum potassium measurement by inductively coupled plasma mass spectrometry
  19. Practical motives are prominent in test-ordering in the Emergency Department
  20. Technical and clinical validation of the Greiner FC-Mix glycaemia tube
  21. Comparison of pneumatic tube system with manual transport for routine chemistry, hematology, coagulation and blood gas tests
  22. Accuracy of cerebrospinal fluid Aβ1-42 measurements: evaluation of pre-analytical factors using a novel Elecsys immunosassay
  23. Evaluation of cannabinoids concentration and stability in standardized preparations of cannabis tea and cannabis oil by ultra-high performance liquid chromatography tandem mass spectrometry
  24. Analytical performance and diagnostic accuracy of six different faecal calprotectin assays in inflammatory bowel disease
  25. Novel immunoassays for detection of CUZD1 autoantibodies in serum of patients with inflammatory bowel diseases
  26. Hematology and Coagulation
  27. Critical appraisal of discriminant formulas for distinguishing thalassemia from iron deficiency in patients with microcytic anemia
  28. Reference Values and Biological Variations
  29. Reference ranges of thromboelastometry in healthy full-term and pre-term neonates
  30. Cancer Diagnostics
  31. Immunoparesis in IgM gammopathies as a useful biomarker to predict disease progression
  32. Cardiovascular Diseases
  33. Assessment of the clinical utility of adding common single nucleotide polymorphism genetic scores to classical risk factor algorithms in coronary heart disease risk prediction in UK men
  34. Time and age dependent decrease of NT-proBNP after septal myectomy in hypertrophic obstructive cardiomyopathy
  35. Infectious Diseases
  36. Higher serum caspase-cleaved cytokeratin-18 levels during the first week of sepsis diagnosis in non-survivor patients
  37. Letters to the Editor
  38. Data mining for age-related TSH reference intervals in adulthood
  39. Intra-laboratory variation and its effect on gestational diabetes diagnosis
  40. Evaluation of long-term imprecision of automated complete blood cell count on the Sysmex XN-9000 system
  41. Sensitivity of the Sysmex XN9000 WPC-channel for detection of monoclonal B-cell populations
  42. Evaluation of biotin interference on immunoassays: new data for troponin I, digoxin, NT-Pro-BNP, and progesterone
  43. Stability of procalcitonin in cerebrospinal fluid
  44. Between-laboratory analysis of IgG antibodies against Aspergillus fumigatus in paired quality control samples
  45. Mass spectrometry vs. immunoassay in clinical and forensic toxicology: qui modus in rebus est?
  46. Great need for changes in higher education in Greece
  47. A note from the Editor in Chief regarding the Letter to the Editor “Great need for changes in higher education in Greece”
https://doi.org/10.1515/cclm-2016-1012
Keywords for this article
faecal calprotectin; immunoassays; inflammatory bowel disease
Creative Commons
BY-NC-ND 3.0

Articles in the same Issue

  1. Frontmatter
  2. Editorials
  3. Reporting LDL-cholesterol levels in the era of intensive lipid management: a clarion call
  4. The challenges of genetic risk scores for the prediction of coronary heart disease
  5. Reviews
  6. Advanced lipoprotein testing for cardiovascular diseases risk assessment: a review of the novel approaches in lipoprotein profiling
  7. A review of the challenge in measuring and standardizing BCR-ABL1
  8. Mini Review
  9. Challenges in the analysis of epigenetic biomarkers in clinical samples
  10. Opinion Paper
  11. Defining a roadmap for harmonizing quality indicators in Laboratory Medicine: a consensus statement on behalf of the IFCC Working Group “Laboratory Error and Patient Safety” and EFLM Task and Finish Group “Performance specifications for the extra-analytical phases”
  12. Genetics and Molecular Diagnostics
  13. Assessment of EGFR mutation status using cell-free DNA from bronchoalveolar lavage fluid
  14. General Clinical Chemistry and Laboratory Medicine
  15. A survey of patients’ views from eight European countries of interpretive support from Specialists in Laboratory Medicine
  16. Verification of examination procedures in clinical laboratory for imprecision, trueness and diagnostic accuracy according to ISO 15189:2012: a pragmatic approach
  17. Expressing analytical performance from multi-sample evaluation in laboratory EQA
  18. A candidate reference method for serum potassium measurement by inductively coupled plasma mass spectrometry
  19. Practical motives are prominent in test-ordering in the Emergency Department
  20. Technical and clinical validation of the Greiner FC-Mix glycaemia tube
  21. Comparison of pneumatic tube system with manual transport for routine chemistry, hematology, coagulation and blood gas tests
  22. Accuracy of cerebrospinal fluid Aβ1-42 measurements: evaluation of pre-analytical factors using a novel Elecsys immunosassay
  23. Evaluation of cannabinoids concentration and stability in standardized preparations of cannabis tea and cannabis oil by ultra-high performance liquid chromatography tandem mass spectrometry
  24. Analytical performance and diagnostic accuracy of six different faecal calprotectin assays in inflammatory bowel disease
  25. Novel immunoassays for detection of CUZD1 autoantibodies in serum of patients with inflammatory bowel diseases
  26. Hematology and Coagulation
  27. Critical appraisal of discriminant formulas for distinguishing thalassemia from iron deficiency in patients with microcytic anemia
  28. Reference Values and Biological Variations
  29. Reference ranges of thromboelastometry in healthy full-term and pre-term neonates
  30. Cancer Diagnostics
  31. Immunoparesis in IgM gammopathies as a useful biomarker to predict disease progression
  32. Cardiovascular Diseases
  33. Assessment of the clinical utility of adding common single nucleotide polymorphism genetic scores to classical risk factor algorithms in coronary heart disease risk prediction in UK men
  34. Time and age dependent decrease of NT-proBNP after septal myectomy in hypertrophic obstructive cardiomyopathy
  35. Infectious Diseases
  36. Higher serum caspase-cleaved cytokeratin-18 levels during the first week of sepsis diagnosis in non-survivor patients
  37. Letters to the Editor
  38. Data mining for age-related TSH reference intervals in adulthood
  39. Intra-laboratory variation and its effect on gestational diabetes diagnosis
  40. Evaluation of long-term imprecision of automated complete blood cell count on the Sysmex XN-9000 system
  41. Sensitivity of the Sysmex XN9000 WPC-channel for detection of monoclonal B-cell populations
  42. Evaluation of biotin interference on immunoassays: new data for troponin I, digoxin, NT-Pro-BNP, and progesterone
  43. Stability of procalcitonin in cerebrospinal fluid
  44. Between-laboratory analysis of IgG antibodies against Aspergillus fumigatus in paired quality control samples
  45. Mass spectrometry vs. immunoassay in clinical and forensic toxicology: qui modus in rebus est?
  46. Great need for changes in higher education in Greece
  47. A note from the Editor in Chief regarding the Letter to the Editor “Great need for changes in higher education in Greece”
Sign up now to receive a 20% welcome discount
Institutional Access
How does access work?
Have an idea on how to improve our website?
Please write us.
© 2025 De Gruyter Brill
Downloaded on 30.12.2025 from https://www.degruyterbrill.com/document/doi/10.1515/cclm-2016-1012/html
Scroll to top button