Automated assessment of immunofixations with deep neural networks

Detection of individual lanes

First, in the immunofixation image each individual lane (IgA, IgG, IgM, Kappa, and Lambda) must be identified and marked with a bounding box (Figure 1). For this the serum albumin band in the ELP lane was chosen as a reference point. Since all lanes are always in the same position relative to each other, the bounding boxes can be determined relative to the reference point. The image content within these bounding boxes are the bases for the assessment and are used as input for a Deep Neural Network.

Figure 1:

Example immunofixation image with the serum albumin reference position (green) and the bounding boxes of the individual lanes (red) shows a monoclonal gammopathy IgG type Kappa.

Automated assessment using deep neural networks

After lane detection, the five lanes of each image are passed through a Deep Neural Network system as shown in Figure 2. As Deep Neural Network a so-called ResNet18 was applied [6]. ResNet18 belongs to the class of Deep Convolutional Neural Networks. With a training dataset, described in the next section, the Neural Network has been taught to differentiate between the different types of monoclonal gammopathies as well as bi-/oligoclonal gammopathies, free chain gammopathies (FC) and non-pathological cases (Negative). Additionally, the system assigns the label “Inconclusive” in cases where no clear assessment can be made. This results in a classification problem with 10 classes.

Figure 2:

Flowchart of the assessment system.

In addition to the class label, the assessment system provides a “confidence” score for the assigned class. This score can be used to further exclude cases where the assessment is not clear enough. In this work we used a threshold of 90% for an assessment to be accepted. This value was not optimized but taken simply as “being plausible”. We will see that the exact value is not critical.

Two datasets of immunofixation images provided by LADR Laboratory Group Dr. Kramer & Colleagues were used, one for training and the other one for testing the assessment system. The images were obtained with the HYDRASYS 2 SCAN (Sebia, Lisses, France).

The patient samples came from the routine analysis of LADR Central Laboratory, a cross-sectoral large, specialized laboratory in northern Germany. 52% of the samples originated from women and 48% from men. The age composition of the patients studied was as follows: 20–29 years 3%, 30–39 years 4%, 40–49 years 7%, 50–59 years 11%, 60–69 years 20%, 70–79 years 23%, 80–89 years 25%, 90–99 years 6%. The sample pool was composed as follows regarding the initiator of the analysis: 24.3% ordered from outpatient internal medicine, 3.2% from outpatient hematology/oncology, 30.7% from outpatient general practitioners, 15.2% from outpatient neurology, and 4.5% from outpatient orthopedics. 12.6% of the orders originated from inpatient hospitals including a hospital of maximum care and specialized hematology sections as well as 9% from medical laboratories. 18% of the images originated from follow-up analyses.

Each image was assigned a label designating the finding provided for the respective image. These labels included monoclonal gammopathies (IgA-Kappa, IgA-Lambda, IgG-Kappa, IgG-Lambda, IgM-Kappa, and IgM-Lambda), bi- or oligoclonal gammopathies, free chain gammopathies, non-pathological cases. The labels were obtained from manual optical evaluation of the immunofixation gels during normal laboratory workflow.

Additionally, the datasets included cases where no clear finding was found (Inconclusive). These were cases where pathology could not be excluded with certainty despite evaluation by two experts. These evaluations also do not only include the immunofixation images, but may include previous findings, clinical information, and supplementary examinations. These background informations were not available for the AI based assessment system, since the goal was to develop an automated assessment system based only on the immunofixaton image data.

The datasets for both training and testing of the AI were generated without particular selection of samples. The samples were picked from different time intervals to ensure there is no overlap between the datasets. The dataset used to train the AI included images from late 2020 and 2021. The dataset used for testing included images from late 2019 and early 2020.

Both datasets had a roughly equal distribution of findings. As can be seen in Table 1, except for the class “free chain gammopathy” with a rather rare occurrence, in both data sets the classes more or less have the same occurrence frequency. A two-sample Kolmogorov-Smirnov test provides a statistic of 0.0082 and a p-Value of 0.9761 showing a very high correlation between the distributions of the two datasets.

The about three times larger set was used for training the Neural Network, and the other set was used for an independent testing.

For the development of the AI models the PyTorch [7] framework was used (Torch version 1.7.1 and Torchvision version 0.8.2). The PyTorch implementation of ResNet18 with weights pretrained on ImageNet [8] was used as a base model. For the training of the models the hyperparameters were set to an initial learning rate of 0.001, a weight decay of 0.00005, and a batch size of 64. The Adam optimizer [9] with a momentum parameter of 0.9 was used. Categorical cross entropy [10] was chosen as the loss function. Training the models for 20 epochs took approximately 20 min on a NVIDIA GeForce GTX 1070 GPU (Nvidia Corporation, Santa Clara, USA).

Evaluation of the assessment system

For the evaluation of the assessment system, each image of the test dataset was passed through it. The results of this test are shown as a confusion matrix in Table 2. The analysis shows that the assessment system automatically assigns a clear label to 74.8% of the images. 25.2% of the cases are either labeled as being “inconclusive” or had a low “confidence” (less than 90%) along with its label. These images are flagged for additional manual review.

From the 765 cases with labels for monoclonal gammopathies (A–K, A–L, G–K, G–L, M–K, and M–L), 131 (17.1%) were sorted out as inconclusive or low-confidence and 9 (1.2%) were assessed differently then the expert label. These include one false-negative case (n) and one case where IgG-Lambda was “wrongly” assessed as a free chain gammopathy (FC). The corresponding images are shown in Figure3A and B. In 3B one can see artefacts in the image which, of course, disturb a correct assessment. In both cases, the laboratory personnel described the monoclonal gammopathies as “hinted”. The other seven cases were assessed as bi-/oligoclonal gammopathies (BO). Manual review showed that in all of these seven cases potential additional bands might indeed hint towards bi- or oligoclonal gammopathies. Three examples are shown in Figure 3C–E.

Figure 3:

Examples where AI assessment and expert label deviate.

(A) Negative (assessment) instead of IgM-Kappa (label); (B) free chain (assessment) instead of IgG-Lambda (label), note the artefacts in the image; (C-E) bi-/oligoclonal (assessment) instead of the labels IgG-Lambda (C), IgG-Lambda (D), and IgG-Kappa (E); (F) IgG-Lambda (assessment) instead of IgG-Lambda + Lambda (label); (G) IgG-Lambda (assessment) instead of IgG-Kappa + Lambda (label); (H) IgG-Kappa (assessment) instead of biclonal IgG-Kappa (label); (I) IgG-Lambda (assessment) instead of single-lambda (label); (J) IgA-Lambda (assessment) instead of single-lambda (label); (K) IgG-Lambda (assessment) instead of inconclusive (label); (L) IgG-Kappa (assessment) instead of inconclusive (label).

From the 181 samples with the label “bi-/oligoclonal gammopathy”, 70 (38.7%) were sorted out as inconclusive or low-confidence and 29 (16.0%) were assessed as rather being monoclonal gammopathies (A–K, A–L, G–K, G–L, M–K, and M–L). 25 of them were described by the laboratory personnel as having one major monoclonal gammopathy with additional bands that are “(very) weak”, “subtle” or “hinted”. One example is shown in Figure 3F. The remaining four cases also included two edge cases where, upon manual review, no obvious label could be assigned based on the scans used in this work. These images can be seen in Figure 3G and H.

From the 21 samples with the label “free chain gammopathy”, most (66.7%) were sorted out as inconclusive or low-confidence, and 2 (10%) were assessed as rather being “complete immunoglobuline-gammopathies”. In both cases the findings describe free light chains but also mention the corresponding complete immunoglobuline paraproteins that were found by the automated assessment as potential options. Additionally, one of the samples again shows a very weak gammopathy (Figure 3I and J).

The highest confusion is given between samples that were labeled as non-pathological and samples where no clear finding could be assigned by the laboratory personnel. 400 out of 2,811 (14.2%) samples with non-pathological labels were evaluated as “Inconclusive” by the automated assessment system. Conversely, 145 out of 638 (22.7%) samples with no clear finding were assessed as non-pathological. Additionally, there were 17 cases where the system assigned a pathological label to samples with no clear finding. Examples of all these cases are shown in Figure 3K and L. Note, that no false positive assessments were made.

Overall, among the 3,143 cases where a clear label (any label except inconclusive) was assigned by both the laboratory experts and the AI (with the AI having more than 90% “confidence”), the AI assessments matched the expert labels in 98.7% of all cases.

The 1.3% deviations are the subtle cases we discussed in the previous paragraphs. For non-pathological samples there were no false positives, thus resulting in an accuracy of 100% for this subset, and from the 967 pathological cases, one (0.1%) was wrongly assessed as being negative.

With the 90% “confidence” threshold, 188 (4.3%) of the 4,416 test samples are sorted out solely because of a low “confidence”. With a 85% “confidence” threshold, it is 3.8%, and with a 95% “confidence” threshold it is 5%. With the 85% “confidence” threshold, the AI assessments match the expert labels in 98.6% of all cases, and with the 95% “confidence” threshold it is 98.9%. This shows that the choice of the “confidence” threshold in this regime is not very critical.