Automated lesion detection in endoscopic imagery for small animal models – a pilot study

Objectives

The primary objective of this work is to evaluate and support the detection of colorectal cancer in mouse models through the use of image analysis and deep learning methods applied to colonoscopy videos obtained from mice. This research aims to investigate and establish an AI based approach for automatic detection and classifying colorectal tumors in mice using a validated tumor scoring system. By comparing automatic detections with findings of experts, we evaluate our approach and depict the usage in cancer research.

Mouse models

Small animal resp. mice models are important tools in cancer research, providing critical insights into tumor biology, genetic and environmental influences on cancer development, and the efficacy of potential therapies [9]. Due to their genetic similarities to humans, short reproductive cycles, and the possibility to manipulate their genome, mice serve as valuable models for studying various aspects of cancer. For example, Tentler et al. [10] discuss specific Patient-derived tumor xenografts (PDTX) disease examples and give an overview of the opportunities and limitations of models n cancer drug development. Hidalgo et al. [11] also discuss challenges and limitations in the field of PDX and introduce a European consortium dedicated to PDX models. Hung et al. [12] detail the creation of a mouse models for sporadic and metastatic colon tumors, and its application in evaluating drug treatments.

Colonoscopy for mice

Colonoscopy is an essential technique in gastrointestinal patient care as well as research, it provides a direct visualization of the colon’s interior. In mouse models, colonoscopy is currently used to study various gastrointestinal diseases such as colon cancer [5], 13] and monitor the disease progression [14], [15], [16]. Olson et al. [13] have investigated the usage of colonoscopy for colonic diseases in animal models and they suggest that less mice models are needed due to serial investigations of animals. Kodani et al. [16] have developed and proposed a procedure for documenting the endoscopy of mice and its results. They introduced a novel scoring system consists of three parts: (1) assessment of the extent and severity of colorectal inflammation, (2) quantitative recording of tumor lesions, and (3) numerical sorting of clinical cases by their pathological and research relevance.

Deep-learning-based lesion detection

Early diagnosis and treatment of gastrointestinal diseases is primary based on lesion detection during an endoscopic examination of the gastrointestinal tract and the outcome of the examination is mainly depending on the expertise of the endoscopist. In the past years machine learning and deep learning approaches, and especially convolutional neural networks (DCNNs), have demonstrated their potential in automatic and enhanced lesion detection [17]. Wittenberg et al. [7] provide an overview of history and current developments in the field of automated machine-leaning-based polyp detection over the past 25 years, they outline the three development stages of AI-based lesion detection colonoscopy. Early experiments (phase 1) utilized handcrafted geometric features and simple decision schemes for the detection of prominent pedicled polyps. This phase was followed by texture-based approaches employing machine learning methods for tissue classification (phase 2). The latest advancements (phase 3) involve featureless methods, relying on deep convolutional deep neural networks for automated polyp detection and classification.

Currently a lot of research regarding novel DCNN architectures (such as YOLO networks) can be observed in this specific field, related to the wide availability of various public colonoscopy and endoscopy image data collections [18], [19], [20], [21] for network training and testing. Besides using the standard scores (F1, precision, accuracy) for evaluation, the run times of the different network types are of interest as well as the hardware used.

Recent studies in automatic polyp detection indicate that the one stage YOLO (‘you only look once’) architecture by Redmon et al. [22] is capable of achieving both high performance scores and real-time inference. Pacal et al. [23] recently improved the performance of their YOLOv3 and YOLOv4 network architectures for polyp detection by employing an advanced augmentation processes during the training step to virtually increase the amount of training data. Similarly, Li et al. [24] utilized large training datasets to achieve sensitivity, specificity, and F1-scores of 74.1 , 85.1, and 83.3 %, respectively. They also demonstrated that a YOLOv3 architecture can process up to 61.2 frames per second on a single Nvidia RTX2070. Recently, Wan et al. [25] evaluated the YOLOv5 on the Kvasir dataset as well as their own collection of colonoscopy data, achieving an F1-score of 0.907. Oliveira et al. [26] trained three versions of a YOLO network, namely YOLOv5, YOLOv7, and YOLOv8 architecture initially on the PICCOLO [18] dataset and then additionally with a comprehensive collection of polyp images. They achieved an accuracy of 92.2 %, a sensitivity of 69 %, an F1-score of 74 % and a mAP of 76.8 %.

Animals and protocols

Both female and male mice were included in the study. These mice were housed in specific pathogen-free conditions and had unrestricted access to a standard diet and water. They underwent routine pathogen screening following the guidelines of the Federation of European Laboratory Animal Science Associations (FELASA) [27]. All experiments were approved by the government of Lower Franconia and followed current guidelines for animal experiments.

To induce colorectal cancer (CRC), syngeneic tumor cells (KPN or VAKPT organoids) [28] were injected orthotopically into the colon wall using a high-resolution micro-endoscopy instrument (see Section “Micro endoscopy system”) while the mice were under anesthesia with Isoflurane. Following the injection (week 0), the mice were monitored on a weekly basis using the micro-endoscopy system until reaching the endpoint at week 6. During these regular follow-ups, high-definition (HD) endoscopic videos were recorded and stored digitally.

Micro endoscopy system

The experimental micro-endoscopy system (Karl-Storz, Tuttlingen, Germany) used for the injection of the syngeneic tumor cells as well as for the regular visual screening and data recording of the mice colonoscopy is depicted in Figure 1 [8]. The setup consist of a camera head (1) attached to a rigid endoscope (2), combined gas source needed for dilating the colon together with a light source (3) required for illumination which is connected by a glass fiber bundle to the endoscope, the endoscopy control system including USB storage device (4), a digital flat-screen monitor to depict the captured video data in real time (5), a Plexiglas chamber to narcotise the mice with Isoflurane gas (6), an absorption filter for the gas (7) and a Falcon tube where the mouse is exposed to the anesthesia (8).

Figure 1:

Setup of the micro-colonoscopy system showing the equipment used for endoscopic mouse examinations camera head with (1) attached rigid endoscope (2), light and gas source (3), endoscopy system with USB storage (4), flat-screen monitor (5), Plexiglas chamber to narcotise mice with Isoflurane gas (6), absorption filter for the gas (7), and Falcon tube (8).

Image data

From the division of molecular and experimental surgery of the Universitätsklinikum Erlangen we received n_seq=150 colonoscopic videos of mice, captured with the micro-endoscopy device, described in Section “Micro endoscopy system”, and prepared with the protocol, defined in Section “Animals and protocols”. The videos were captured from n_mice=28 different animals and started at week 1 of the clinical study. Within this data collection n_tumor=125 videos contain at least one tumor and while n_notumor=25 depict none. In total, n_lesions=130 are depicted in the data. The collected videos have a total length of t_total=1:05:08 h. The average length is t _μ =26 s, with a standard deviation t _σ =26 s. The median length is t_median=18 s, with a minimum video duration of t_min=7 s and a maximum of t_max=3:30 min. All videos have a spatial resolution of 1,920 × 1,080 pixels and a frame rate of 30 frames per second. Every video was manually classified by a clinical expert according to a tumor score based on Becker et al. [6]. Each individual mouse was implanted with exactly one single tumor, and each video corresponds to one colonoscopy procedure of one mouse. Based on the video data, the experts classified the depicted tumor into one out of six possible classes c ∈ {0, 1, 2, 3, 4, 5}. To achieve this, they maneuvered the endoscope to examine the entire colon and assessed the depicted tumor’s size in relationship to the colon size. They also annotated the sub-sequence along with the corresponding areas where the tumor was visible. Figure 2 shows some example frames from the recorded videos with increasing tumor scores ranging from ‘0’ to ‘5’ (top left to bottom right).

Figure 2:

Example images for different tumor scores (“0”–“5”) from the received video sequences; (a) tumor score ‘0’, (b) tumor score ‘1’, (c) tumor score ‘2’, (d) tumor score ‘3’, (e) tumor score ‘4’, and (f) tumor score ‘5’.

Figure 3 depicts the distribution of the annotated tumor scores within the video data collection. The pie chart (left) shows the distribution of every score within the whole data set. The violin plot (right) reveals the trend of the distribution from the starting point week t₀=1 to the endpoint at week t₅=6.

Figure 3:

Distribution of different tumor scores of the available video sequences (left); distribution of tumor score from week 1 to 6 of the clinical study (right).

Network architecture

For our proposed lesion detection approach, we utilized a YOLOv7-tiny detection network proposed by Wang et al. [29]. The YOLOv7-tiny architecture demonstrates effective performance in lesion detection while ensuring real-time processing capabilities, even on lower-end hardware [8]. It delivers a good balance between performance and efficiency.

Training of the network

The network used was originally trained on a image data collection consisting of 36,596 human colonoscopic images with annotated lesions. The training data combines both publicly available colonoscopy images from the LDPolyp data collection [30] (21,839 images) as well as a non-public data collection (14,757 images) from the Deep Colonoscopy project [31].

During the training process of the YOLOv7-tiny network, a learning rate of 0.01 was used, along with the stochastic gradient descent (SGD) optimizer with a momentum of 0.937. Data augmentation techniques were automatically applied, including random variations of the weights for the HSV color space, with fractions of hue h=0.015, saturation s=0.7, and value v=0.4. The images were also subjected to random translations (±0.1 %), scaling (±0.5 %), as well as flipping (50 %). Additionally, the YOLO framework’s own augmenting steps were employed, such as mix-up (5 %) and creating a mosaic (100 %) with input data. The batch size for the training data was set to 32. After 200 epochs of training the network was tested on various publicly available datasets, namely the CVC-ClinicDB [32], the Kvasir [19], and the PICCOLO [18] datasets. Overall, the trained network achieved for human colonoscopic lesion a precision of prec=0.92, recall of rec=0.90, and a F1-score of 0.91, using an intersection over union (IoU) threshold of 30 %. These performance metrics demonstrate the effectiveness of the trained network in accurately detecting lesions in human colonosocopic image data.

Pre- and post-processing steps

As there exist some visual differences between colonoscopic images of ‘mice and men’ (Steinbeck [33]),such differences in the textures and vascularization of the colon tissue as well as missing flexures and missing haustria in the mice colon, some pre- and post processing steps were added to the DCNN lesion detection approach (see Section “Network architecture”).

Fecal debris

In some videos, there exist image segments that depict fecal debris (stool), see Figure 4b, which are often misclassified as tumors by the deep neural network, as fecal debris in mice colonoscopy depicts similarities to lesions in the human colons, hence resulting in false-positive detections. To address this issue, we included the artifact detector proposed by Kress et al. [35] into our system, which is able to detect organic artifacts such as fecal debris or blood as well as objects such as surgical instruments. Specifically, we utilized the ‘stool’ class from the segmentation network, considering only detected objects with a confidence threshold of θ conf debris = 0.01 . However, not all fecal debris parts are detected correctly by this approach. To further improve the accuracy, we applied an additional image analysis filter to identify ‘yellowish’ pixels. This was achieved by converting the RGB image to the HSV color space and extracting the pixels from the hue-channel within the range of h ∈ 20,65 . The extracted ‘yellowish’ pixels were then combined with the detections from the segmentation network. If the combined area exceeded 50 % of the bounding box area, the detection was classified as ‘stool’ instead of a ‘tumor’.

Figure 4:

Example images of the detection network and the stool detector. The green bounding box represents the prediction of the neural network. (a) True positive detected stool (stool augmented in pink), (b) false negative stool, and (c) false positive detected tumor.

Classification algorithm

According to Becker et al. [6] tumors in mice can be classified into six different groups according to the average diameter d of the lesion. For the classification of the detected lesions we approximate the average diameter d = (w + h)/2, where w is the width and h the height of the detected bounding box (see Figure 5). The approximated diameter d (in pixel) is then scaled by the diameter D of the endoscopic image yielding r=d/D. Using this ratio r, the detected lesions can be classified according to the tumor score S:

Figure 5:

Example images of lesion detection network and classification. The green bounding boxes represent the prediction of the neural network. Images (a–e) are correctly classified as lesions with tumor scores ‘1’ – ‘5’; (f) lesion classified falsely as tumor score ‘4’ instead of ‘3’.

During the imaging procedure the colonocope is withdrawn through the colon and hence the tumor size depends strongly on the viewing angle and the distance between endoscope tip and lesion. To detect and classify a tumor in the mouse colon, the endoscopist tries to keep the tip of the endoscope steady to capture a comprehensive view of the colon for visual classification (relationship tumor size to colon size). This specific time span was used to calculate the tumor score for our evaluation. The class c that was most frequently predicted by the algorithm within the subsequence was utilized for evaluation.

Detection results

The complete sequence of a video, S=1, …, N, includes all frames of the video, while the sub-sequence S_pred ⊂ S includes all frames where the AI predicts a tumor. S_label ⊂ S is defined as sub-sequence where an expert detects a tumor. A frame is defined as correct when the Intersection over Union (IoU) between the predicted region and the annotated region in this single frame is over 50 %. Otherwise if the predicted region has a IoU ≤50 % the frame is defined as incorrect. If the experts annotated a region in a frame but the AI detects no tumor the frame is classified as overseen. Based on the available image data and AI-based methods for lesion and debris detection in colonoscopic image sequences of mice, the following groups can be identified:

1. True-positive (TP): A sub-sequence is true-positive if more than 50 % of S_label is classified as correct, see example in Figure 4a.

2. False-positive (FP): A sub-sequence is false-positive if more than 50 % of S_pred is incorrectly predicted. Such objects are either remains of stool or tissue folds. See Figure 4b and c for examples.

3. False-negative (FN): A sub-sequence is false-negative if more than 50 % of S_label is overseen.

4. True negative (TN): A sequence is defined as true-negative if there is no S_label and S_pred in the sequence.

The performance metrics are calculated with the following equations

The inclusion of the fecal debris detector has significantly improved the performance of the lesion detection approach, as documented in Table 1. Including the fecal debris detection achieved a precision of prec=1.000, recall of rec=0.940, and an F1-score of 0.969.

Classification results

The performance metrics of the classification approach are presented in Table 2, while Figure 5 showcases true-positive examples for each tumor score and one example with an incorrect classification.

The confusion matrix (Figure 6) provides a visual representation of the classification performance.

Figure 6:

The confusion matrix provided depicts the classification of tumor scores, ranging from ‘0’ to ‘5’, as described by Becker et al. [6].