EAP4EMSIG – enhancing event-driven microscopy for microfluidic single-cell analysis

3.1.1 Defining “perfect focus” and ground truth acquisition

The ground truth for our model was established in a two-stage process. First, for each field of view, an experienced microscopist manually adjusted the objective to find the visually optimal focus. This position served as the center-point (z ₀) for the subsequent automated acquisition. A symmetric z-stack was then acquired around this point. For annotation, the mid-frame of this stack was systematically defined as the ‘perfect focus’ and all other frames were labeled with their known physical distance from this center.

3.1.2 Model architecture and feature extraction

We selected an MLP as our NN architecture, prioritizing both computational efficiency and inherent ease of use. This architecture offers a balance between predictive performance and ease of implementation, making it highly accessible for domain experts such as microbiologists. Its simpler structure allows for more straightforward understanding, troubleshooting, or retraining as new data becomes available. In contrast to more complex architectures like transformers, which, while powerful, often come with higher computational demands and a more intricate setup, our choice of an MLP provides a more efficient and user-friendly solution. This facilitates transparent integration into experimental pipelines and supports broader adoption by the scientific community.

Our MLP takes as input features from a single image (the current, potentially out-of-focus view). We extract a set of features known to correlate with image focus:

1. Image Pyramids: Laplacian and Gaussian pyramids capture multiscale edge and intensity information.

2. Wavelet Transforms: Bi-orthogonal wavelets (types 1.1 and 1.3) analyze texture and detail at different frequencies and orientations.

3. Image Characteristics: Information about image orientation and resolution are also included as features.

These extracted features form the input vector for the MLP. The MLP consists of two hidden layers with Rectified Linear Unit (ReLU) activation functions, AdamW optimizer with a learning rate of 3 × 10⁻⁴ and an output layer that predicts a single continuous value: the focus offset (in µm), including its direction (positive or negative), required to reach the optimal focal plane.

3.1.3 Training and validation

To enhance robustness, data augmentation techniques, including random flips and rotations, were applied during training. The model was trained using Mean Absolute Error (MAE) as the loss function, and K-fold cross-validation was employed to ensure robustness.

3.1.4 Real-time operation

In operation, the trained MLP predicts the focus offset from the features of a single, currently acquired image in under 100 ms. This predicted offset is then relayed to the microscope control module. While a single prediction is designed to bring the image close to optimal focus, often within the depth of the field, an iterative process is implemented for enhanced robustness. This iterative capability enables the system to reliably correct larger initial deviations from focus and actively compensate for potential focal drift resulting from environmental factors or mechanical settling over extended experimental durations.

3.3.1 Event detection

The event detection submodule is designed to identify predefined biological or technical occurrences at various levels (cultivation chip, chamber or individual cell). Events are defined based on rules specified by domain experts. These expert-defined rules are typically implemented as logical conditions or thresholds applied to quantitative data extracted from the image processing pipeline (e.g., cell count exceeding a defined limit, growth rate changes surpassing a specific speed, or image quality metrics from the autofocus module falling below a set value for a defined duration). These events can be categorized into two main types:

1. Technical Events: Issues such as focus loss (characterized by persistent large offsets reported by the autofocus system or a decline in image quality metrics below acceptable thresholds), chamber defects (identified through image analysis that assesses the integrity of the chamber, revealing structural issues that may compromise performance), or fluidic anomalies can occur and potentially disrupt the imaging process and affect sample quality. For detecting such focus loss events using image-derived features, we define the following rule: an out-of-focus event is triggered if the MLP-based autofocus module reports a predicted offset greater than 0.5 µm for three consecutive time points in a given chamber.

2. Biological Events: Significant changes in microbial behavior, such as rapid alterations in growth rates, cell death exceeding a certain percentage (identified via morphological changes or specific stains if used), or specific morphological transitions. For example, to identify a rapid growth event, indicative of accelerated proliferation, we establish a detection rule. This event is triggered if the cell count (i.e., the number of individual cells identified through instance segmentation by the real-time segmentation module) shows an increase of at least 10 % relative to the preceding time point for a given chamber.

Upon detection of events, the system can trigger automated responses (via the real-time experiment planner module) or notify the user, for instance, via Slack^[1] messages to a dedicated channel, enabling timely intervention.

3.3.2 Data analysis dashboard

The dashboard provides a user-friendly interface for biologists to monitor experiments, visualize data in real-time and manage experimental parameters. Key methodological considerations in its design include:

1. Modular Architecture: Ensuring new features, visualizations, or control elements can be integrated without overhauling the existing codebase, making it adaptable to diverse experimental needs.

2. Intuitive User Interface: Designed for ease of use by biologists, offering clear visualizations (e.g., heatmaps of chip status, time-series plots of cellular metrics per chamber) and straightforward controls.

3. Real-time Feedback: Displaying up-to-date information on experiment status, key cellular metrics (cell count, size, growth rate, focus score), and detected events.

4. Interactive Control: Allowing users to adjust experimental parameters or trigger specific actions based on observed data, facilitating biologist-in-the-loop operation.

4.1.1 Experimental setup

The images were captured using a Nikon TI microscope equipped with a Nikon Plan Apo λ 20×/0.75 NA objective (air immersion). For each field of view, a z-stack was acquired with a defined step size of 0.1 µm over a range of −5 µm to 5 µm. It is essential to note that these images were acquired under low-variance conditions, originating from a single, long-running experiment type where factors such as the microfluidic chip design, illumination, and temperature remained consistent. To ensure the physical capability for fine-grained adjustments, we confirmed with the manufacturer (Nikon) that the microscope’s z-drive features a mechanical step resolution of 0.025 µm and a positioning accuracy of 0.065 µm. Initially, 5 % of the total 13,000 high-resolution images (2,560 × 2,160 pixels) were reserved for testing the autofocusing MLP-based model (see Section 3.1) using a stack-based splitting strategy to prevent data leakage, ensuring that all frames from the same experiment remained grouped together. Subsequently, a further split was applied to the remaining 95 % of the dataset, resulting in 76 % of the total data used for training and 19 % for validation.

Model performance was primarily assessed using the MAE by comparing the predicted z-offset values with the ground truth z-offsets. All models were implemented using TensorFlow/Keras and trained and evaluated on an NVIDIA RTX 3090 GPU, reflecting a realistic laboratory hardware deployment.

4.1.2 Results and discussion

The performance of the trained MLP-based autofocusing model on the independent test set is detailed in Figure 2 to 4. This scatter plot of predicted values versus ground truth focusing offset (see Figure 2) demonstrates a strong linear correlation. The model achieved a high coefficient of determination R ² of 0.998, indicating that the predictions align closely with the ideal one-to-one relationship. The overall MAE for the test set was 0.105 µm. These results indicate a precise system, with the reported MAE is remarkably close to the 0.1 µm z-stack sampling interval used for training. This sub-step-size accuracy is possible because the MLP, trained as a regressor, learns to interpolate the optimal focus position by analyzing continuous changes in image features. This predictive capability is physically supported by the microscope’s hardware, which has a positioning accuracy (0.065 µm) finer than the sampling interval.

Figure 2:

Comparison of predicted and actual focus offset values. The scatter plot shows the focus offset predicted by the MLP model (Y-axis) in relation to the ground truth values (X-axis) for the test dataset. The red dashed line represents a perfect match. The model achieves a high determination coefficient R ² of 0.998 and a MAE of 0.105 µm.

Figure 3:

Distribution of the absolute prediction error. The histogram shows the frequency distributions of absolute errors on the test dataset. The right-skewed distribution illustrates that the majority of prediction errors are very small, clustering near zero, while larger errors are infrequent.

Figure 4:

Absolute prediction error by percentile. The figure quantifies the error distribution on the test dataset. The curve shows the absolute error values for each percentile. For instance, the median error (50th percentile) is 0.074 µm, while 75 % of all errors are below 0.153 µm. The 99th percentile is 0.44 µm.

A more detailed analysis of the prediction error is provided in Figure 4. The histogram of absolute errors (see Figure 3) shows a right-skewed distribution, with the vast majority of errors being very small, bellow 0.2 µm, confirming that substantial prediction errors are rare.

The MLP-based autofocusing model, trained as described in Section 3.1.3, achieved a final test MAE of 0.105 µm with a standard deviation of 0.15 µm. The 25th, 50th (median), and 90th percentiles of the absolute error were 0.036 µm, 0.074 µm, and 0.232 µm, respectively, with a 99th percentile of 0.440 µm and a maximum observed error of 0.68 µm (see Figure 3). These results indicate a generally precise system: most prediction errors fall well below 0.153 µm, which is tolerably close to z-stack acquisition step size of 0.1 µm and within a fraction of the microscope’s mechanical step resolution of 0.025 µm (see Figure 4).

In terms of speed, the model predicts focus adjustments with an average inference time of 87 ms per image on the GPU, meeting our real-time requirements. This performance, combined with the low MAE, underscores the model’s suitability for closed-loop focus control in high-throughput MLCI experiments.

The presented low MAE and high speed demonstrate the model’s effectiveness for its intended application. However, it is crucial to frame these results within the context of the model’s design and training data. The MLP-based approach was trained on data with low experimental variance (see Section 3.1.3) and therefore performs as an experiment-specific model.

Its primary strength lies in maintaining focus during long-running experiments where the core setup (e.g., chip type, illumination, temperature) remains stable. In such scenarios, the model can reliably compensate for focus drift over hours or days. Conversely, the model is not designed to generalize across different experimental setups. If a user changes the chip or significantly alters other environmental conditions, its performance would likely degrade, necessitating retraining. This highlights a deliberate trade-off: the current MLP provides a real-time, computationally inexpensive solution for stable experiments, while a more general-purpose autofocusing tool would require more complex models (e.g., CNNs) and a significantly more diverse training dataset.

4.2.1 Experimental setup

The performance of eleven SOTA DL segmentation methods (as categorized in Section 2.3) was evaluated on the benchmark dataset presented by Seiffarth et al. [42], [46], which contains 4,000 images of Corynebacterium glutamicum microcolonies from five video sequences, representing typical MLCI experiments. Ground truth instance segmentation masks are provided with the dataset. The benchmark was performed on an Ubuntu 22.04 workstation with an Intel Core i9-13900 CPU, an NVIDIA RTX 3090 GPU, and 64 GB RAM. All models were evaluated using their default settings to ensure a fair comparison of their out-of-the-box capabilities.

Segmentation accuracy was assessed using Average Precision (AP), including AP@50 and AP@75 and PQ [47], which comprises SQ and Recognition Quality (RQ). These metrics were calculated using TorchMetrics. As AP-based metrics require confidence scores, they could not be computed for all evaluated methods. Average inference times were measured using 32 bit floating-point precision, defined as the duration from inputting the image to receiving the model’s prediction as an instance mask, including any necessary post-processing.

4.2.2 Results and discussion

The qualitative and quantitative results of the segmentation benchmark are presented in Figure 5 and Table 2, respectively.

Figure 5:

The original image (a) and the zero-shot instance segmentation predictions for one sample image from [42] (b to l). (a) Original. (b) Omnipose [33]. (c) Distance [31]. (d) CellPose 3 [35]. (e) StarDist [34]. (f) CPN [36]. (g) SAM [37]. (h) SAM 2 [38]. (i) SAM 2.1 [38]. (j) 4M21 [41]. (k) Florence 2 [39]. (l) BiomedParse [40].

Among the evaluated methods, Cellpose 3 achieved the highest PQ of 93.58 % and RQ of 99.46 %, largely due to its automatic cell diameter estimation. However, this came at the cost of significantly longer inference times (1,115 ms), making it nearly ten times slower than the Distance-based method. The Distance-based method, while achieving a slightly lower PQ of 93.02 %, was the fastest among the highly accurate models at 121 ms. This substantial speed advantage is likely due to its comparatively simpler model architecture, which requires fewer computational steps than the more complex network used by Cellpose 3. Omnipose also performed well with a PQ of 93.36 % and an inference time of 271 ms. Visually, the results from Cellpose 3, Omnipose and the Distance-based method were nearly indistinguishable, all reliably detecting microcolonies and their constituent cells. Given its speed and high accuracy, the Distance-based method presents the best option for real-time applications, followed closely by Omnipose.

As anticipated, the Foundation Models (FMs) (SAM, SAM 2, SAM 2.1, 4M21, Florence-2, BiomedParse) were generally unsuitable for this specific real-time instance segmentation task. They primarily identified the microcolony as a whole rather than resolving individual cells, and their inference times were often unacceptably high (e.g., >100 s for 4M21). This is likely due to these models being trained on vastly different and broader datasets, not optimized for the fine-grained instance segmentation of small, densely packed microbial cells without specific prompting or fine-tuning. BiomedParse, trained on medical objects like organs, failed to segment microcolonies effectively. StarDist, optimized for bright objects on dark backgrounds, struggled with the dark microcolonies against a dark background in our dataset. CPN, while faster than Cellpose 3, had difficulty with densely packed regions.

Regarding the PQ metric, we acknowledge that for very small and densely packed objects like bacteria, PQ can be sensitive to minor contour inaccuracies, potentially impacting the absolute scores [48]. However, PQ is a widely adopted metric in cell segmentation benchmarks, and its components (SQ and RQ) provide valuable insights into both segmentation and detection aspects [49], [50]. Furthermore, the relative performance rankings and the substantial differences in inference times observed in our benchmark remain informative for selecting suitable models. The AP metrics reported in Table 2, where available, offer an additional perspective on performance.

This benchmark highlights the critical trade-off between segmentation quality and inference speed, emphasizing the need to select models based on specific experimental requirements for real-time, event-driven MLCI.

4.3.1 Event detection demonstration

The event detection submodule was configured with rules to identify exemplary events. As stated in Section 3.3.1, an out-of-focus event was triggered if the MLP-based autofocus module reported a predicted offset greater than 0.5 µm for three consecutive time points of a specific chamber. A rapid growth event was triggered if the cell count in a specific chamber increased by more than 10 % within consecutive time points. Upon triggering, the system effectively generated and dispatched notifications to our Slack communication channel. The event detector itself requires only approximately 100 ms to check for new events within a chamber and send notifications via both Slack and internal logs. These log times are prominently displayed on our dashboard (Figures 6 and 7), showcasing the system’s ability to alert users to critical events in real-time and facilitating prompt intervention.

Figure 6:

EAP4EMSIG dashboard: experiment monitoring interface. The primary interface for monitoring experiments, highlighted in the red box, allows users to select the cultivation chip and microscope to initiate an experiment or load a predefined protocol. The heatmap, structured according to the cultivation chip’s layout, visually represents the status of each cultivation chamber based on the selected metric, in this case, cell count. The yellow box contains the context window that is displayed when hovering over the chamber. Clicking on a chamber opens a detailed view (blue box), which includes the raw region of interest and the segmented image.

Figure 7:

Detailed view of a selected chamber. The view displays time-series data for key metrics, including cell count, cell size, growth rate, and focus score, providing users with comprehensive insights into the chamber’s performance over time.

4.3.2 Dashboard functionality demonstration

The EAP4EMSIG dashboard (methodology detailed in Section 3.3.2) provides comprehensive visualization and control features, as demonstrated in Figures 6 and 7. Figure 6 (red box) presents the primary experiment monitoring interface, where users can select the cultivation chip and microscope to either initiate an experiment or load a predefined protocol. Experiment management is facilitated through control buttons labeled “Start”, “Pause/Resume”, and “Stop”. The heatmap, structured according to the cultivation chip’s layout, visually represents the status of each cultivation chamber based on selected metrics, including cell count, cell size, event occurrences, and focus score.

Hovering over a chamber on the heatmap reveals a context window that displays the current visual state of the chamber along with additional details (Figure 6 yellow box). As illustrated in Figures 6 and 7, clicking on a chamber opens a detailed view. This view includes the raw region of interest from the image alongside the segmented image, highlighting individual cells (Figure 6, blue box). Additionally, time-series data for key metrics, including cell count, cell size, growth rate, and focus score, are presented. These plots visualize the expected dynamics of microbial growth, such as the exponential increase in cell count, while derived metrics like the growth rate naturally exhibit higher volatility. A sidebar displays experiment metadata and allows for data export. This empowers users to track the history of individual chambers and filter messages, enabling in-depth analysis of ongoing experiments and supporting informed decision-making.

The presented functionalities demonstrate the dashboard’s capacity to provide biologist-in-the-loop control and real-time insights, significantly enhancing the ability to oversee and interpret complex MLCI experiments efficiently. The pipeline has been in routine operation since 01/2025 and is being continuously expanded based on user feedback.