Band 2, Heft 2 - Special Issue: AI in Microscopy; Guest editors: Hubert Mantz and Paula Sampaio

Microscopy has revolutionized life sciences by enabling detailed visualization of cellular and subcellular processes. Recent advancements in microscope technology have enhanced our ability to capture complex biological events, generating vast amounts of high-dimensional data. While this opens new avenues for discovery, it also introduces significant challenges in data analysis and interpretation. Modern microscopes can produce terabytes of data in a single experiment, often combining multiple imaging modalities across three or four dimensions. Techniques such as light sheet microscopy (J. Huisken, J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. Stelzer, “Optical sectioning deep inside live embryos by selective plane illumination microscopy,” Science , vol. 305, no. 5686, pp. 1007–1009, 2004), STED (S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. , vol. 19, no. 11, pp. 780–782, 1994), STORM (M. J. Rust, M. Bates, and X. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM),” Nat. Methods , vol. 3, no. 10, pp. 793–795, 2006), and lattice light sheet microscopy (B.-C. Chen, et al ., “Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution,” Science , vol. 346, no. 6208, 2014) have dramatically improved spatial resolution and acquisition speeds. These approaches yield datasets reaching tens of terabytes or more, exposing the limitations of traditional manual and semi-automated analysis methods, which are time-consuming, biased, and struggle with the multidimensional nature of modern microscopy data. To address these challenges, researchers are increasingly adopting artificial intelligence (AI), particularly deep learning models such as convolutional neural networks and transformers (“Deep learning in microscopy,” Nat. Methods , 2019). AI-based tools can automate complex tasks like denoising (A. Krull, T. O. Buchholz, and F. Jug, “Noise2Void – learning denoising from single noisy images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 2129–2137), segmentation (J. Ma, Y. He, F. Li, L. Han, C. You, and B. Wang, “Segment anything in medical images,” Nat. Commun. , vol. 15, p. 654, 2024), and virtual staining (Y. N. Nygate, et al ., “Holographic virtual staining of individual biological cells,” Proc. Natl. Acad. Sci. (PNAS) , vol. 117, no. 17, 2020), adapting to diverse imaging conditions and enabling scalable analysis of large datasets. This VIEWS article presents the perspective of ZEISS on how AI is transforming microscopy workflows. We highlight practical applications through real-world case studies and discuss how emerging computational tools are accelerating scientific discovery by making sense of complex, high-volume image data.

Training of semantic segmentation models for material analysis requires micrographs as the inputs and their corresponding masks. In this scenario, it is quite unlikely that perfect masks will be drawn, especially at the edges of objects, and sometimes the amount of data that can be obtained is small, since only a few samples are available. These aspects make it very problematic to train a robust model. Therefore, we demonstrate in this work an easy-to-apply workflow for the improvement of semantic segmentation models of micrographs through the generation of synthetic microstructural images in conjunction with masks. The workflow only requires joining a few micrographs with their respective masks to create the input for a Vector Quantised-Variational AutoEncoder (VQ-VAE) model that includes an embedding space, which is trained such that a generative model (PixelCNN) learns the distribution of each input, transformed into discrete codes, and can be used to sample new codes. The latter will eventually be decoded by VQ-VAE to generate images alongside corresponding masks for semantic segmentation. To evaluate the quality of the generated synthetic data, we have trained U-Net models with different amounts of these synthetic data in conjunction with real data. These models were then evaluated using real microscopic images only. Additionally, we introduce a customized metric derived from the mean Intersection over Union (mIoU) that excludes the classes that are not part of the ground-truth mask when calculating the mIoU of all the classes. The proposed metric prevents a few falsely predicted pixels from greatly reducing the value of the mIoU. With the implemented workflow, we were able to achieve a time reduction in sample preparation and acquisition, as well as in image processing and labeling tasks. The idea behind this work is that the approach could be generalized to various types of image data such that it serves as a user-friendly solution for training models with a smaller number of real images.

High-resolution X-ray microscopy is crucial for non-destructive materials characterization, but achieving both high resolution and maintaining a wide field of view often necessitates time-consuming approaches. Deep learning super-resolution methods based on convolutional neural networks are bridging this gap to obtain high-resolution usable data for analysis from low-resolution images. This study evaluates a novel deep learning-based algorithm designed to overcome traditional limitations by learning a spatially varying point spread function from a set of registered low- and high-resolution image pairs. With a systematic methodology, we evaluated the algorithm’s superior performance in recovering features across a wide range of resolutions with increasing image quality degradation. It was also benchmarked against a classical iterative Richardson-Lucy deconvolution algorithm, and a well-known deep-learning-based super-resolution convolutional neural network SRCNN algorithm for the same images. Qualitative and quantitative evaluations using simulated foam phantoms showed that our algorithm shows excellent feature recovery, within 5 % of the ground truth, even for a large resolution ratio of 7:1 between the high- and low-resolution image pairs. Multiscale investigations on real data of porous material and a semiconductor device are also presented to highlight its feature recovery performance and versatility in real-world scenarios.

Counting objects is an important task in biomedical image analysis, for example to count cells in microscopy or microbial colonies in laboratory applications. Similar to most image analysis tasks, modern approaches to counting rely on deep learning, predominantly by solving an object detection or instance segmentation task and deriving the count from it. Here, we revisit regression-based counting, a simpler approach to counting, introducing a new method called STACC that makes use of the object size to derive the density map used as regression target. In our experiments, it consistently performs better than state-of-the-art methods automated for cell and microbial counting. We also provide a user-friendly tool to apply it to counting problems in practice.

Fluorescence microscopy is a critical tool in bio-cellular research, enabling the visualization of biological tissues and cellular structures. However, the inevitable aging of microscopes can degrade their performance posing challenges for long-term scientific investigations. In this study, we introduce µPIX , a personalized deep learning workflow based on a Generative Adversarial Network (GAN) utilizing a Pix2Pix architecture. The network is trained in a supervised manner to denoise images, optimize pre-processing for binary segmentation, and compensate for equipment aging. Our results, evaluated using standard image quality and binary segmentation metrics, demonstrate that µPIX outperforms popular deep learning architectures based on convolutional auto-encoder networks for similar tasks. Additionally, our generative model effectively rejuvenates older detectors to perform on par with newer ones, not only by improving image quality but also by preserving resolution in depth and maintaining a near-linear response between original and generated images in terms of pixel intensity (crucial for quantitative imaging). These findings suggest that generative deep learning approaches can significantly contribute to more sustainable, cost-effective microscopy, fostering continued innovation and discovery in biological research.

Deep learning-based image super resolution (SR) is an image processing technique designed to enhance the resolution of digital images. With the continuous improvement of methods and the growing availability of large real-world datasets, this technology has gained significant importance in a wide variety of research fields in recent years. In this paper, we present a comprehensive review of promising developments in deep learning-based image super resolution. First, we give an overview of contributions outside the field of microscopy before focusing on the specific application areas of light optical microscopy, fluorescence microscopy and scanning electron microscopy. Using selected examples, we demonstrate how the application of deep learning-based image super resolution techniques has resulted in substantial improvements to specific use cases. Additionally, we provide a structured analysis of the architectures used, evaluation metrics, error functions, and more. Finally, we discuss current trends, existing challenges, and offer guidance for selecting suitable methods.