Leveraging transformers for semi-supervised pathogenicity prediction with soft labels

1 Introduction

The field of genomics has experienced a dramatic transformation with the arrival of Next-Generation Sequencing (NGS) technologies [1], leading to a surge in the volume and complexity of genetic data available for research and clinical applications. This wealth of data presents both opportunities and challenges, particularly in the identification of pathogenic genetic variants associated with various diseases [2]. Effective analysis of such extensive datasets requires the development of advanced computational methods capable of distinguishing pathogenic variants from benign ones with high accuracy.

In recent years, machine learning (ML) and deep learning (DL) have emerged as powerful tools in genomics, offering the ability to automatically extract meaningful patterns from complex and high-dimensional data [3]. Traditional approaches for variant pathogenicity prediction often rely on fixed biological features, such as conservation scores or functional annotations, which may not fully capture the underlying complexities of genetic data [4]. In contrast, DL models, particularly those leveraging architectures like transformers, have shown promise in capturing intricate relationships within data, enhancing the predictive capabilities of these models [5].

This paper presents a novel deep learning model that leverages the Feature Tokenizer Transformer (FTT) architecture [6] to predict the pathogenicity of genetic variants. Our model is designed to integrate both numerical and categorical features from genomic data, utilizing a semi-supervised learning framework [7] to make the best use of labeled and unlabeled data. By employing a combination of supervised learning for well-defined cases and unsupervised techniques for ambiguous examples, the model aims to provide a more nuanced understanding of variant pathogenicity.

Furthermore, our approach addresses the significant challenge of label uncertainty in genomic datasets, particularly in the context of variants of uncertain significance (VUS). By refining the classification process into a binary format, we enhance the model’s ability to discern between benign and pathogenic variants while accounting for the probabilistic nature of the predictions. This capability is crucial for advancing personalized medicine and improving clinical decision-making based on genomic information.

In the following sections, we detail the methodology, including data preprocessing, model architecture, and training strategies, and present a thorough evaluation of the model’s performance. The results underscore the potential of this DL model in enhancing the accuracy of pathogenicity predictions, providing a foundation for further research and clinical applications.

This work is based on the initial findings presented at the 2024 PACBB conference [8].

2 State of the art

Predicting the pathogenicity of genomic variants has long been a central task in bioinformatics and clinical genetics. Over time, numerous methodologies have emerged, drawing on an expanding catalog of annotations that include evolutionary conservation, biochemical properties, and functional assays, as well as collective knowledge encoded in publicly available databases. The following section provides an overview of representative tools and their underlying principles, accompanied by a measured examination of commonly discussed limitations. Many of these tools are continually evolving, so updates to their software or databases may address some of the issues described below.

ClinPred [9] is a machine learning-based tool that employs random forest and gradient boosting models over features derived from dbNSFP and gnomAD. It specializes in missense variants, demonstrating strong predictive performance for protein-altering substitutions and outperforming other ensemble methods. ClinPred is widely used in clinical and research pipelines, though its performance on noncoding or splice-altering mutations is limited due to its training data focus on missense variants. Additionally, because it relies on periodically updated databases, older static versions may lack newly discovered variants.

REVEL (Rare Exome Variant Ensemble Learner) [10] integrates outputs from 13 functional prediction tools, including SIFT, PolyPhen-2, MutationTaster, and PROVEAN, within a random forest model. Optimized for rare missense variants, REVEL demonstrates superior performance in distinguishing pathogenic from benign substitutions. However, as an ensemble model, its interpretability can be affected by systematic biases or disagreements among its constituent predictors.

MetaSVM and MetaLR [11] are ensemble-based classifiers that aggregate multiple functional scores, utilizing a support vector machine (MetaSVM) and logistic regression (MetaLR) to enhance predictive sensitivity and specificity. These tools perform well for coding variants but remain limited for large insertions/deletions (indels) and deep intronic variants, which lack comprehensive feature representation.

MAGPIE [12] extends predictive capabilities to synonymous, nonsynonymous, and certain noncoding variants using a machine learning pipeline trained on ClinVar. MAGPIE integrates population frequency data, structural annotations, and additional genomic features, making it a versatile method for clinical variant interpretation. However, its performance depends on the completeness of reference databases, and frequent retraining may be necessary to adapt to newly discovered variants.

Ensemble-based classifiers like VEST4 [13] and M-CAP [14] refine pathogenicity predictions by consolidating multiple computational scores. VEST4 prioritizes rare missense variants using supervised learning, while M-CAP employs gradient boosting to enhance discrimination of uncertain missense variants. Both tools demonstrate high accuracy but are limited by feature coverage variability across different genomic regions and populations.

Evolutionary conservation is central to predictors like MutationAssessor [15] and PrimateAI [16]. MutationAssessor derives functional impact from multiple sequence alignments, while PrimateAI utilizes primate-specific genomic data within a deep learning framework. Both methods excel at evaluating missense variants in highly conserved regions but provide limited insight into noncoding regions and structural variations.

Structure-based predictors, including PolyPhen-2 [17], SIFT4G [18], and LIST-S2 [19], leverage biochemical properties and protein structures to assess mutation effects. These tools are highly informative when reliable structural data exist but offer limited utility for variants in noncoding regions or within genes lacking high-resolution structural models.

Deep learning approaches, such as DANN [20], extend variant interpretation by encoding complex, nonlinear relationships across multiple genomic features. DANN improves classification for both coding and noncoding variants but requires extensive, well-labeled training data to prevent overfitting.

Overall, computational approaches have significantly improved the sensitivity and specificity of pathogenicity predictions, with many integrated into clinical workflows. However, persistent challenges remain, particularly in classifying non-missense variants, addressing the reliance on periodically updated databases, and resolving ambiguities in Variants of Uncertain Significance (VUS). The semi-supervised deep learning framework introduced in this paper seeks to address these challenges by systematically incorporating both well-labeled and uncertain data, broadening model coverage, and facilitating frequent updates as genomic knowledge expands.

3 Methodology

3.1 Dataset

The dataset employed to train the deep learning model was developed internally and is designed to reflect the typical format and content produced by advanced next-generation sequencing (NGS) analysis tools. The input data is in the form of a CSV file, derived from the annotation of a VCF file sourced from the ClinVar database, updated as of August 19, 2024. This file was processed using a custom-built bioinformatics pipeline. ClinVar, managed by the National Institutes of Health (NIH), compiles and curates information linking genetic variants to their clinical implications, drawing on submissions from research labs, clinical testing services, and expert review panels to provide comprehensive variant classifications and associated phenotypes [21].

3.1.1 Genetic analysis pipeline

This pipeline, which processes Illumina [22] germline whole-exome sequencing data, operates under the Nextflow workflow management system [23]. It is designed to ensure parallel task execution, reproducible results, and compatibility with diverse computing environments. Its primary steps include:

1. Quality Control and Preprocessing: Initial quality control is performed on the data to filter out low-quality reads, preparing it for deeper analysis.

2. Alignment: The Burrows-Wheeler Aligner-Maximum Exact Match (bwa-mem) [24] algorithm is employed to align sequencing reads to a reference genome, locating genetic variants.

3. Variant Calling: Advanced algorithms are used to identify a range of genetic variants, such as single nucleotide variants, insertions, deletions, and structural variants.

4. Annotation: In this step, the functional impact of genetic variants is analyzed and contextualized within biological and clinical frameworks. The present analysis begins at this stage, utilizing the VCF file from ClinVar as input. Annotation was conducted with Ensembl Variant Effect Predictor (VEP) v.109 [25], OpenCravat v. 2.2.9 [26], and TAPES v. 0.1 [27], each contributing distinct insights into variant significance.

3.1.2 Dataset composition

The resulting dataset is extensive, offering a thorough overview of the genomic data utilized for pathogenicity prediction. It comprises 376 columns, which are organized into six primary categories.

1. Basic Annotation: Variants are documented with essential genomic details, such as gene names, chromosomal positions, and allelic variations.

2. Pathogenicity Predictors: Computational tools integrated into VEP and OpenCravat assess the likelihood of a variant being pathogenic. These tools use various models and algorithms to evaluate the potential impact of variants on gene function and structure.

3. Population Allele Frequencies: Data on the frequency of specific alleles in various populations is incorporated. This information helps determine the rarity of variants, which is crucial for assessing their potential pathogenicity.

4. Clinical Information: Data linking genetic variations to clinical presentations, including patient symptoms and known disease associations, is included to evaluate the real-world impact of variants. This category is supported by information from ClinVar, which classifies variants based on clinical significance, expert reviews, and established guidelines.

5. Evolutionary Metrics: Conservation of genetic regions across species is assessed to understand their biological importance. Tools that evaluate evolutionary conservation help in understanding the functional relevance of genetic regions and highlight potential pathogenic consequences of mutations.

6. Other Annotation Sources: Additional insights are provided through integration with resources that offer valuable context on drug-gene interactions and literature-backed variant-disease associations.

The VEP utilizes a variety of plugins to analyze genetic variants. These plugins are designed to provide insights into different aspects of variant effects, such as pathogenicity, functional impact, evolutionary conservation, and gene function (refer to Table 1).

Table 1:

Ensembl VEP plugins grouped by functionality.

Category	Plugin examples	Description
Pathogenicity and functional impact prediction	CADD [28], FATHMM [29], FATHMM_MKL [30], REVEL [10], LoF [31], LoFtool [32], SpliceAI [33]	Predicts variant impact and pathogenicity using multiple models.
Conservation and evolutionary constraint	Conservation [34], AncestralAllele, PrimateAI, G2P [35]	Evaluates conservation and predicts functional consequences.
Phenotypic and disease association	DisGeNET [36], Mastermind [37], phenotypes	Links variants to diseases and phenotypes.
Gene function and expression	LOEUF [31], GO [38], miRNA	Provides insights into gene function and expression.
Splicing and protein structure	MaxEntScan [39], GeneSplicer [40], ProteinSeqs	Analyzes impacts on splicing and protein structure.
Variant frequency and population	dbNSFP [41], gnomADc [42], Gwava	Provides allele frequencies and population-based predictions.
Miscellaneous	IntAct [43], NearestGene, LocalID	Additional context like protein interactions and local IDs.

OpenCravat employs several annotators to enhance the interpretation of genetic variants. These annotators are designed to provide detailed information about variant pathogenicity, clinical relevance, and population frequencies. Annotators focused on functional impact prediction evaluate the potential deleterious effects of variants, whereas those addressing clinical relevance link variants to known diseases and cancer mutations. Additionally, OpenCravat includes resources for population frequency data and additional context from various databases, which support the assessment of variant rarity and its implications (refer to Table 2).

Table 2:

OpenCravat annotators grouped by functionality.

Category	Annotator examples	Description
Functional impact prediction	CScape coding [44], dbscSNV [45], ChasmPlus [46]	Assesses the impact of coding variants and splicing.
Clinical relevance and disease association	CIViC [47], COSMIC [48], cancer genome interpreter [49], CIViC gene	Links variants to diseases and cancer mutations.
Population frequency and GWAS	gnomAD gene, GWAS catalog [50], GRASP [51]	Provides allele frequencies and links to GWAS traits.
Conservation and evolutionary constraint	LINSIGHT [52], LOFtool, RVIS [53]	Scores conservation and mutation intolerance.
Gene and protein function	GO, NCBIGene [54], HG19	Provides insights into gene function and gene-related data.
Regulatory elements	Ensembl regulatory build [55], ESS gene	Annotates variants within regulatory elements.
Miscellaneous	dbSNP [56], MUPIT [57], LitVar [58]	Provides additional context including local IDs and variant effects.

In addition to the VEP and OpenCravat tools, which provide detailed insights into genetic variants through functional impact prediction and clinical relevance, TAPES is employed specifically for variant prioritization. TAPES uses American College of Medical Genetics and Genomics (ACMG) criteria [59] to systematically evaluate variants, focusing on distinguishing between those with pathogenic potential and those considered benign or of uncertain significance. The specific criteria are displayed in Figure 2.

3.1.3 Data preprocessing

The data preprocessing phase was essential in preparing the genomic data for our deep learning-based pathogenicity prediction model. The script used for this process applied multiple steps to ensure the data was well-suited for training (see Figure 1).

1. Loading Data: Data was loaded from CSV files in chunks, using custom converters to enforce data types and manage missing values. This method facilitated efficient memory usage and allowed processing of large genomic datasets.

2. Feature Categorization and Processing: Dataset features were grouped according to their type. Some features were one-hot encoded, others were binarized based on keyword presence, and specific features underwent custom processing depending on their characteristics.

3. Imputation and Scaling: A heuristic approach was taken for imputation, filling missing values with worst-case assumptions when domain knowledge supported it. In cases without such assumptions, the mean or median was used based on the feature’s distribution. Features with more than 25 % missing values were excluded to maintain data reliability. For scaling, we applied standard scaling to normally distributed features, while min-max scaling was used for non-normally distributed features, ensuring values ranged between zero and one. This dual approach minimized biases and optimized the data for various analytical methods.

4. Feature Selection: Features with significant missing data were discarded, and a variance threshold was applied to remove those with minimal variability, as such features are unlikely to enhance predictive performance. Given the dataset’s high dimensionality and the model’s ability to learn complex patterns, no additional feature selection was performed. The deep learning model is capable of handling large feature sets by identifying important patterns within the data. This strategy simplified preprocessing and allowed the model to leverage the full scope of available data, maximizing both information usage and predictive accuracy without compromising performance.

5. Categorical Encoding: Categorical variables were encoded using an ordinal method, especially for features with inherent rankings, such as confidence levels and annotations. The target variable was numerically encoded based on ACMG guidelines.

6. Final Dataset Assembly: The processed features, along with the encoded target variable, were consolidated to create the final dataset, fully prepared for model training. This preprocessing pipeline ensured the data was consistent, properly formatted, and optimized for the deep learning algorithms, ensuring that the model could efficiently learn from and generalize across the dataset.

Figure 1:

Data preprocessing pipeline.

Figure 2:

The chart categorizes each criterion by evidence type and strength for benign (left) or pathogenic (right) assertions. Abbreviations: BS (benign strong), BP (benign supporting), FH (family history), LOF (loss-of-function), MAF (minor allele frequency), path. (pathogenic), PM (pathogenic moderate), PP (pathogenic supporting), PS (pathogenic strong), PVS (pathogenic very strong). Extracted from [59].

4 Architecture and training

The model architecture chosen for our pathogenicity predictor is the FTT [6]. This selection was driven by the necessity to effectively handle both numerical and categorical features in our dataset, a key strength of the FTT architecture. In an FTT, after all features are properly encoded, they are tokenized using an embedding layer within the model, as depicted in Figure 3.

Figure 3:

Illustration of feature tokenization within the FTT architecture for k features and latent dimension d. Adapted from [6].

This embedding or tokenization process converts features into dense vectors of a predetermined size, allowing the model to capture more intricate patterns in the data. In the context of genomic data, this method is particularly advantageous for handling features such as gene sequences or categorical variables with a large number of unique values, as it enables the model to develop a more nuanced understanding of the data.

A dedicated ‘classification’ token ([CLS]) is then added at the beginning of the latent representation for each sample. This is a standard technique when utilizing transformers for classification tasks (as seen in [61]). The stack of tokens created through this process is passed through the model’s multiple transformer layers. Ultimately, the transformed [CLS] token is fed into a classification head, which consists of a simple linear layer that outputs the class probabilities. The overall architecture is summarized in Figure 4.

Figure 4:

Summary of FTT architecture. Extracted from [6].

Before feeding the data to the DL model for the training process and the subsequent evaluations, it was split with a 70/15/15 ratio between training, validation and testing datasets. This split was stratified to maintain the class imbalance consistent across the three splittings of the dataset.

5 Results and discussion

We developed a DL model to predict the pathogenicity of genetic variants, and its effectiveness was evaluated using a confusion matrix. The analysis considered the six specified labels, and aimed to distinguish between benign and pathogenic variants through a binary classification approach (refer to Table 3).

We also extracted the distribution of predictions for each target label before applying the final sigmoid layer used to binarize the output of the model. This is shown in Figure 5. The plot shows the great confidence the model has in its predictions, with only a negligible number of predictions lying in the interval between −2 and 2. This has no medical implications, but establishes clear predictions to test in future works.

1. The model demonstrated strong accuracy in correctly identifying hard labels, as shown by its results with ‘Benign auto’, ‘Benign’, and ‘Pathogenic’ categories. Accurately predicting ‘Pathogenic’ variants is particularly crucial due to the significant implications of misclassification.

2. For ‘VUS’, the model applied a sophisticated classification method, indicating its potential effectiveness in identifying pathogenic variants in uncertain cases.

3. The handling of ‘Likely Pathogenic’, ‘VUS’, and ‘Likely Benign’ variants appears to effectively simplify the soft label challenge into a binary format. However, due to the inherent uncertainty in these labels, additional study is necessary to refine their interpretation.

4. The strong confidence of the model predictions provides a great basis for testing when more hard-labeled data is publicly available.

Figure 5:

Violin plot showing the distribution of predictions, before applying sigmoid function, for each target label in the dataset.

Beyond these technical observations, there are potential implications for clinical practice. In particular, the semi-supervised approach could aid in reclassifying VUS by identifying those most likely to be pathogenic, thus guiding more focused laboratory investigations or functional studies. Likewise, the ability to integrate uncertain samples may eventually help genomics laboratories or clinical testing facilities reduce the time spent on manual curation, provided that large-scale prospective validations confirm the reliability of these semi-supervised predictions. However, because the data in this study derive primarily from public repositories without longitudinal follow-up, we do not claim immediate applicability for clinical diagnostics. Future studies that incorporate patient-level outcomes, cross-reference the model’s predictions with established clinical reports, and conduct experimental assays on borderline cases will be essential steps toward realizing this framework’s practical utility in genomic medicine.

We have performed a direct comparison of our method with ClinPred [9] and REVEL [10], both based on decision trees (Table 4). Our method achieves higher specificity and precision, at the cost of lower sensitivity and accuracy. This is due to the large number of “false negatives” predicted by our model (see Likely Pathogenic variants in Table 3). However, it is important to highlight a key difference between our approach and the one in [9]. While they consider some soft-labeled variants (Likely Pathogenic and Likely Benign) as ground truth, we handle them differently, classifying them dynamically as seen in 4.1. Rather than assuming certainty in these labels, our semi-supervised approach allows the model to learn from them adaptively, refining predictions based on broader patterns in the data. This not only reduces the risk of propagating label noise but also enhances the model’s ability to generalize to novel variants with uncertain classifications. Future work could compare the performance of ClinPred under a similar semi-supervised setup, which would provide valuable insights into the trade-offs between precision and recall in pathogenicity prediction.

6 Conclusions

This study introduced a DL model designed to predict the pathogenicity of genetic variants, a crucial advancement for personalized medicine and the integration of genomics into clinical practice. The model employs a semi-supervised learning strategy combined with the FTT architecture to manage the inherent complexities of genomic data. Our results indicate that the model can effectively classify genetic variants ranging from benign to clearly pathogenic, although the full implications of the soft labels remain somewhat ambiguous.

Although the initial findings are encouraging, there are still areas for potential improvement. Future research will aim to enhance the model by expanding the training dataset, incorporating a wider array of genomic features, and improving its ability to detect subtle genetic variations. Moreover, continued validation through functional studies and clinical correlations will be necessary to ensure the model’s applicability and reliability in real-world clinical environments.