Towards a more accurate and reliable evaluation of machine learning protein–protein interaction prediction model performance in the presence of unavoidable dataset biases

1 Introduction

Protein-protein interactions (PPIs) control most biological activities in living organisms, making them fundamental to cellular functions [1]. PPIs can be classified based on their composition, affinity and life time/stability [2], 3]. According to composition, they can be homo- and hetero-oligomeric complexes, if PPIs occur between identical chains, or if PPIs take place among non-identical chains [4]. Regarding affinity, they can be classified into non-obligate and obligate complexes. These differ in affinity, and components (protomers, monomers) of non-obligate interactions can exist independently in contrast with obligate complexes where the constituents are unstable on their own. According to lifetime and stability, PPIs can be permanent interactions (interactions that occur for a long period of time, such as those of metabolic and regulatory networks) or transient (interactions happen under certain biological contexts, in which association/dissociation occurs quickly, such as those of signaling and secretory pathways [5], 6]. PPIs identification is essential for the characterization of protein functions and to the understanding of the molecular mechanisms in the cells [7]. Such information is also essential to the identification of disease mechanisms, which facilitate the therapeutic target identification and novel drug design [8], 9]. Innovative techniques, including low-throughput methods like nuclear magnetic resonance spectroscopy and high-throughput approaches such as Yeast two-hybrid (Y2H) and Affinity Purification coupled with Mass Spectrometry [10], have transformed the identification of protein-protein interactions (PPI) across various organisms. However, challenges remain due to the time and economic costs associated with studying the extensive proteome of living organisms and the potential number of interactions of interest. For this reason, computational prediction of PPI using machine learning (ML) algorithms is gaining increasing relevance [11]. These algorithms are used not only as complementary evidence to experimental methods but also to predict novel interactions that can then be experimentally addressed. ML algorithms can use protein sequences, protein structure, and co-fractionation mass-spectrometry data to extract relevant features to make predictions [12].

Among machine learning (ML) approaches, deep learning (DL), can automatically extract effective features from large-scale original datasets without prior knowledge [12]. Deep neural networks (DNNs) are capable of identifying long-range interactions in data, such as the amino acid residues used in protein interactions in a protein sequence in 1D space, and thus are used for task classification [12]. These methodologies use patterns within the protein sequence such as amino acid combinations, and the protein sequence that define folding motifs, from known interacting polypeptides, that can be used to predict PPIs based on similarity of protein sequences [12]. Examples of PPI prediction sequence-based prediction methods that use DNN for capturing features of primary sequences in 1D space (Table 1 in [12]) are DeePPI (Deep neural networks for Protein–Protein Interactions prediction) [13], DNN-PPI (deep neural network framework) [14], and PIPR (Protein–Protein Interaction Prediction Based on Siamese Residual) [15]. DeePPI extracts high-level discriminative features from sequence specificities of DNA and RNA binding proteins, DNN-PPI capture features in high-dimensional position-specific profiles of protein sequences such as semantic associations between amino acids, position-related sequence segments (motif), and their long- and short-term dependencies, features learned automatically from protein primary sequences, and PIPR leverages on robust local features and contextualized information, and incorporates deep residual recurrent neural network for capturing features of primary sequences and do not use spatial information of protein molecules, in the Siamese architecture.

The datasets used for PPI prediction based on protein sequence in 1D space methods are made of reference protein sequences and data on whether two proteins interact or not, that can be split into training and validation sets. It should be noted that in order to determine the real performance of ML methods, the training and validation datasets should be carefully analyzed to avoid potential biases and pitfalls. Data on interacting proteins is obtained from databases such as STRING (Search Tool for the Retrieval of Interacting Genes/Proteins, https://string-db.org/) which provides known and predicted protein-protein interactions and is regularly updated by the authors [16]. Another well-known database used both at a general level and filtering by organisms whose content focuses on interactions between experimentally verified proteins is DIP (Database of Interacting Proteins, https://dip.doe-mbi.ucla.edu/dip/Main.cgi). There are also specialized databases such as HPRD (Human Protein Reference Database) [17] focused on human PPIs. It is also crucial to have knowledge of non-interacting proteins, although there are few databases focused on collecting such data. One such database is Negatome [18], which gathers information on protein pairs known not to interact with each other from scientific literature and experimental data.

The prediction accuracy of these primary sequences in 1D space methods has been reported to be higher than 95 %, which is surprising since they do not take into consideration the 3D structure of protein complexes, binding pockets, domains, surface residues and binding affinities [10]. Nevertheless, recently, it has been suggested that data leakage (sequence similarities and node degree information) in training and test sets are associated with high prediction accuracy, and that when data leakage is avoided performance becomes random [19], 20]. In this context, we conducted a previous study on biases in two protein–protein interaction (PPI) datasets that may overestimate the performance of machine learning models with random splits such as cross-validation [21]. We used pre-trained language models (SeqVec and ProtBert) to extract embeddings from protein sequences, combining them through concatenation, addition and multiplication. We trained several machine learning algorithms and developed a baseline model (PPIIBM) to predict protein interactions. Our results revealed a significant bias in the data, which could inflate performance metrics by relying on prior protein positivity (see Section 3).

Here, in this follow-up study, we present a metric that aims at better estimating the real performance in biased PPI datasets. We analyze whether the proposed metric is sufficient to avoid overestimating the power of PPI prediction based on protein sequence in 1D space methods, in comparison to ensuring that validation splits are only conformed of interactions of unseen proteins. Additionally, we incorporate ESM-2, a new language model for processing protein sequences.

2 Related work

Dealing with raw protein sequences in machine learning methods, as in natural language processing, often requires first extracting the relevant features that capture the full sequence information and embed it into a fixed-length space. Recently, DL word embedding methods are also being ported to the protein language, i.e. amino acid sequences.

We could differentiate among three generations of word embeddings. In the first generation, word embeddings are always the same regardless of the surrounding words in the document. Word2Vec [22], through its Skip-Gram model [23], is one of the most representative.

In the second generation, word embeddings became context-sensitive, meaning that the same words will have a different embedding, depending on the surrounding words. Recurrent neural networks, such as LSTM (Long Short-Term Memory) networks, were one of the most popular approaches. In the case of protein sequences, SeqVec [24], is a representative example of this generation, where ELMo (Embeddings from Language Models), a type of LSTM, were trained over proteins in UniRef50 database. Authors highlight the effectiveness of ELMo to capture biophysical properties of the proteins from unlabeled datasets (UniRef50). They tested the usefulness of their 1,024-length embeddings for several downstream tasks, including protein-level tasks, such as subcellular location prediction. The 1,024-length protein-level embeddings were obtained by averaging amino acid embeddings across the entire sequence.

The third generation takes advantage of the more recent transformer architectures [25] to model protein sequence’s language. Transformers are more efficient than previous recurrent neural networks for training, and are able to handle wider context windows, which enables them to model natural language with an unprecedented success. From the same team as SeqVec, ProtTrans [26] trained Transformer-XL and XLNet and four auto-encoder models (BERT, Albert, Electra, T5) in Uniref50, Uniref100 and BFD100. As with SeqVec, 1,024 fixed-length embeddings seem to capture biophysical features and authors have used them as feature extractors for two per-protein downstream prediction tasks, including the subcellular location prediction and membrane versus water-soluble prediction by using a single feed forward layer. Authors found that bi-directional models (auto-encoders), such as BERT, outperformed SeqVec in protein-level predictions tasks.

Along these lines, Meta AI’s ESM (Evolutionary Scale Modeling; https://github.com/facebookresearch/esm) also emerged, a family of language models developed specifically for biology and bioinformatics, focusing on protein sequences. These transformer-based models are designed to understand and predict the structures and functions of proteins from their amino acid sequences. ESM-2 represents the latest generation of the ESM family, where there are different models that vary in size and dimensionality of the embeddings (320–5,120 fixed-length embedding). In this case, we focus on an embedding size similar to that of ProtBert, so we will work with a 1,280 fixed-length embedding [27].

Focusing on the PPI prediction task, Chen et al. [15] developed a model named PIPR that pre-trains the Skip-Gram model [23] on protein sequences to obtain amino acid embeddings of the two input proteins, that are then directly fed into an specific end-to-end DL model, based on Siamese Residual Recurrent Convolutional Neural Networks (R-RCNN), able to extract features for the input protein sequence pairs, designed for several prediction tasks, including binary PPI prediction.

Song et al. [28] proposed another end-to-end DL framework, known as TAGPPI, which employed SeqVec [24] for amino acid embeddings, that are then used to (i) extract sequence features by using TextCNN and (ii) structural features, by using Graph Attention Networks (GAT) over the AlphaFold DB contact map predictions (https://alphafold.ebi.ac.uk), respectively. After fusing both feature sets, a final feature vector for each protein is obtained, and finally concatenated to feed a MLP for PPI prediction.

Wu et al. [29] proposed a new DL architecture, named DL-PPI, that relies on Skip-Gram and InceptionV3 to get protein-level embeddings that are then fed to Graph Neural Networks (GNNs) for relationship-level features of each protein, followed by a self-attention layer to concentrate in the more relevant information of each protein and a final Neural Tensor Network (NTN) for PPI prediction.

Among others, all these studies included the Yeast dataset [30], widely used as a benchmark dataset in this area. Predictive performance results in general are very high (e.g. PIPR reached 97 % of balanced accuracy). However, a recent work from Bernett et al. [19] just reported that almost all PPI datasets used for evaluating such approaches are randomly split into train and test sets using cross-validation. This fact has been already pointed out by other studies that showed that this causes an inflation of prediction performance due to training data leakage [21], 31]. Bernett et al. trained and evaluated six sophisticated DL sequence-based PPI prediction methods, including PIPR, on seven well-known datasets, demonstrating that the high levels of accuracy reported in academic papers can be entirely attributed to the leakage of training data. More concretely, they report that methods seem to (i) exploit node degree information, since their performance is unaffected when the PPI graph is rewired while preserving node degree and (ii) sequence similarity, since all models fell to random-level performance when train/tests splits were done in such a way that minimizes the inter-split protein sequence similarity. Remarkably, Bernett et al. [19], 32] also provided a gold standard dataset for sequence-based PPI prediction that was specifically designed to prevent data leakage and minimize biases due to sequence similarity.

In this context, studies such as TUnA (Transformer-based Uncertainty Aware Model for PPI prediction) [32] appear with the aim of predicting protein-protein interactions with high accuracy and whose model evaluation is reliable. TUnA uses protein embeddings generated by ESM-2 to capture rich and complex representations of protein sequences, together with Transformer-based architecture to process these representations to capture intra-protein and inter-protein relationships and a Spectral-normalized Neural Gaussian Process (SNGP) in the final layer to provide uncertainty estimates in the predictions.

3 Materials and methods

3.5 Per-protein utility metric

To evaluate the effect per-protein imbalance in performance metrics, here we propose to evaluate MCC at protein level during validation (Figure 1). First, all folds of the external CV are aggregated into a single confusion matrix. Then, for each protein, we considered only the interactions involving it and computed the MCC on the corresponding confusion submatrix. Then, we computed an average over all proteins, weighted by the number of interactions in which such protein appears in the validation set, to obtain the final performance measures adjusted by protein, that is, the per-protein MCC (pp_MCC). If a given protein does not have true positive interactions or does not have true negative interactions in the test set, the MCC is not computed (zero division, since recall or specificity has zero denominator) and this protein is skipped.

Figure 1:

Steps to compute the per-protein MCC.

We aimed at providing a kind of utility metric, useful in real scenarios, where, for example, multiple hypothetical ligands are searched for a “reference” protein. In such scenario, we want to get the performance for each protein, for example this “reference” protein. A model that only relies on the learned positivity of the first protein, such as the PPIIBM singleItemMode baseline model (see 3.3 in our previous work [21]) will predict all true or all false for the same protein, that is, random guesses (MCC = 0).

Finally, we have chosen the MCC metric as the base for pp_MCC, since it is a popular metric in similar PPI studies, but other metrics (such as F1, AUC, Balanced Accuracy, Youden Index, etc.) could be used and are easy to calculate inside the proposed framework.

4 Results and discussion

In general terms in the random splits setting, as we can see in Figure 2 (two left columns), we observe a decrease when comparing pp_MCC with raw MCC, evidencing that our metric robustly drops the raw MCC. However, none of the models reach a pp_MCC of 0, meaning that models seem to not only exploit the positivity bias present in the dataset. In this sense, not all models show the same performance. RF models tend to achieve superior performance compared to KNN and LR models in most combinations (embedding and combination) and metrics. Furthermore, the combination of concatenation with the RF model using the pretrained ESM-2 model produced the highest performance (MCC = 0.514). This combination was also the best in the pp_MCC metric (0.319).

Figure 2:

Comparison between the random splits (dark columns) and the unseen protein splits (soft columns) settings. Results are provided by combination (concatenation, addition, or multiply) and embedding (ProtBert or ESM-2 model).

Regarding embeddings, ESM-2 seems to have a slightly better performance compared to ProtBert, with the exception of the multiplication combinations in the KNN model. The average raw MCC for ProtBert is 0.303 compared to 0.331 for ESM-2, and for pp_MCC, the average is 0.183 for ProtBert and 0.196 for ESM-2. We have observed that, although the improvement by using ESM-2 is small, it is consistent, and thus embeddings seem to have influence in the final downstream performance.

We also wanted to evaluate the performance of the models in a more challenging setting, where the proteins present in the interactions of the validation split are not used for training, as explained before. As it can be observed in Figure 2, making unseen protein splits leads to worse performance compared to the previous random splits. We observe that, in general, the MCC and pp_MCC values for unseen protein splits are lower than for random splits, which is in line with the published literature stating that random splits allow the models to learn from the same protein, leading to overoptimistic performance. Interestingly the estimated performance obtained by using pp_MCC in the random split is very similar to the raw performance obtained in the unseen proteins split (second and third columns in Figure 2).

Moreover, in the random splits setting, we observe high differences in MCC and pp_MCC values, in comparison with the unseen protein splits setting, where the differences between both metrics are more moderate, suggesting more consistent performance and a more realistic evaluation of the model using the more challenging unseen protein split.

Figure 3 shows how both the split strategy and the use of the pp_MCC affect the obtained median performance. Starting from median values slightly above 0.3 in random splits with raw MCC, lowering to values around 0.2 when using protein split or pp_MCC, and reaching floor values around 0.15 if both strategies are combined. Figure 3 also illustrates these two effects: lower dispersion in unseen protein splits and closer MCC and pp_MCC. The differences between MCC and pp_MCC are also statistically significant (p-values <0.01 in a paired t-test in both settings).

Figure 3:

Distribution of MCC and pp_MCC for the random splits and unseen protein splits settings.

All these findings highlight the importance of considering bias in protein interaction databases and adapting evaluation strategies to obtain more robust and generalizable models, as traditional evaluations (i.e. non-adjusted metrics on random splits) tend to overestimate performance due to the per-protein unbalanced distribution of interactions.

When a well-constructed dataset is used, such as the gold standard dataset, the values of the raw MCC metric do not present the overestimations (Figure 4). This dataset has been carefully designed by the authors by minimizing similarities between proteins and removing redundancies, which prevents the common biases that artificially inflate model performance. In this context, the proposed pp_MCC metric demonstrates that its utility can also complement traditional metrics, as it reflects the same performance when the dataset is well-constructed.

Figure 4:

Comparison of MCC and pp_MCC across different models and combination strategies with ProtBert and ESM-2 embeddings on the gold standard dataset.

Furthermore, when analyzing the distribution of the MCC and pp_MCC metrics, as illustrated in Figure 5, pp_MCC consistently shows similar values compared to MCC, even in this controlled scenario. This highlights its robustness as a metric that adjusts model performance estimates, providing a more realistic perspective. While the primary objective of pp_MCC is to detect biases in poorly constructed datasets, these results confirm that its application is also suitable for validating models in contexts where the data has been carefully curated.

Figure 5:

Distribution of MCC and pp_MCC on the gold standard dataset.

The pp_MCC metric is not only effective in detecting biases in datasets but also ensures that model performance is not artificially inflated, offering a more reliable and generalizable evaluation in any scenario.

5 Conclusions

The pp_MCC metric proposed in this work offers a pathway to improve the reliability and interpretation of results in PPI prediction, promoting the development of more effective models. Here we have taken into account and addressed a specific bias in the PPI datasets related to the per-protein unbalanced positivity. As we expected, we obtained generally lower performance estimated after this methodological adjustment, showing their usefulness and highlighting the importance of considering bias in PPI databases and adapting evaluation strategies to obtain more robust and generalizable models, in line with other studies pointing in the same direction [17], 18], 29], 32].

We tested the pp_MCC in two splitting strategies. A random split and a more challenging unseen protein split, which prevents any interaction of the same protein from being seen in the training set, that also could mitigate the bias of identifying the protein and exploiting its positivity, but also hinders learning the biological manner in which that protein interacts with others. In this sense, since the pp_MCC aims to eliminate the effect of bias even in a random split, it allows to still train with other interactions of the to-be-predicted proteins, which could be interesting for some studies that are more protein-centric. For example, we could be interested in studying potential novel interactions of a given protein under study while allowing us to use models that were trained with other interactions of the same protein.

The adjusted performance values were still above random guess (pp_MCC > 0), meaning that models seem to capture real biological information during training. However, the presence of more biases and data leakages exploited by the models could not be absolutely discarded, which warrants further investigation. For example, datasets containing repeated or similar sequences, will produce very similar embeddings. If the positivity of repeated or similar proteins remains unbalanced and in the same direction of the to-be-predicted proteins, the bias would persist. In fact, similar proteins have a tendency to share their ligand networks. Therefore, if we want to evaluate the power of ML primary sequences in 1D space methods based on features other than high overall protein similarity, we should also exclude from the evaluation set the proteins sharing similarity with those used in the training set. This could be done using tools such as CD-HIT [35]. Building on this, we further tested the pp_MCC metric on a rigorously curated gold standard dataset that employed CD-HIT to remove sequence redundancy and minimize similarity between training, validation, and test sets. The results confirmed that both MCC and pp_MCC metrics displayed significantly lower values under these stringent conditions, highlighting the inherent limitations of models trained solely on primary sequence data. Nevertheless, the pp_MCC metric demonstrated its robustness by avoiding overestimated performance, even in the absence of detectable bias. This reinforces its value as a reliable evaluation tool, particularly on datasets that were not carefully split, where a discrepancy between raw metrics and the pp_MCC could indicate the presence of biases in the split. Our results underscore the importance of adopting rigorous data preprocessing and robust metrics to ensure unbiased and generalizable evaluations of protein–protein interaction prediction models.