Biomedical event extraction using pre-trained SciBERT

1.1 Overview

Along with technological developments in both the computational and biological fields, biomedical literature has also increased, making it difficult to retrieve information manually. This poses a new challenge in natural language processing (NLP) for extracting information from biomedical literature, which contains vocabulary not commonly used. Information extraction is a task to retrieve one or several facts in natural language, either in text or in sound, which are then represented in a structured form [1]. NLP is a branch of artificial intelligence that processes natural language, aiming to obtain the information or knowledge contained therein [2,3]. Biomedical NLP, also known as biomedical NLP, operates in the biomedical field and focuses on several tasks related to managing biomedical texts. One of these tasks is extracting information on biomedical events (biomedical event extraction) [4].

1.2 Biomedical event extraction and motivation

Event extraction is a form of information extraction from text data that identifies the entities involved in an event and the relationships between them [5,6]. The primary purpose of event extraction is to obtain 5W1H (“who,” “when,” “where,” “what,” “why,” and “how”) information from text data [5,6]. The use of event extraction, in general, and biomedical-specific domains has differences. In the biomedical domain, event extraction focuses on identifying interactions between biomolecules and other biomolecules [6]. Therefore, extraction often involves named-entity recognition (NER) and relation extraction (RE) [5–8]. Various approaches are used for event extraction, such as machine learning, but the key to each stage is the need for a predefined event type [5,6,8,9]. Event types can be constructed manually by experts or with the help of machine learning and then used for subsequent event extraction [5,6,8,9].

Biomedical event extraction research is growing rapidly in English, and one of the methods that can be used is the classification of sequence labeling [10]. Sequence labeling is a task used to label each token in a sentence. It is commonly applied in NER and part-of-speech tagging to classify the name or role of an entity in each token in a sentence [11]. In the research of biomedical event extraction conducted by Ramponi et al. [10], the implementation of sequence labeling focused on NER with the aid of BERT encoders, specifically BioBERT v1.1, which were trained on biomedical texts. However, research on BERT encoders indicates that BioBERT has lower performance than SciBERT and PubMedBERT in the case of NER [12,13]. This is because SciBERT is trained from the ground-up with a biomedical-specific domain vocabulary, whereas BioBERT initially uses vocabulary from the original BERT and then continues training with text sources from PubMed and PMC.

Ramponi et al. [10] developed a model for event extraction using a multi-task learning approach. The first task classified a multi-class event trigger from each token in a sentence, followed by the second task that performed RE and identified proteins or events related to the trigger event through multi-label classification. Multi-label classification is used when an item can be classified into one or more classes, and it involves assigning a value of 0 or 1 to determine the class’s participation [14]. This is a different concept with multi-class classification, where an item can only belong to one class. In the biomedical domain, multi-label classifications are widely used for various purposes, such as categorizing biomedical literature [15].

However, there are several limitations in the model developed by Ramponi et al. [10], as follows:

• In the context of multi-task learning, the applied approach is the joint approach, in which each task is executed independently, resulting in a lack of inter-task information exchange. This phenomenon can lead to overfitting due to recurring word token patterns in different event trigger types. Additionally, the relationship patterns between proteins and other events as entities within an event are also determined by the word token patterns used in the respective event trigger types [16].

• In the task of event trigger classification, there exists a phenomenon where a single-word token may represent more than one type of event trigger. For example, the word of “expression” in sentence “PMA induced the expression of both CD14 and CD23 mRNA and protein.” contains two type of event trigger (Gene Expression and Transcription). If this is approached using multi-class classification, where multi-label is regarded as single-class, it can lead to overfitting in classes that are more dominant in occurrence. Furthermore, it can impact the performance of both tasks.

• BioBERT v1.1 encoder, which is constructed by continuous training on the original BERT model with biomedical text, exhibits sub-optimal performance compared to BERT models trained entirely from scratch using biomedical text. This disparity can significantly impact the overall model performance, as the model heavily relies on the encoder’s representations provided for the event extraction task.

1.3 Research contribution

In this research, modifications were applied to the previous biomedical event extraction model [10] to address the limitations mentioned in Section 1.2. This research article contributes a modification to the model architecture, enabling the model to handle multi-label classification and replacing the encoder with a newer one. The following are the contributions of this research:

• The information between tasks is crucial in determining the output result. Instead of using the joint method, this research adopts the pipeline method to share information between the tasks of NER and RE. Theoretically, this change is expected to improve the performance of extracting biomedical events, as evaluated using the F1-score, precision, and recall metric.

• There is a research that explains that the performance of RE with BERT can be improved by applying masking tokens to sentence to avoid overfitting word patterns [10, 17]. In this study, we used the masking tokens to avoid overfitting. The output of the first task (classifying event triggers) will go through masking process first and then will serve as input for the second task.

• The performance of event trigger identification tasks, which exhibit multi-label characteristics with a dominant single label, is enhanced by changing the output layer from softmax to sigmoid.

• The substitution of the encoder trained on specific domains can improve the semantic representation of each word. Semantic representation plays a critical role in the learning process, leading to an improved performance of the event extraction model.

1.4 Organization of the research

This research article consists of several sections. Section 2 provides a brief report on the related and previous studies relevant to this research’s topic. Section 3 explains the details of tasks involved in biomedical event extraction. Section 4 presents the proposed method, which is the solution to the research problem. Section 5 describes the experimental results, analysis, and discussion. Finally, Section 6 presents the conclusion of this research.

3.1 Event data structure

The provided data consist of biomedical texts, notations for each entity contained in the text, and notations for events and entities involved in the events. The event types used are limited to nine events (Table 1). The “Theme” of the event represents the entity that is the object of the event, while the “Cause” indicates the entity that influences the occurrence of the event. An example of the data format can be seen in Table 2: “T” represents the Trigger, which can be either the notation for a protein entity or the event trigger, and “E” represents the Event, which includes the biomedical event along with its theme, cause, and event type.

Each event type described in Table 1 has different argument characteristics. The first five types of events (Gene expression, Transcription, Protein catabolism, Localization, and Phosphorylation) have only one argument, namely the theme. The event type “Binding” can have more than one theme argument. The last three event types (Regulation, Positive regulation, and Negative regulation) have a theme argument and can also include an optional cause argument. The event-type notations in the data are illustrated in Table 2: (1) “T1” to “T4” represent the protein notations that are entities in this data; (2) “T5” to “T8” represent the event trigger notations; and (3) “E1” to “E4” represent the relationship between event notations and protein notations or other events based on the argument pattern of each event-type. The use of theme and cause arguments in the last three event types allows them to be populated with protein entities or other events (nested events). An example of a nested event can be observed in “E2” and “E3,” where both events have another event (“E1”) as the value of the theme argument. The data usage, as described, involves identifying event types, theme arguments, and cause arguments if there are any values for the cause argument.

3.2 Event structure labeling

The first step in the event representation process is the token labeling process. In this step, three types of labels are used: a label to indicate whether the token is a trigger event (annotated as “dependency” or “d”), another label to indicate the type of relationship between the event triggers (annotated as “relation” or “r”), and a label to indicate the position of the value (protein or another event) based on the argument pattern of the event type (annotated as “head” or “h”). An example of the resulting notation is shown in Figure 1.

Figure 1

Examples of biomedical event annotations on a biomedical sentence [10].

Each token in the sentence shown in Figure 1 has three pre-described labels, and when combined, they create a label in the format ⟨ d , r , h ⟩ . The “ r ” label is assigned a special code to simplify the notation process. For example, in Figure 1, the code “+Reg₊₁” indicates that “This protein entity or event has a relationship with the first +Regulation (Positive regulation) event (denoted by +1) located on the right (denoted by the + sign on the +1 sign).” If the event’s position is the first on the left, the position annotation will be “ − 1 .” Similarly, if the event’s position is the second on the right, the position annotation will be “+2,” and so on. An example of an annotated token is <Expression, Theme, Reg_-1>. In the case of multi-labels, the label annotations of each token are adjusted according to the order. For example, the word “activation” has two “h” labels, resulting in the labeling <+Regulation, [Theme, Cause], [Reg_-1, Reg₊₁]> [10].

4.1 Pipeline method implementation

Applying the joint method to run event extraction provides a model that treats each task individually, without access to information from other tasks. However, effective biomedical event extraction requires using all available information to achieve excellent performance. Therefore, we propose a change in the decoder architecture, transitioning from the initial joint method to the pipeline method. In this approach, module one of the pipeline identifies event triggers, and module two performs relationship extraction. By adopting a sequential training approach and allowing information flow between tasks, the pipeline method ensures that relevant information from one task is used in subsequent tasks. Details of the architectural changes are illustrated in Figure 3.

Figure 3

Change of joint method to pipeline method.

Figure 3 illustrates the transition from the joint method used in the study by Ramponi et al. [10] to the pipeline method proposed in this study. In the joint method, each task is trained and executed independently, lacking access to information from other tasks. In contrast, for successful extraction of biomedical events, comprehensive information is essential. Therefore, we propose modifications to the event extraction model using the pipeline method, where each task is sequentially trained and used. The output of module one, the event trigger identification module, serves as a reference for event masking in module two, the relationship extraction module. Through gradual training and inter-task information exchange, each task can contribute additional information to enhance the overall event extraction performance.

4.2 Task decoder changes identify event triggers and event masking

Handling the event trigger identification task (label “d”) with multi-label characteristics is currently accomplished through multi-class classification using softmax decoders. However, this approach can lead to data imbalance, with single-label data dominating over multi-label data. To address this issue, we modify the decoder for the event trigger identification task by switching from softmax to sigmoid. This change enables the model to effectively handle the multi-label characteristics of event trigger labels. Additionally, we introduce an event-masking mechanism to prevent overfitting of event patterns based on specific words or word sets acting as event triggers. The event-masking system is detailed in Table 3.

Table 3 explains the event-masking process based on the results from the first module of the pipeline, where all event triggers undergo event masking. The system is designed to handle events with similar argument patterns. For instance, events such as Gene expression, Transcription, Protein catabolism, Phosphorylation, and Localization, which share the same argument pattern (having only a theme argument), are event-masked as S V T . The same approach is applied to Binding events and Regulation, Positive regulation, and Negative regulation events. Events with multiple class values receive special masking events based on the following rules: (1) if a token has more than one event label class value, all from the same event group, the masking event corresponds to the associated event group, and (2) if a token has more than one event label class value, all from different event groups, the event masking corresponds to the associated event group join. An example of event masking is provided in Table 4.

4.3 BERT encoder substitution

BioBERT, as an encoder, provides a suboptimal representation of word sequences because it was built from the original BERT and continued its training process with biomedical literature text data. Consequently, the vocabulary used in BioBERT is not specifically tailored to the biomedical domain. To address this limitation, we propose substituting the BERT encoder used in the initial joint approach with BERT models that have been trained with biomedical text and domain-specific vocabulary. Specifically, we explore SciBERT and PubMedBERT as alternatives to BioBERT, as they have demonstrated superior performance and vocabularies better aligned with the biomedical domain. Using these BERT models is expected to yield more effective semantic word representations, thereby enhancing the performance of the event extraction model. We conducted experiments to determine the BERT model that best provides performance values for our proposed method. A comparison of the vocabularies used in BioBERT, SciBERT, and PubMedBERT is shown in Table 5.

5.1 Experiment results on module one of the pipeline

The substitution of the BERT encoder was carried out using the same configuration parameters (Figure 4) as the previous study, with 20 epochs in module one using a sigmoid decoder. The experimental results, compared with the performance obtained using a softmax decoder, are shown in Table 6.

Figure 4

Default parameter in research [10].

Table 6 demonstrates the experimental results, highlighting the decrease in performance when applying multi-label classification using sigmoid decoders compared to multi-class classification using softmax decoders for predicting event triggers (label “d”) on tokens in module one. The average decrease in performance, as measured by the F1-score evaluation metric, was 14.69 for each BERT encoder. The shortcomings of multi-class classification using softmax decoders are evident in the F1-score values for event trigger labels with multiple label values. These results demonstrate poor performance when considered as a case of multi-class classification using softmax decoders. The limited number of tokens with event triggers having multiple label values contributes to this disparity. Thus, if the multi-label value is transformed into the perspective of multi-class classification using a softmax decoder, it can result in an imbalanced class, as each multi-label event trigger is transformed into a new event trigger class. However, the experimental results also reveal that the SciBERT encoder produced the best performance, achieving an F1-score value of 70.28 using a softmax decoder. This superior performance can be attributed to SciBERT being specifically trained in the biomedical domain compared to BioBERT. Additionally, both SciBERT and BioBERT are trained using biomedical literature texts, supplemented by text sources outside the biomedical domain. Consequently, using SciBERT with the default architecture and hyperparameters yields a higher performance value of 11.83 in the F1-score evaluation metric.

5.2 Experimental results on module two of the pipeline

The second experiment focused on module two of the pipeline, which performs RE tasks. Similar to module one, the experimental scenario involved replacing the BERT encoder and using the same configuration parameters (Figure 4) as the previous study, with 20 epochs in module two using a sigmoid decoder. The experimental results were compared with the performance obtained using the joint method described in the study by Ramponi et al. [10].

Table 7 presents the experimental results, revealing a decrease in event extraction performance when applying the pipeline method compared to the joint method. Using the F1-score evaluation metric, the average decrease in performance for each BERT encoder was 15.18. Once again, the SciBERT encoder demonstrated the best performance, achieving an F1-score value of 64.50 using the joint method. This superior performance can be attributed to SciBERT being specifically trained in the biomedical domain compared to BioBERT. Additionally, both SciBERT and BioBERT are trained using biomedical literature texts, supplemented by text sources outside the biomedical domain. Using the default architecture and hyperparameters, SciBERT yielded a higher performance value of 0.81 in the F1-score evaluation metric. Consequently, the biomedical event extraction model using the SciBERT encoder with the joint method was selected for further experiments involving hyperparameter tuning.

5.3 Hyperparameter tuning experiment results

Based on the modifications made in the three previous steps (pipeline method implementation, decoder replacement for event trigger identification, and BERT encoder substitution), it was observed that modifying the BERT encoder led to improvements in model performance. This observation is supported by the experimental results presented Tables 6 and 7. However, changing the decoder to a sigmoid activation did not result in improved performance. On the other hand, using a softmax decoder with the SciBERT encoder led to increased model performance values. Given these findings, the experiment focused on hyperparameter tuning specifically related to BERT.

To assist in tuning the hyperparameters, the model base used in the study by Ramponi et al. [10] was considered. The hyperparameters closely related to BERT are BERT dropout and mask probability [27]. BERT dropout is a dropout layer applied after embedding the BERT layer, while mask probability refers to the random masking mechanism (transforming tokens into “[MASK]”). Both hyperparameters are known to mitigate overfitting in the model during task execution. To determine the optimal hyperparameter values, a grid search strategy was employed using the hyperparameter space described in Table 8.

Six scenarios were tested using the hyperparameters of the best model, i.e., the joint method, softmax decoder for event trigger identification tasks, and SciBERT encoders, as specified in Table 6. The evaluation metric used to assess the experiment’s results was the F1-score. The experimental results are summarized in Table 9. Table 9 displays the results of the grid search

Table 9 shows the results of grid search experiments for hyperparameters of the best models. The experimental results show that the best performance value is generated in the default parameters, namely, BERT dropout with a value of 0.1 and mask probability with a value of 0.1. The F1-score result with the best model was 64.50. Hyperparameter tuning experiments did not provide significant changes. In the case of BERT dropout worth 0.1, the result will be even worse if the mask probability is increased. It may be because the sentence sequence used is too difficult to obtain semantics because the application of mask probability with a higher value and the presence of a protein masking system and event masking makes it difficult for models to learn the semantics of the sentence. This is the same with the scenario where the BERT dropout is worth 0.2. However, on models with a BERT dropout worth 0.2, there is an increase in the mask probability of 0.2. This may be due to the generalization of the BERT encoder input and the BERT encoder output in a balanced way.

5.4 Best result model from across experiments

Based on the results of the experiment scenario that has been successfully run, the biomedical event extraction model was successfully built with a performance value higher than the replication model performance value [10]. The performance value was 64.50 (Table 10) in the F1-score evaluation metric, with an increased value from the previous model of 0.81. Table 11 shows the comparison of the performance results of the best models and replication models [10] in performing biomedical event extraction using the distribution of validation data from GENIA Event Extraction data. Table 12 shows the comparison of the performance results of the best models and previous work with the GENIA 2011 test set. Our model successfully achieved the highest precision result among the previous works, but the recall result dropped, resulting in a lower F1-score than previous works. The details of the test set performance are shown in Table 13.

5.5 Best result model error and analysis

The successfully built biomedical event extraction model exhibits several error patterns in both the first and second modules of the pipeline. Some of these errors are consistent with those observed in the model from the study by Ramponi et al. [10], including: (1) under-prediction, where the event trigger that should have been identified is missed; (2) over-prediction, where a non-existing event trigger is falsely identified; and (3) wrong-prediction, where an incorrect event trigger is identified. However, the proposed method introduces additional errors. Notably, the first module of the pipeline tends to produce event trigger patterns that do not exist. This phenomenon is likely due to the sigmoid decoder’s broader output range, which allows for more variations than softmax. An illustrative example of this error is presented in Table 14.

The deficiencies observed in the second module of the pipeline can be attributed to errors in the first module, as inaccuracies in the initial output can have significant repercussions on the second module’s performance. Errors in the second module vary depending on the event type, as each event type entails distinct argument patterns. Nonetheless, there are four recurring patterns. First, RE may not be appropriately performed when the relevant protein entity does not exist in the same sentence. As the model receives only a single sentence as input from the entire biomedical text, this limitation can impede accurate RE. Second, the model may incorrectly select protein entities or other events in the biomedical sentence as the values of event arguments. This arises from the dynamic nature of argument value locations, necessitating a more profound semantic understanding of the sentence to rectify this issue. Third, the model may fail to predict the number of themes for binding events. This occurs as binding events often entail varying theme arguments, although they typically comprise only one or two themes. A more comprehensive investigation is warranted to enhance the accurate prediction of binding event themes. Finally, the model may overlook the presence of cause arguments in complex events. This happens because cause arguments are optional and can accept other events as values, leading to the extraction failure of many complex events in cases where the ground truth of the event involves a cause with another event value.