Automatic labelling of transitivity functional roles

1. Introduction

Recent years have seen data-driven approaches to natural language processing successfully applied to a wide range of problems including syntactic (Collins 2003; Klein and Manning 2003), semantic (Gildea and Jurafsky 2002; Pradhan et al. 2004) and discourse (Soricut and Marcu 2003; Hernault, Prendinger, and Ishizuka 2010) analysis. Computational processing of linguistic data for functional analysis using functionally oriented frameworks such as systemic functional linguistics (SFL) (Halliday and Matthiessen 2004), on the other hand, remains a relatively under-explored research area. The functional approach to the description and analysis of the various aspects of language (including structure, meaning and use) has gained increasing influence and adoption as an alternative to formalist theories (Huang 2002). Linguistic analysis using functional theories is still largely manual, and the lack of relevant resources has limited progress in automating the often time- and effort-consuming process.

In this article, I present work being carried out to automate parts of the functional structure using a recently constructed corpus annotated within the framework of SFL (Yan and Webster 2013). Specifically, I focus on the assignment of functional labels in the transitivity system, a core system within SFL which is known to be particularly difficult to process using purely rule-based approaches (Honnibal and Curran 2007). Trained on lexical, syntactic and contextual features obtained from the corpus, automatic procedures are employed to automatically identify and classify the process types as well as accompanying participant roles in unrestricted texts. The proposed system will significantly enhance the potential for applying the SFL framework to the task of automating large-scale text analysis. Finally, I discuss some of the difficulties and future directions, proposing methods such as active learning and annotation projection and extending the current system to handle other languages such as Chinese.

2. Related works

Recent work (Costetchi 2013a, 2013b) has been carried out to automatically generate simplified SFL transitivity parses for unrestricted sentences. Semantic role labelling is performed using a dependency graph-based method and a pattern matching method, resulting in reported accuracy of 72.65% on simple sentences. Although this approach shows respectable results, the current implementation is still largely dependent on handcrafted patterns that limit its application to more complex, unrestricted texts.

To address the bottlenecks of current state-of-the-art functionally based parsers, a small-scale corpus (Yan and Webster 2013) has recently been annotated with SFL functional roles. The annotation of the corpus was done in four successive layers:

• Clausal: clausal boundaries, including boundaries of embedded clauses. The clause boundaries are aligned with the RST Treebank where clausal boundaries are also annotated.

• Process: processes are the core of a clause, typically realized by a verbal group headed by the root verb of the clause. As described in Halliday (1994), there are 6 common types of processes (material, behavioural, mental, verbal, relational and existential), subdivided into 10 more refined types (with material subdivided into doing, happening, mental into perception, cognition, affection and relational into attributive, identifying). Each of the process types is associated with a set of nuclear and non-nuclear participants.

• Participant: participants are the central nominal groups of the clause typically realized by the grammatical Subject or Object of the clause. A summary of the processes with its related participants is shown in Table 1.

• Circumstance: peripheral units related to time, place, manner, etc. typically realized by adverbial groups. There are in total nine broad types of circumstances: Extent, Location, Manner, Cause, Contingency, Accompaniment, Role, Matter and Angle, each with its own subtypes. The Extent circumstance, for example, is subdivided into three subtypes: duration, frequency and distance.

In total, 81 documents from the Wall Street Journal section of the Penn Treebank have been annotated, with a total of 43,351 words, divided into 1621 sentences and 4620 clauses. The corpus, though still relatively small-scale, serves as the basis for further statistical modelling and supervised training in our proposed system. In the following sections, I describe the procedure for constructing such a data-driven system.

3. Classification

The focus of this article is on the system constructed for labelling functional roles in the transitivity system (Table 1). In an SFL-based analysis, three strands of meaning (called metafunctions) operate concurrently: the ideational (experiential and logical), interpersonal (social interaction) and textual (communicative organization) metafunctions. The transitivity system is a major system in the ideational system. Specifically, it involves a configuration of processes and participants involved (such as Actor, Goal) and the accompanying circumstances (such as time, place, manner). The task of labelling transitivity roles is divided into two steps: (1) clause boundary identification and (2) transitivity role classification.

Clause boundary identification includes the task of dividing texts and sentences into clauses, or Elementary Discourse Units (EDU). Identification of clause boundaries is usually the first step in a functional analysis. Although considered a kind of lower-level segmentation, clause boundary identification is crucial to the quality of further discourse parsing. Recent advances in discourse parsing (HILDA) have yielded good results in clause segmentation, achieving an F-score of 93.8%, or 96% of the human performance level (Hernault, Prendinger, and Ishizuka 2010). Our role labelling system uses the state-of-the-art discourse segmenter provided by the HILDA parser for clause boundary identification.

On the other hand, the second step of classifying transitivity roles is less well developed. We may approach the problem first as one of disambiguating the different senses of the process as realized by a verb, and second as a Semantic Role Labelling problem whereby the participant roles associated with the process are identified. Transitivity analysis begins with identifying the process and process type in the clause. For example, in the clause John hit the ball, when the type of the process hit is identified as material, the related participant roles of John and the ball can be inferred from their grammatical functions (John being the grammatical Subject and the ball being the grammatical Object) in the clause. The step may be divided into two subtasks: (1) classification of the process type and (2) identification of the participant roles based on the identified process type.

det(Court-2, The-1)

poss(decision-4, Court-2)

nsubj(have-6, decision-4)

aux(have-6, will-5)

dobj(have-6, consequences-8)

amod(consequences-8, billion-dollar-7)

prep_for(consequences-8, manufacturers-10)

mark(continues-4, If-1)

det(controversy-3, that-2)

nsubj(continues-4, controversy-3)

advcl(are-9, continues-4)

amod(producers-8, other-6)

amod(producers-8, foreign-7)

nsubj(are-9, producers-8)

root(ROOT-0, are-9)

acomp(are-9, likely-10)

aux(grab-12, to-11)

xcomp(likely-10, grab-12)

dobj(grab-12, most-13)

det(sales-16, the-15)

prep_of(most-13, sales-16)

nn(Europe-19, Eastern-18)

prep_in(sales-16, Europe-19)

4. Evaluation and performance

The data from our corpus are divided into a tuning set and an evaluation set. We use 20 of the 81 documents for feature engineering and model development, and the remaining documents for evaluation, performing a 10-fold cross-validation on a random forests classifier (Breiman 2001). Random forests are an ensemble classifier based on decision trees. The idea is that growing an ensemble of decision trees and letting them vote in a classification task can lead to significant improvement in accuracy. The theoretical underpinnings for the learning model are detailed in Breiman (2001) and Liaw and Wiener (2002). The advantages of random forests over other learning algorithms are that (1) it is an effective tool in prediction that yields state-of-the-art performance, (2) it does not overfit, (3) it is fast, (4) it gives estimates of the relative importance of variables (useful in feature selection) and (5) it is effective in estimating missing data (such as the participant roles which are often missing in our feature sets). We also compare the performance of the random forests classifier with another state-of-the-art classifier based on Support Vector Machine (SVM) (Chang and Lin 2011). We use a simple baseline to compare with the performance of the classifier. The baseline performance is obtained by always choosing the most frequent class in the test data. In our corpus, the most frequent process type is doing, and thus, the baseline classifier would just pick this as the predicted class without considering any other features (Table 11).

The classification results seem to suggest that the classification of process types is still heavily lexically grounded. Most of information for the classification can be deduced from looking at the word itself and guessing its possible function without taking other contextual features such as syntactic dependencies and surrounding content words into account. On the other hand, the information as encoded in the syntactic and contextual features conflates with the lexical information to a large degree. This is reflected from our observation that even without using the words themselves as features we still get more or less the same level of performance. As a result of the conflation, although the syntactic and WSD features have proven useful in disambiguating the process types, the performance increase is not as great as expected. To some degree, this is not surprising since the disambiguation of functions, with its similarity to WSD, is an AI hard problem, and only a few WSD systems have managed to achieve marginal gains over the most frequent sense on “all-words exercises” (as opposed to limiting the number of words to only a handful) despite years of research into the problem.

Looking at class-specific performance (Table 12), we see that the behavioural process has the lowest accuracy. This conforms with feedback from corpus annotators during the annotation phase when they complained about the difficulties in differentiating the behavioural from other types of process such as material:doing and mental. The second worst performing process type is affection. This might be explained by its low number of instances in the testing corpus, since the performance of a class can be affected adversely by insufficient training samples. The affection process type is followed by identifying and happening, both of which are subcategories of a major process type (relational and doing) and may be more easily confused with their close “relatives” (attributive and doing). The class with the highest accuracy is verbal, presumably due to there being only one class in the major process type and there are relatively few ambiguous word types that realize verbal functions (e.g. most commonly say, state, announce). Another evaluation is performed on the classification of participant roles. Similar to process type evaluation, we also set up a naïve baseline that always picks the most frequent participant role in the corpus. We also compare the performance of the classifier using a few ablations of the features (the numbers refer to the feature ID in Table 9).

As seen from Table 13, the random forest classifier has yielded relatively high performance in classifying participant roles. Starting with only the process type as features, the performance is poor. However, with the addition of grammatical role information – such as whether the participant is in the Subject position – the most substantial gain in performance is achieved. This confirms our intuition that knowing the process type of a clause may put the participants in the right functional configuration, and information on grammatical relations within the configurations can be crucial in disambiguating the participant roles (Table 14).

The detailed performance across the classes seems to be quite divided, ranging from perfect prediction to zero accuracy. The class that achieved 100% accuracy is the relatively simple case of Existent, which is the only possible participant given the existential process. The overall performance for the nuclear participants is high, while the worst performing classes are all non-nuclear participants (Receiver, Client, Beneficiary, Behaviour, Recipient). This may have been due to the low percentage of the non-nuclear participants as well as the inability of the feature for the relative position of the grammatical roles to distinguish between nuclear and non-nuclear participants. Overall, while there is still room for further improving the accuracy of the transitivity classifier, initial experiments have shown promising results and support the notion that automating transitivity analysis is both practical and feasible using data-driven machine-learning techniques.

5. Extending to resource-poor languages

So far our discussion has been limited to the labelling of transitivity structure in the English language. Compared with English with its lexical resources (e.g. WordNet), annotated corpora (e.g. Penn Treebank), syntactic parsers (e.g. the Stanford Parser) and semantic analysis tools (e.g. UKB), available linguistic resources in most other languages are scarce in comparison. For example, Chinese, while being one of the most widely used languages, has a relative paucity of lexically and semantically annotated resources, and it is only in recent years that efforts have been made to build Chinese counterparts of existing English corpora and tools (Li et al. 2003; Xue et al. 2005). Given this resource scarcity, extending our system to deal with languages other than English is a challenging task. Despite this, we propose to overcome the problems by adopting two general approaches.

The first is active learning (Zhu, Lafferty, and Ghahramani 2003) in the annotation process. Our current corpus has been annotated mostly manually, with the help of a web-based collaborative platform. The uneven distribution of words and their senses often means that most of the efforts go into annotation of frequently appearing words, which is often unnecessarily repetitive and adds little value to subsequent disambiguation of semantic senses. Active learning is where a learning algorithm tells us which set of unlabelled data to label. This is often more desirable than selecting our own set of data to label randomly. In addition to expanding the current functional corpus, active learning is also particularly useful when creating new functional resources in a new language such as Chinese, substantially reducing the time and cost involved in the annotation.

The second is the use of annotation projection (Pad and Lapata 2009). The key idea behind annotation projection is that, given a pair of parallel corpora that are translations of each other, one in English (E) and one in a less resource-rich language such as Chinese (C), we can first annotate E for its functional structure and then project this functional structure onto C by relying on word alignment information in the translation pairs. After a sizable set of the sentences in C has been annotated with functional structure through annotation projection, a classifier can then be trained on the set independently of the parallel corpora.

With the use of semisupervised learning and annotation projection, we envision that the extension of our current system to cover resource-poor languages will be made significantly more efficient.

6. Conclusion

In this article, I have discussed the problem of automating the analysis of SFL transitivity function structures. A small-scale corpus was annotated for its transitivity structure to which has been applied state-of-the-art machine-learning algorithms for the automatic classification of the process types and participant roles of clauses. I modelled the tasks as a role labelling task and a sense disambiguation task, detailing the process in which we engineered key features for the classification. The labelling system was able to achieve performance significantly better than our baselines.

Due to difficulties in automatic text analysis using SFL, current state-of-the-art work is still limited both in scope and in functionality. As with other semantically oriented tasks in NLP such as semantic role labelling, determining the multifaceted functional roles of a functional component is an AI hard problem, more complicated than their syntactic counterparts as this depends more on the configuration of a wide range of contextual factors such as the functional semantic configuration in different usage patterns. Due to such difficulties, a parser for producing the complete SFL structures of free-form texts has yet to be developed. What is proposed here is a new perspective inspired by advances in other fields of NLP which have achieved success. Work is ongoing in The Halliday Centre at City University of Hong Kong to build on this system to create a more comprehensive and robust parser for SFL analysis.