Treading carefully: a genre analysis of “accept with revision” peer reviews of linguistic journal submissions using the appraisal system

3.1 Data

Eighteen “accept with revision” peer review texts of published research articles (approximately 1,500–2,500 words each) were collected and analysed. The reviews were obtained from the authors of the articles that the reviews were written about, as the reviewers could not be contacted because they were, of course, anonymous. These articles were published in seven different linguistics journals including: Journal of English for Academic Purposes, Discourse & Society, Australian Journal of Social Issues, Social Semiotics, The Australian Journal of Language and Literacy, Linguistics and Education, and Functional Linguistics between 2014 and 2020. Due to the small sample size, we treat this study as an in-depth pilot study, indicating further analysis of these kinds of texts is needed to make more comprehensive claims about the nature of these texts.

3.2 Theoretical foundations

Given peer reviews are inherently evaluative, we focused primarily on the interpersonal meanings with specific attention to the language of evaluation. To do this, we used the appraisal framework with the aim of ascertaining whether and how appraisal resources function to shape the structure of the reviews in our corpus, though there were a number of other theoretical “tools” that assisted the identification of the generic structure.

The appraisal system comprises three intersecting semantic systems or domains: attitude, engagement, and graduation. The subsystem of attitude covers linguistic construals of feelings and opinions, and itself has three further semantic regions or subsystems: affect, judgement, and appreciation. Affect is concerned with feelings (e.g. I enjoyed reading the article); judgement is concerned with evaluation of people’s behaviour according to social norms (e.g. The authors are always clear about their stance); and appreciation is concerned with the aesthetic evaluation of phenomena (e.g. The analysis is superficial ). Each of these three subsystems have further choices and we elaborate on appreciation here, because this is the most common appraisal resource found in our corpus (see Martin 2017; Martin and White 2005, for details of affect and judgement). Appreciation covers resources for the evaluation of natural and semiotic phenomena and comprises three sub-types: reaction, composition, and valuation. reaction is divided into impact (how something affects someone) and quality (what the aesthetic qualities are of something). Composition is divided into balance (how well things hang together) and complexity (how complex things are). Valuation refers to how worthy something is. Appreciation valuation is particularly sensitive to field (Martin and White 2005), and in the peer review and publishing context, primarily refers to the significance, importance and publishability of the article submissions under review. For each attitude item, we also analysed the target (the person/thing that is being evaluated) and the emoter or appraiser (the person/thing doing the evaluating), as these can contribute to the delineation of stages and phases.

The subsystem of engagement, which deals with the interplay of voices in a text (Martin and White 2005), is also important for this study as certain engagement resources invoke evaluation. For example, the resources of deny (The arguments not clear ), counter (e.g. comparing two things using but), or the use of modality (This would enhance your discussion), all serve to position the paper under examination as in need of further work.

The third domain of appraisal, graduation, increases or decreases the intensity (“force”) or the prototypicality (“focus”) of meanings made within the attitude and engagement systems. graduation enabled us to identify invoked evaluation in peer reviews. According to Hood (2010) resources that intensify processes (write carefully ) or resources that represent degrees of fulfilment of a process ( tried to argue) and actualisation ( seems to suggest) can invoke attitudinal meanings.

Finally, evaluations can be explicit (inscribed) or implicit (invoked). Inscribed attitude is recognised using expressions that carry intrinsic positive or negative evaluation. These are typically realised as Qualities (e.g. interesting research), a small set of preferred infused lexis (e.g. strengths of the paper), and a small group of processes that are infused with positive/negative attitudinal meanings (e.g. I question that assumption).

We also considered Don’s (2016) invocation spectrum because it enabled us to make visible some invoked strategies such as intra-textual linkage, i.e. creating a link between two parts of the text, and in-group allusions, i.e. the use of references that are interpreted attitudinally only within a particular community. However, contrary to the use of a system network by Martin and White (2005) and a spectrum by Don (2016), in studying peer reviews we found it useful to consider the strategies that invoked attitude as choices along a “cline of invocation”. This is because rather than being separate categories, it is more useful to think about these invocation strategies as different degrees of implicitness.

We found that in analysing the language of evaluation in peer reviews using appraisal, there is a group of language resources that invoke attitude are strategies that need a further description for this genre as they occur frequently. For instance, when a comment says: The authors should include implications of the findings, modality is employed to construct a preferred future paper, which we name “irrealis comments”, i.e. the comments that point forward to the ideal paper to be shaped in the future.

Examining the targets of evaluation revealed what is being evaluated and afforded a shunting between the interpersonal and ideational meanings, helping to make visible the kinds of entities and processes that are being evaluated (Hood 2010). This is important in the publishing context, given the importance of evaluating the research rather than the authors of that research.

Hood’s (2008) study on the way meaning changes between a text and a summarised version of that text found that ideational meanings can be instantiated in more general or more specific ways. She argues that this range of instantiation forms a cline of commitment from the most general to the most specific. The degree of commitment of ideational targets helped us identify the structure of peer review by creating the distinction between the generic stages.

3.3 Genre analysis using appraisal

Within SFL, genres are described as staged, goal-oriented social processes (Martin 1994). They are also described by Hyland (2007: 149) as “abstract, socially recognised ways of using language”, emphasising the fact that people in any given culture recognise and share common ways of using language. Describing genres, their structure and language features has been a key pursuit within many schools of linguistics, such as SFL and English for academic purposes (e.g. Swales 1990).

There are a small number of studies that have found the appraisal system useful for determining the structure – both stages and phases – of written texts. Rose (2019), for example, used appraisal (among other systems) to help determine phases within stories from different cultures, such as the “problem”, “comment”, and “refection” phases, to name a few. However, it is academic texts that are of more relevance here. Within this context, Hood (2010) used appraisal to identify the different phases in introductions to research articles, whilst Humphrey and Dreyfus (2012) deployed appraisal framework to help identify aspects of point phases in tertiary students’ interpretive genres in the discipline of applied linguistics. Additionally, Szenes (2017) used appraisal framework to assist in the delineation of phases, such as “reservation”, within the stages of tertiary student economics assignments called “country reports”. These studies demonstrate that the appraisal framework is useful for the identification of stages and phases, particularly in academic genres where evaluation is critical to the meaning making.

4.1 The generic structure of “accept with revision” peer reviews

Our linguistic analysis showed that the generic structure of the 18 “accept with revision” peer reviews has two stages, which we have named Overview and Recommendation for revisions (hereafter Recommendation). The Overview stage, which always precedes the Recommendation, has four phases while the Recommendation has two, as per Figure 1.

Figure 1:

The generic structure of “accept with revision” peer reviews in our data.

The Overview stage functions to provide general insights about the article under review. It unfolds through four distinct phases of praise, summary, publishability, and critique, of which only praise occurred in all the reviews. The Recommendation stage consists of just two phases: issue, where the reviewer points out a particular problem or issue within the paper; suggestion, where the reviewer provides guidance on how the issue can be remedied. The issue is the only phase found in every Recommendation stage as not all reviewers make suggestions for remediation. Each comment in a review comprises either the issue and suggestion as a single pair or the issue phase on its own. The Recommendation stage can include one or more comments about the paper (as indicated by “1–n” in Figure 1). An example of each of these phases from the same review can be seen in Table 2:

In the following sections (Sections 4.2–4.7) we discuss each phase separately to make visible the evaluative language features; however, this should not imply that the structure of peer review was determined without looking at the whole texts. What unites the stages in the whole text can be seen through an examination of the targets of evaluation. Further, the Overview and Recommendation stages are distinguished based on two principles: (1) the target of evaluation, and (2) the type of invoked attitude.

The appraisal analysis showed that there are generally four types of target of evaluation in peer reviews: the “authors”, the whole of the “paper”, “general parts of the paper” (e.g. methodology), and “specific parts of the paper” (e.g. example on page 15). The Overview stage is distinguished in that it contains the more “general” targets of evaluation, whereas the Recommendation stage has more “specific” targets. That is to say, from the Overview stage towards the Recommendation stage, the target of evaluation (excluding authors) changes from the superordinate “paper” and its “general meronyms” (e.g. methodology) to its “specific meronyms” (e.g. the example “She […] me to take […]”). This reflects the way reviews gradually narrow their focus to more specific targets that are about the object of the study in the paper. Thus, as the review unfolds, there is an increased commitment of ideational meaning via the targets, which makes the revisions more tangible.

Regarding the second distinguishing principle, the type of invoked attitude, there is a difference in the resources used to invoke negative attitude between the two stages. The Overview stage displays a predominance for graduation resources whereas the Recommendation stage displays a wider variety of other resources including from the sub-system of engagement as well as invoked negative attitude realised by different speech functions, such as questions and imperatives (see Sections 4.6 and 4.7).

4.2 Overview stage, phase 1: praise

As all 18 reviews in this data set started with the praise phase, we argue this is obligatory in this data set. In this phase reviewers provide general positive evaluation about the authors, the paper as a whole, some “general” sections of the paper (e.g. methodology), or an activity related to the paper, such as reading it. The evaluation is predominantly inscribed and resources from all the three appraisal systems of affect, appreciation, or judgement were found, though appreciation: valuation and reaction are the most frequent. The following praise examples (Examples 1–3) show these appreciation resources with different targets: the paper, general parts of it or the activity of reading it. The attitudinal resources are bolded, and the targets are underlined:

Evaluation of the authors themselves is rarely found, though when it is present, it is only of the sub-type of positive capacity (see Example 4):

All evaluation in the praise phase is both positive and inscribed, and realised as Qualities (rigorous, interesting and clear), lexis infused with positive attitudinal meaning (e.g. strengths) or a process infused with positive attitudinal meaning (e.g. enjoyed).

4.3 Overview stage, phase 2: summary

The second phase found in the Overview stage is summary. This is an optional phase where reviewers provide a general short synopsis of the main points and aims of the paper under review. While we relied on the appraisal choices and the polarity of evaluation to distinguish phases, the summary phase is determined by the fact that it has no evaluation, as shown in Examples (5) and (6); that is to say, it simply summarises:

4.4 Overview stage, phase 3: publishability

The third phase in the Overview stage is publishability, which is another optional phase where reviewers provide their opinion about the suitability of the paper for publication. This phase comprises two sub-phases: (1) Publishability status and (2) Condition.^[1] Publishability status refers to whether the paper is publishable, and Condition names the conditions under which the paper is publishable, i.e. after revisions are made, as can be seen in Examples (7) and (8):

The linguistic criteria that distinguish the two sub-phases are the polarity and the type of evaluation. In the Publishability status sub-phase, the main resource is invoked positive [appreciation: valuation] whereas in the Condition sub-phase, the main resource is invoked negative [appreciation: composition], as the reviewer highlights the problems that need revising. Invoked evaluation is shown by a “t” inside the square brackets. We also found that in the publishability phase, the target of evaluation is the whole paper or some general sections of the paper:

Publishability status: I would recommend accepting [t, +appreciation: valuation] the paper [target: whole paper]

Condition: following certain revisions [t, -appreciation: composition] in the statistics and organization of the paper [target: general sections of the paper].

As shown above, the Publishability status and Condition sub-phases can either appear as separate sentences (see Example 7) or woven together as a single sentence (see Example 8) (see Humphrey and Dreyfus 2012).

4.5 Overview stage, phase 4: critique

The fourth phase of the Overview stage is critique. This is also an optional phase, where reviewers provide general criticism about either the authors, the paper and/or its general parts, foreshadowing the more specific comments to come in the Recommendation stage. In contrast to the praise phase which has inscribed positive evaluation, the negative evaluation in the critique phase is predominantly invoked, because, as discussed above, it is considered face-threatening to criticise one’s peers (Paltridge 2015). This invoked evaluation occurs as either appreciation or judgement, with appreciation being the most frequent. As shown in Examples (10) and (11), both negative appreciation valuation and composition are used:

One of the distinct ways the negative appreciation is invoked in this phase is not by saying what the paper under review does not do, but rather what it should or could do, as per Example (10). This invocation strategy enables the reviewer to gently criticise what is there, without explicitly saying so. As discussed in the method section, We use the term “irrealis” after Halliday and Matthiessen (2004) to describe this invocation strategy. Irrealis is associated with an as-yet-unrealised state of affairs and, through the use of modals, reviewers create an unreal situation which invokes negative attitude about the current paper under review by describing what the as-yet-unreal paper, which will come into being after revision, should look like.

Like the praise phase, in the critique phase, authors are rarely the target of negative judgement as it is the paper that is the focus, not the researchers/authors per se. However, there is the occasional example of this, as shown in Example (11):

Other strategies that invoke negative judgement in the critique phase use passive voice and personification of some parts of the paper. These resources function to remove the negative evaluation of authors, which according to Hood (2010) helps de-personalise the criticism. For instance, in the clause Stats are described in ways that are hard to understand, the use of passive voice, are described, backgrounds the author as the doer of the action and thus the target of evaluation is the stats rather than the author, which preserves the author’s face. This means the appraisal resource is instantiated as negative appreciation rather than negative judgement.

4.6 The recommendation stage, phase 1: issue

The second stage in the generic structure of “accept with revision” peer reviews, Recommendation, has two phases: issue and suggestion. As stated earlier, the issue phase is obligatory and outlines the problem that needs to be addressed. This is a complex phase with three sub-phases: Location, Criticism, and Justification as shown in Example (12):

The Location sub-phase, as the name suggests, functions to show exactly where in the paper the problem exists. Though frequently found, this is an optional element which means that not all comments in a Recommendation stage specify the location of the problem. This sub-phase contains no appraisal, just circumstantial (location: place) meanings. The Criticism sub-phase, which is obligatory, represents the problem that needs to be fixed. The Justification sub-phase provides reasons for the criticism, and it is an optional element as not all comments in our data include this sub-phase, as shown in Example (13). The phase name Justification comes from Samraj’s (2016) study of peer review as our findings regarding this sub-phase are consistent with her findings and we agree that this term usefully represents the function of this sub-phase.

Regarding the appraisal resources of these sub-phases, the Criticism sub-phase is distinguished by negative evaluation of ‘specific’ parts of the paper (e.g. tables 6 to 9). Like the critique phase, in the Criticism sub-phase, invoked evaluation is more typical and is instantiated as either appreciation valuation and composition or judgement (mostly tenacity) as shown in three examples below:

The most frequent strategy that invokes attitude in the issue phase as a whole is irrealis comments. These are realised by modals (e.g. should in Example 15 and need to in Example 16), which function to invoke criticism of the paper or the authors by describing how the final paper for publication should look and what authors are required to do if they want their paper to be published.

While we did not find one particular appraisal feature that identified the Justification sub-phase, it was distinguished primarily by causal resources, which function to provide reasons for the problem highlighted by the reviewer in the Criticism sub-phase. This causal relation occurs either explicitly as bolded in Example (17), or implicitly as highlighted in red in Example (18):

In the Justification sub-phase, the negative evaluation itself can sometimes serve as Justification for the problem, as per Example (14), or Justification can be a separate clause, as per Example (12). In this sub-phase, to show the rationale behind their criticism, reviewers sometimes use the content of the paper to support their criticism, as per Example (17), or they rely on literature and other sources to support their negative evaluation, as per Example (19). In this way, criticisms are presented as referenced and supported rather than as personal opinions.

4.7 Recommendation stage, phase 2: suggestion

The second phase in the Recommendation, suggestion, is optional as reviewers do not always provide solutions or guidance for authors on how to revise their paper. Nonetheless, this phase comprises two sub-phases: Advice and Benefit, as per Example (20):

The Advice sub-phase provides guidance on how the issue can be solved whereas the Benefit sub-phase, following Samraj (2016), shows the benefits of acting on the advice. If a suggestion phase is present, Advice is obligatory, whereas Benefit is optional, as can be seen from Example (21).

The Advice sub-phase is instantiated by a range of negative evaluations which are all invoked. These can be either judgement or appreciation and they target the authors or specific parts of the paper:

Advice sub-phase from Example (20): The authors need to include [t, -judgement: veracity, target: the authors] work from […] edited collection about […] and […].

Advice sub-phase from Example (21): Delete ‘explicitly’ at the end of paragraph 4 [t, -appreciation: valuation, target: specific part of the paper].

The Advice sub-phase is sometimes instantiated as an imperative which functions to direct the authors to make changes. We found that in the discourse of peer review, imperatives also function to negatively evaluate some parts of the paper. The use of imperatives here invokes a kind of attitudinal meaning which can be picked up by those inside the community of practice of peer review (Don 2016).

While the target of evaluation in Example (20) is the authors, at another layer of meaning, the paper in general can be taken as the target of negative appraisal. Every comment about a specific or general part of the paper or the author’s conduct can be seen to represent negative appreciation of the whole paper, which is in-line with the reviewer’s recommendation as ‘accept with revision’. The collection of all comments and patterns of evaluation in this genre show that the paper is not publishable as it is, fulfilling the function of this type of peer review.

In contrast to the Advice sub-phase, which involves negative evaluation, the Benefit sub-phase is instantiated by positive appreciation of either the paper and its parts or the author’s activity for improving the paper. This positive appreciation is mainly invoked as it describes the merits of the irrealis future paper that will come into existence after the revisions are made i.e. the paper as satisfactory for publication, and can be seen from the modal would in the following example:

Benefit sub-phase from Example (20): as this would enhance your discussion of […] [t, +appreciation: valuation, target: activity to improve the paper].