Internal and external appraisals of analytical writing. A proposal for assessing development and potential improvement

1.1 Analytical writing

The overall purpose of a discourse, its “notional structures”, may encode in the form of differing surface structures or genres (Longacre 1996). Analytic notional structures are realized in diverse genres such as essays, descriptions, or reports in which communicative functions – explaining, describing, arguing – are differently weighted (Schleppegrell 2004). While some analytical realizations are mainly expository and focus on analyzing the topic, others are mainly argumentative presenting two- or more-sided perspectives on the topic. The diversity of analytical genres allows students to build and express command on a certain topic, and to shape a (well) supported standpoint when dealing with controversial topics.

Skills involved in critical thinking play a crucial role (Driver et al. 2000). Writers must be willing to be well-informed about the topic addressed to accumulate evidence for grounding a solid standpoint while putting personal biases aside (Willingham 2007). Analytical skills facilitate the writer to work out what is relevant to the problem in hand. Those are useful skills especially in educational contexts, where argumentation is formally introduced, practiced, and assessed. Attaining proficiency in analytical essays contributes to academic success. We assume that the very process of analytical text production has an epistemic effect: it activates writers’ topic knowledge and affects their perception of how much they know about it.

1.2 Theoretical framework of DAW

Pivotal to our approach to text analysis is Longacre’s (1996) crucial remark on the futility “[…] to look in a discourse for a feature which is not characteristic of the type to which that discourse belongs” (p. 7). In line with Berman and associates (Berman and Verhoeven 2002) we applied functionally motivated quantifiable criteria for characterizing different facets and quality of text construction in the light of specific type of discourse.

Genre distinction proved to play a key role in vocabulary, grammar, discourse structure and content of expository and narrative texts (Berman and Nir-Sagiv 2007).

Our approach is also nurtured by Mann and Thompson’s (1988) Rhetorical Structure Theory which conceptualizes text structure as a hierarchical organization with compulsory foregrounded (nuclei) and other backgrounded (satellites) elements. For example, the sequence of events is nuclear in narrative texts – there is no narrative without events – while evaluative components, although adding to text richness, are peripheral. In our analysis, claims and grounds are nuclear components whereas counterclaims following a claim, although useful for the communicative purpose of argumentation, are dispensable. This distinction is useful for examining texts produced by novice writers and allows us to appreciate writers’ awareness of genre constraints.

We also consider several aspects from Toulmin’s pioneer model of argumentation. Toulmin (1958) points at the role played by different elements of an argument. Claims, the writer’s assertions on the topic, and grounds, appealed to as support of the claims, are the main elements of an argument. Other no obligatory elements such as qualifiers and scopes indicate the strength of the claim and limit its span.

Finally, for text parsing, we resort to a proposal of rhetorical moves (RMs) as strands of discourse that fulfill a concrete communicative goal which are identified applying pragmatic, semantic and syntactic criteria (Vilar Weber and Tolchinsky 2021). This approach is particularly useful for analyzing texts in terms of the functional/communicative structures that typically make up particular genres.

1.3 Main features of DAW

The model provides a methodological framework to account for developmental changes – resulting from ordinary developmental course – and micro-developmental changes – occurring in the short run in the context of circumscribed pedagogical scaffolding – in text construction. It was designed to assess analytical text writing but is applicable to any type of discourse. The main features of the model are the following: 1) repeated text writing controlling for topic content and pedagogical input; 2) focus on linguistic expression of written products; 3) multilayered text-analysis; and 4) researcher and reader-based appraisals of the texts. Applications of the model open crucial questions (and provide some answers) on learning and teaching of writing.

1.4 Researcher and reader-based text appraisals

Research has been devoted to identifying the text features that support subjective impressions of text quality (Aparici et al. 2021). The assessment of these features by researchers that have designed and participated in the research process provide an internal appraisal of quality. In DAW the internal appraisal is performed by means of qualitative and quantitative assessments of text-embedded features (e.g., text structure) that, according to previous studies, indicate text quality (McNamara et al. 2010). This researcher-based assessment has been complemented with an external appraisal performed by judges who neither design nor participate in the research process (McMaster and Espin 2007). In DAW, teachers and graduates in language-related studies provide a reader-based appraisal by means of analytical rubrics embracing different aspects of text (e.g., lexical richness, text organization) followed by global scoring of text quality. This two-sided approach is an additional strength of our model.

1.5 Applications of DAW

We followed the above-described methodological framework in previous studies aimed at tracing how students appropriate the expected features of analytical writing from elementary to higher education, and which (if any) of these features are sensitive to pedagogical scaffolding. Across studies, we assessed different sets of text-embedded features that were both assumed to fulfill rhetorical aims (Langacker 1999), and to serve as indicators of developing writing proficiency. The researcher-based assessment was complemented with a reader-based one to test which features best explained it.

In Aparici et al. (2021) we focused on lexical, syntactic, and discursive indicators. We also tracked students’ productivity as a predictor of writing quality and a proxy of number of ideas (Weston et al. 2011), and found that up to university they use more words and clauses for responding to the prompt. The implemented classroom activities also affected productivity, but not at university level.

At a lexical level, we traced word length, lexical diversity, lexical density, and use of adjectives and nominalizations. Across schooling, students used more sophisticated and diverse words, as well as more adjectives. In contrast, nominalizations decreased with school level, while lexical density did not show any development. Classroom activities affected positively lexical diversity and word length, but only among the youngest groups.

We found similar divergences in the syntax and discourse domains: the use of different syntactic linkages (e.g., coordination, subordination) increased across schooling, but syntactic density (e.g., number of words per discourse-level units) remained stable after elementary school (Crossley and McNamara 2010). Pedagogical scaffolding affected only the use of relative clauses and syntactic density, and only in elementary school.

At the discourse level we assessed the use of Discourse Markers creating texts’ texture (Choi 2007) to find that experienced writers produced better-connected discourse units, although differences were found only between elementary and higher school, and were not affected by the implemented activities.

In Rosado et al. (2021) we examined high school and university students use of intra- and interclausal connectivity devices, and productivity as indicators of text quality. Students contained significantly longer texts with age. However, younger students’ texts produced a higher proportion of intraclausal connectors than older ones, suggesting that text cohesion does not necessarily reside in a profusion of connectors (Crossley et al. 2011). The analysis revealed a “trade-off effect” whereby conjunctions were most frequent in high school, while optional discourse markers were more frequent in university.

Parallel studies inquire into the structural features of analytical essays of Spanish monolinguals (Vilar Weber and Tolchinsky 2021) and Catalan/Spanish bilinguals (Tolchinsky et al. 2021; Tolchinsky et al. submitted). We assessed (1) completeness of text structure (i.e., the extent to which the argumentative component – assertive and argumentative moves – and the non-argumentative component – expository moves – are realized), and (2) the relative weight of the non-argumentative component. We found an unequal pattern of development in bilinguals and monolinguals. While the texts of bilingual elementary school students included assertive, argumentative, and expository moves, monolinguals completeness of TS was not attained until high school. Moreover, no further development in completeness of structure was observed in the two samples. Only the expository component, containing evidence and justifying reasons, showed a steady increase, especially among bilinguals. We attributed this difference to bilinguals’ enhanced metalinguistic awareness (Hsin and Snow 2017) which may raise their concern for potential addressees to justify their standpoint.

To sum, the reported applications of DAW enabled us to distinguish linguistic features that improved in the course of ordinary development (e.g., text and word length, syntactic/discursive connectivity), from fewer features that were also sensitive to pedagogical scaffolding (e.g., lexical diversity; use of relative clauses), and showed a higher potential development. Changes in writing quality were facilitated by a developmental basis rather than induced only by pedagogy.

An exception to this rather clear developmental pattern of indicators of writing proficiency was text structure that showed mixed findings. The presence of the expository component in analytical essays continued to grow up to university and was sensitive to pedagogy. In contrast, completeness of structure grew with schooling among monolingual students but was already attained among elementary school bilinguals, while appearing resistant to pedagogical work across the board.

Moreover, by applying a distributional approach to text structure we failed to capture the features of overall text organization that make a difference in text quality. To clarify, once the main communicative function of rhetorical moves (e.g., to assert an opinion, to provide evidence) was identified, we assessed the distribution of moves in the texts to chart the text structure. We disregarded the differential qualities of moves performing a similar function, and how they relate to each other. For instance, whether the asserted opinion favors or opposes the writer’s standpoint, or the provided data support the writers’ opinion. We speculated that dwelling on the differential qualities of moves would enable us to capture developmental differences in text well-formation that were not detected in previous analyses. To explore this hunch, we revisited our corpus of essays and applied a qualitative, content-related analysis of text structure in light of relevant features of analytical writing.

2.1 Goals and expectations

In this study, we aimed at depicting development and potentiality of improvement of the explicitness of writers’ standpoint, the quality of writers’ positioning, and the expression of reflective thinking by one between subject factor – school level (Elementary, High school, University) – and one within subject factor – Text (T1, T2, T4) – as well as by the interactions of school level by Text. Specifically, we asked what changes these dimensions of analytical writing experience across school levels, and which of these changes indicate increased skillfulness in the expression of standpoint, positioning, and reflective thinking.

For T1, students were prompted to write about freedom of dressing code, a hot topic for adolescents and young adults. For T2 and T4, students had to write about freedom of move from own country, a debated topic at a time of strong migration throughout Europe. T1 and T2, on different topics, had to be produced before the classroom activities while T4, on the same topic than T2, was produced after these. This enabled us to discuss the possible influence of topic versus the influence of pedagogical input.

We expected a positive effect of school level on the explicit expression of a central standpoint, a greater incidence of explicit expression with schooling. We expected a more flexible positioning as well: students’ texts would include not only claims or aspects of the topic that support their own view, but also elements attenuating and even opposing their own views. We also anticipated increasing production of reflective thinking as students advance in schooling. Given that the targeted dimensions are genre-constrained, we expect these would be more sensitive to genre-oriented instruction than to topic. Finally, informed by previous studies (e.g., Tolchinsky et al. 2021), we expected that external evaluation of text quality will be positively affected by school level and pedagogical work.

2.2 Methods

2.3 Dimensions and strategies of analysis

We focused on three textual dimensions assumed to indicate well-formedness of analytical essays: (1) explicitness of a central standpoint; (2) quality of students’ positioning, and (3) expression of reflective thinking. Each dimension was built on the subtypes regarding the differential qualities of rhetorical moves.

2.4 Results

Analyses were aimed at determining the impact of school level and pedagogical input on three textual dimensions: explicitness of the writer’s standpoint, quality of the writer’s positioning, and expression of reflective thinking.

2.5 Summary and discussion

One of DAW’s goals is to detect the developmental route of text features that indicate students’ appropriation of genre relevant features. To this end, we have traced development in three text dimensions committed to fulfill analytical writing communicative goals and text well-formedness: Explicitness of a central standpoint, Flexible positioning, and Expression of Reflective thinking. Explicit formulation of central standpoint is a core element in essays on controversial topics (Mann and Thompson 1988), and establishes a hierarchical differentiation of text components (Longacre 1996). While the Expression of (R)reflective thinking plays a crucial role in writing analytically, Flexible positioning is an additional asset in such essays (Kuhn and Crowell 2011). Overall, the results of the study showed how challenging is for novice writers to fulfill analytical genre expectancies regarding the realization of the argumentative component. The formulation of a central standpoint was realized in less than 30 % of the texts, and 80 % of the texts contained only devices supporting own position rather than deviating from own contention to build a flexible positioning. The realization of the expository component was less difficult, and 60 % of the texts contained at least one reflective move.

Analyses of the differential qualities of the texts’ rhetorical moves disclosed a similar developmental pattern in the probability to express a central standpoint and to include reflective commentaries. Both skills developed with schooling although significant differences were found only between elementary and higher school levels across texts. In contrast, the development of a flexible positioning was relatively conditioned by Text. University students displayed a higher probability to deviate from own contentions than younger schoolers but only in T1, when writing on freedom of a dressing code. Moreover, highschoolers showed a more flexible positioning than elementary and university students, but only after their participation in the classroom activities. These findings fit partially our expectation of a steady development across schooling for the three dimensions. The greatest developmental gains occurred toward high school and slowed down afterward. This stronger improvement during the years of learning to compose texts replicates previous findings for other text-features (Aparici et al. 2021).

In line with expectations, differences in topic did not affect the pattern of responses. Nevertheless, the development of a Flexible positioning – clearer when writing about a dressing code than when writing about freedom to move – hints at a facilitating effect of certain topics on decentering. This finding calls for research controlling for the relative influence of the tested factors (pedagogical input and topic-content) on the quality of analytical essays.

An additional goal of DAW’s was to detect potential improvement of genre relevant features. To this end we traced changes in the assessed dimensions after the implementation of pedagogical work. Unlike what we found regarding the developmental pattern, the three dimensions were similarly impinged by pedagogical scaffolding: it benefited mainly high school students and less university students. High schoolers were more prone to highlight their standpoint, attenuate their one-sided perspective and reflect on the target topic after their participation in the classroom activities. Elementary school students showed a relative improvement in the three dimensions although not significantly so.

Against expectations, pedagogical work or, perhaps, repeated text writing on the same topic seemed to negatively impact university students: their probabilities to underlie a standpoint, to show a flexible positioning and provide reflections on the topic decreased after classroom activities. The three dimensions assessed showed potentiality for improvement up and to the university level. However, university students would need pedagogical activities different than those used in earlier school levels to impinge their improvement.

Finally, results of regression analyses supported the foundational role of development which explained the highest proportion of variance, and the significant contribution of the three assessed dimensions. Flexible positioning, though, contributed less than Explicit central standpoint and Reflective thinking to reader’s appreciation of text quality. This finding hints at a certain dissonance between researchers’ and evaluators’ criteria regarding the features that better contribute to the quality of analytical essays.

3.1 DAW’s basic tenets

A basic tenet underlies the developmental focus and the control for pedagogical input as distinguishing features of our framework: tracing the trajectory on a particular domain of knowledge is valuable for teaching and learning. This value would increase if we could identify not only what students have to learn in order to become proficient in a certain domain, but also what they bring to the learning process.

Children’s growing sensitivity to different types of discourse is part of their pragmatic development (e.g., Donovan and Smolkin 2002). Over the years from childhood to adolescence and beyond, the heterogeneity of discourse practices increases. During this transition, young people are required to gain command in different types of discourse. This involves learning discourse conventions that result both from experience with written language and individuals’ cognitive development (Uccelli et al. 2012). We assumed that if we promote classroom activities that raise participants’ awareness of the particular motives and rhetorical patterns of analytical writing, we would be able to identify the insights learners gain through their participation in such practices. That is, we would be able to assess how pedagogical input impinges on how skills are transferred from social expectancies of genre to individuals, as part of their ongoing development (Kuhn and Crowell 2011).

A second set of tenets grounded two additional features of DAW: focus on the written product and multilayered analysis. Following Longacre’s (1996) proposal we assumed that the quality of a text can only be appreciated in light of the characteristic type of discourse that the text it belongs to. Discourse realizations in specific genres compel individual writer’ decisions concerning the amount and structuring of information, as well as the selection of devices at every level of language (Alamargot and Chanquoy 2001). The features of the written product account for the writer’s decisions. And, as semantic units, written texts proper functioning result from a delicate interplay between different linguistic levels – from words to overall text organization.

Consequently, we took a multilayered approach both in the different studies dwelling on text features at different language levels, and in the very process of text analysis. We focused on different levels of the written product – from the lexical, through syntactic-discursive to textual overall organization. However, when centering on one level, we were attentive to how the others were affected. For example, the relevance of particular syntactic constructions (e.g., use of relative clauses) was interpreted according to the role they fulfilled in the text. Likewise, when centered on featuring text structure, we paid attention to the incidence of modal expressions (i.e., qualifiers).

Finally, what novice and expert writers pursue with writing in academic settings prompted us to include a reader-based text appraisal as an additional feature of DAW. After all, writers’ ultimate goal is to persuade their readers―teachers, colleagues, reviewers, editors ― that they qualify as good enough to be members of the target discourse community. Besides our researcher-based assessment, we resorted to a reader-based assessment and tested the relevance of the selected features in the eyes of the actual judges/readers in natural academic settings, that is, teachers. This approach yielded crucial implications for research and teaching.

3.2 DAW’s applications and (possible) contribution to teaching

Across studies, we assessed diverse text-embedded features as indicators of developing writing proficiency. In Aparici et al. (2021) we examined lexical and syntactic-discursive indicators of quality. We could tell apart linguistic features that improved in the course of development from those that show malleability to pedagogical scaffolding i.e., higher potentiality for improvement by means of pedagogical work. However, none of the identified features showed a simple effect of pedagogical input. In the domain of analytical writing, pedagogical action impinges on development rather than the other way around. The sensitivity to pedagogical scaffolding shown by the improvement of most indicators of quality in a short time span of a genre-oriented instruction highlights the significance of pedagogical input for developing analytical writing proficiency but provided age/school level be considered when choosing which features to focus pedagogical work on.

The findings described for the impact pedagogical work has on proficient writing (expression of a central standpoint, mitigation of own contentions, and supply of reflective elaboration) confirmed the possibilities of pedagogical input, but also exposed some limitations. The applied classroom activities enhanced developmental gains in proficiency, but neither at every school level nor for every dimension.

Beyond the identification of indicators of essay writing proficiency, additional findings of this and previous studies might be helpful for educators. The first concerns the impact of pedagogical input on specific lexical and syntactic-discursive features rather than on particular language levels (e.g., the effect on lexical diversity but not on lexical density); this calls for diversified pedagogical intervention within each language domain. The second relates to the inverse relation between the experience of the writers and the efficacy of classroom activities. Less experienced writers obtained greater benefits than more experienced university students. This finding points at the need to tailor pedagogical work to suit the specific needs of writers to warrant further improvement. Again, this conclusion is reinforced by the above-reported results on the greatest developmental gains during elementary school for Explicit central standpoint, Flexible positioning and Reflective thinking.

An additional finding that calls for reflection concerns the strong and generalized impact of productivity on external assessment of text quality. Across schooling, producing more text made a better impression on external judges than appropriate text cohesion or high lexical richness. This puts into questions current advice on the benefits of concise writing (Williams and Bizup 2017). Being concise may serve to grab and hold reader’s attention but does not appear to be highly appreciated by regular teachers.

3.3 Future research

Our incursion in the qualitative, content-related differences of rhetorical moves enabled us to grasp the specific challenges that students must face to gain proficiency in analytical writing, in particular, the importance of expressing a central standpoint and a flexible positioning by providing pros and cons to their own views. Students’ difficulties to face these challenges, as reflected in the results we obtained, underlie the need of raising students’ awareness on how explicit expression of standpoint affects the “climatic development” (Longacre 1996) of their essays, and providing adequate pedagogical scaffolding to this end.

In light of the fruitful implications that we drew from applying a qualitative, content-related approach to the specific nature of rhetorical moves, future research should extend this approach to explore the type and quality of the grounds that students provide for the claims included in argumentative moves.

The applications of the model served to show developmental changes in the way novice (and more experienced) writers mobilize their linguistic repertoire to fit readers’ expectations on analytical essays. Two extensions are needed to probe DAW’s level of generality. We should extend this methodological framework to different languages and/or linguistically diverse populations to determine how language-specific or dependent on participants’ linguistic conditions the results are. Finally, only by extending this model to other types of discourse will we be able to demonstrate its useful gauge for the study of rhetorical flexibility.