Volume 33, Issue 3 - Data in generative grammar: The stick and the carrot

This paper is intended to lay out for broader discussion some arguments for the importance of data in work in generative syntax. These are accepted by many linguists, but a significant number of others still seem reluctant to accept them. The basic claim is that it is no longer tenable for syntactic theories to be constructed on the evidence of a single person's judgements, and that real progress can only be made when syntacticians begin to think more carefully about the empirical basis of their work and apply the minimum standards we propose. We advance two groups of reasons for syntacticians to do this, negative and positive. The negative ‘stick’ group concerns the inadequacy of current practice. We argue that linguists are producing unsatisfactory work with these methods. Data quality is a limiting factor: a theory can only ever be as good as its data base. The positive ‘carrot’ group concerns the descriptive and theoretical advantages which become available with more empirically adequate data. We hope to tempt linguists to adopt new methods by showing them the insights which better data makes available.

In this commentary, we discuss the limitations of linguistic judgements by comparing them to a variety of other empirical data types. In particular, we show that judgements are inherently multimensional in that they incorporate a range of linguistic and extra-linguistic influences, which cannot be teased apart by only considering the final judgement itself. Judgements of the gradient and time-insensitive type therefore give rise to the same interpretation problems as those faced by introspection. Thus, while we agree with Featherston that an improvement of data replicability in theoretical linguistics is highly desirable, we disagree with his claim that gradient judgements can provide significantly more grammatically-relevant information than other types of empirical data.

1. Introduction In the abstract of his target piece, Featherston states that ‘it is no longer tenable for syntactic theories to be constructed on the evidence of a single person's judgements’. In our commentary, we focus on this issue and on (what we perceive to be) Featherston's claims that individual speakers' judgments are intrinsically unreliable and ‘noisy’, that variation among individuals' judgments should be smoothed out by averaging the judgments of a large pool of informants, and that only this golden mean counts as genuine data. We argue that these claims are at odds with the basic premises of Chomskian linguistics, which is centered on the I-language of the individual speaker/hearer, not the E-language of the speech community. This is not to say that the way generative grammar approaches and accumulates its data is in no need of improvement: we will make some specific recommendations of our own to this end in the last section of our commentary.

0. One of the merits of generative syntax is that it has broadened the data base for syntactic research in a very profound way. There was, e.g., little (if any) discussion of the role syntax plays for anaphoric and pronominal binding before generative syntax came into being, and there was little or no knowledge about restrictions on the placement of elements into the left periphery of clauses, such as the island constraints of Ross (1967) or the crossover phenomena (Postal 1971). The pertinent facts were simply not known before generative grammar. The debate concerning the configurationality of German in the eighties of the last century is a further case in point; it led to the discovery of a multitude of facts about German syntax. This rapid and deep expansion of the empirical base of syntax was due to at least two factors: first, only highly predictive theories such as generative syntax can guide scientists as to where to look for interesting data, and the full acceptance of native speaker judgments as linguistic evidence yields an easy way of (dis-)confirming model assumptions – which was particularly important in times of limited corpus sizes and the absence of effective methods for searching in corpora.

In his paper “Data in generative grammar: the stick and the carrot”, Sam Featherston offers an interesting plea for a change in linguistic methodology. He criticizes the “inadequate praxis” of the majority of generative linguists to use their own intuitive judgments as evidence for their syntactic hypotheses. He claims that with this bad scientific practice, which has not changed in the thirty years after Greenbaum's (1977) harsh criticism of “dubious” empirical reasoning, generative linguists are producing unsatisfactory work and are thus undermining the reputation of syntax. On the basis of their introspective judgments, they have stated wrong empirical generalizations and, as a consequence, suggested inadequate theoretical accounts. As an alternative for the better he postulates “that theory building requires a better basis than this” and calls upon generative linguists to take more care with their data and to subject their theoretical claims to objective empirical tests such as acceptability rating experiments which are carried out with multiple informants, permit multiple degrees of well-formedness, and use multiple lexical variants of the structures to be checked. He illustrates the benefits of this empirical strategy with experiments on object-related anaphors in German as well as on that -trace and superiority effects in German, and thereby demonstrates how syntacticians have been led astray by their intuitive judgments.

It is easy to agree and equally easy to disagree with the two main positions of Sam Featherston, respectively, namely, that grammar theory would benefit from improving standards of data assessment, and, that syntactic well-formedness is inherently gradient rather than dichotomist. I subscribe to the first position and I see good grounds for not subscribing to the second claim. Since the two positions are dependent, denying the second has implications for accepting the first.

Since Featherston (henceforth ‘F’) has labeled me as a ‘conservative supporter of generative practice’ (272), I would guess that he expects me to devote my commentary to a defense of this practice. My first thought was to invoke what in Anglo-American legal tradition is called the ‘tu quoque defense’, essentially the rebuttal that, guilty as I may be, ‘you did it too’. That turned out not to be a viable option, since, to my knowledge, F has not published in syntactic theory per se. I will therefore broaden my defense to one of ‘omnes quoque’. From the title of his paper to its conclusion F implies that there is something intrinsic to generative grammar that invites partisans of that framework to construct syntactic theories on the evidence of a single person's judgments. Nothing could be farther from the truth. The great bulk of publications in cognitive and functional linguistics follow the same practice. Of course, rhetorically many of the latter decry the use of linguists' own intuitions as data. For example, in his editor's introduction to an important collections of papers in cognitivefunctional linguistics, Tomasello 1998: xiii deplores the data that are used in generative grammar, which ‘are almost always disembodied sentences that analysts have made up ad hoc’. Yet only two contributors to the volume (Wallace Chafe and Paul Hopper) present segments of natural discourse, neither filling even a page of text. All of the other contributions employ examples constructed by the linguists themselves. It is quite difficult to find any work in cognitive linguistics (and functional linguists are only slightly better) that uses multiple informants. It seems almost disingenuous of F to fault generativists for what (for better or worse) is standard practice in the field, regardless of theoretical allegiance.

The commenting authors have provided thoughtful contributions which I read with pleasure. All of them lend their weight to the call for work in synax to improve its empirical practice and apply a little more rigour in its treatment of data. The contributors also differ from my own positions in a number of ways, but most clearly on the issue of the theoretical construct grammaticality. They debate the virtues of my adoption of a gradient model of well-formedness, and discuss whether and to what extent the findings of empirical studies can bear upon the issue, an important topic for workers in the field to explore if work is to become more empirically adequate. Since the aim of my article was to stimulate debate, these differences can only be welcome, since they provide the reader with a direct comparison. The goal of this volume will have been achieved if each reader first, notes the degree of consensus on the issue of the reform of practice in syntactic theory, and second, is able to weigh up the differentpositions adopted on the way forward.