Presented to you through Paradigm Publishing Services

John Benjamins Publishing Company

Visit our Partner Page See all our books
Chapter
Licensed
Unlicensed Requires Authentication

Chapter 2. From lexical bundles to surprisal and language models

Measuring the idiom principle in native and learner language
  • and

Abstract

We exploit the information theoretic measure of surprisal to analyze the formulaicity of lexical sequences. We first show the prevalence of individual lexical bundles, then we argue that abstracting to surprisal as an information-theoretic measure of lexical bundleness, formulaicity and non-creativity is an appropriate measure for the idiom principle, as it expresses reader expectations and text entropy. As strong and gradient formulaic, idiomatic and selectional preferences prevail on all levels, we argue for the abstraction step from individual bundles to measures of bundleness. We use surprisal to analyse differences between genres of native language use, and learner language at different levels: (a) spoken and written genres of native language (L1); (b) spoken and written learner language (L2), across selected written genres; (c) learner language as compared with native language (L1). We thus test Pawley and Syder (1983)’s hypothesis that native speakers know best how to play the tug-of-war between formulaicity (Sinclair’s idiom principle) and expressiveness (Sinclair’s open-choice principle), which can be measured with Levy and Jaeger (2007)’s uniform information density (UID) which is a principle of minimizing comprehension difficulty. Our goal to abstract away from word sequences also leads us to language models as models of processing, first in the form of a part-of-speech tagger, then in the form of a syntactic parser. While our hypotheses are largely confirmed, we also observe that advanced learners bundle most, and that scientific language may show lower surprisal than spoken language.

Abstract

We exploit the information theoretic measure of surprisal to analyze the formulaicity of lexical sequences. We first show the prevalence of individual lexical bundles, then we argue that abstracting to surprisal as an information-theoretic measure of lexical bundleness, formulaicity and non-creativity is an appropriate measure for the idiom principle, as it expresses reader expectations and text entropy. As strong and gradient formulaic, idiomatic and selectional preferences prevail on all levels, we argue for the abstraction step from individual bundles to measures of bundleness. We use surprisal to analyse differences between genres of native language use, and learner language at different levels: (a) spoken and written genres of native language (L1); (b) spoken and written learner language (L2), across selected written genres; (c) learner language as compared with native language (L1). We thus test Pawley and Syder (1983)’s hypothesis that native speakers know best how to play the tug-of-war between formulaicity (Sinclair’s idiom principle) and expressiveness (Sinclair’s open-choice principle), which can be measured with Levy and Jaeger (2007)’s uniform information density (UID) which is a principle of minimizing comprehension difficulty. Our goal to abstract away from word sequences also leads us to language models as models of processing, first in the form of a part-of-speech tagger, then in the form of a syntactic parser. While our hypotheses are largely confirmed, we also observe that advanced learners bundle most, and that scientific language may show lower surprisal than spoken language.

Chapters in this book

  1. Prelim pages i
  2. Table of contents v
  3. Acknowledgements vii
  4. Chapter 1. Present applications and future directions in pattern-driven approaches to corpus linguistics 1
  5. Part I. Methodological explorations
  6. Chapter 2. From lexical bundles to surprisal and language models 15
  7. Chapter 3. Fine-tuning lexical bundles 57
  8. Chapter 4. Lexical obsolescence and loss in English: 1700–2000 81
  9. Part II. Patterns in utilitarian texts
  10. Chapter 5. Constance and variability 107
  11. Chapter 6. Between corpus-based and corpus-driven approaches to textual recurrence 131
  12. Chapter 7. Lexical bundles in Early Modern and Present-day English Acts of Parliament 159
  13. Part III. Patterns in online texts
  14. Chapter 8. Lexical bundles in Wikipedia articles and related texts 189
  15. Chapter 9. Join us for this 213
  16. Chapter 10. I don’t want to and don’t get me wrong 251
  17. Chapter 11. Blogging around the world 277
  18. Index 311
Applications of Pattern-driven Methods in Corpus Linguistics
This chapter is in the book Applications of Pattern-driven Methods in Corpus Linguistics
https://doi.org/10.1075/scl.82.02sch

Chapters in this book

  1. Prelim pages i
  2. Table of contents v
  3. Acknowledgements vii
  4. Chapter 1. Present applications and future directions in pattern-driven approaches to corpus linguistics 1
  5. Part I. Methodological explorations
  6. Chapter 2. From lexical bundles to surprisal and language models 15
  7. Chapter 3. Fine-tuning lexical bundles 57
  8. Chapter 4. Lexical obsolescence and loss in English: 1700–2000 81
  9. Part II. Patterns in utilitarian texts
  10. Chapter 5. Constance and variability 107
  11. Chapter 6. Between corpus-based and corpus-driven approaches to textual recurrence 131
  12. Chapter 7. Lexical bundles in Early Modern and Present-day English Acts of Parliament 159
  13. Part III. Patterns in online texts
  14. Chapter 8. Lexical bundles in Wikipedia articles and related texts 189
  15. Chapter 9. Join us for this 213
  16. Chapter 10. I don’t want to and don’t get me wrong 251
  17. Chapter 11. Blogging around the world 277
  18. Index 311
Downloaded on 5.4.2026 from https://www.degruyterbrill.com/document/doi/10.1075/scl.82.02sch/html
Scroll to top button