Skip to main content
Presented to you through Paradigm Publishing Services

John Benjamins Publishing Company

Visit our Partner Page See all our books
Chapter
Licensed
Unlicensed Requires Authentication

A method for the comparison of general sequences via type-token ratio

  • , and

Abstract

This article proposes a new method for analyzing and comparing general linear sequences with the minimum prior knowledge on the sequences needed. Sequence analysis is a broad problem studied by various fields from sociology and computer security to linguistics or biology. The method presented here applies the simplest quantitative linguistic tools in order to achieve methods transparency and easily interpretable results. The results form a vector describing the sequence and allow their clustering, machine learning and simple visualizations by line charts or multidimensional methods as MDS or tSNE. For completeness, artifacts and several formal models are derived to describe methods behavior in both common and extreme cases.

Abstract

This article proposes a new method for analyzing and comparing general linear sequences with the minimum prior knowledge on the sequences needed. Sequence analysis is a broad problem studied by various fields from sociology and computer security to linguistics or biology. The method presented here applies the simplest quantitative linguistic tools in order to achieve methods transparency and easily interpretable results. The results form a vector describing the sequence and allow their clustering, machine learning and simple visualizations by line charts or multidimensional methods as MDS or tSNE. For completeness, artifacts and several formal models are derived to describe methods behavior in both common and extreme cases.

Chapters in this book

  1. Prelim pages i
  2. Table of contents v
  3. Introduction 1
  4. Part I. Theory and models 7
  5. On the impact of the initial phrase length on the position of enclitics in Old Czech 9
  6. Term distance, frequency and collocations 21
  7. A method for the comparison of general sequences via type-token ratio 37
  8. Quantitative analysis of syllable properties in Croatian, Serbian, Russian, and Ukrainian 55
  9. N -grams of grammatical functions and their significant order in the Japanese clause 69
  10. Linking the dependents 93
  11. Grammar efficiency and the One-Meaning–One-Form Principle 109
  12. Distribution and characteristics of commonly used words across different texts in Japanese 121
  13. Part II. Empirical studies 135
  14. The perils of big data 137
  15. From distinguishability to informativity 145
  16. A Modern Greek readability tool 163
  17. Phonological properties as predictors of text success 177
  18. Calculating the victory chances 195
  19. Topological mapping for visualisation of high-dimensional historical linguistic data 209
  20. Book genre and author’s gender recognition based on titles 225
  21. Quantitative analysis of bibliographic corpora 239
  22. Analysis of English text genre classification based on dependency types 257
  23. In memory of Gabriel Altmann 271
  24. Index 277
Language and Text
This chapter is in the book Language and Text
https://doi.org/10.1075/cilt.356.03mat

Chapters in this book

  1. Prelim pages i
  2. Table of contents v
  3. Introduction 1
  4. Part I. Theory and models 7
  5. On the impact of the initial phrase length on the position of enclitics in Old Czech 9
  6. Term distance, frequency and collocations 21
  7. A method for the comparison of general sequences via type-token ratio 37
  8. Quantitative analysis of syllable properties in Croatian, Serbian, Russian, and Ukrainian 55
  9. N -grams of grammatical functions and their significant order in the Japanese clause 69
  10. Linking the dependents 93
  11. Grammar efficiency and the One-Meaning–One-Form Principle 109
  12. Distribution and characteristics of commonly used words across different texts in Japanese 121
  13. Part II. Empirical studies 135
  14. The perils of big data 137
  15. From distinguishability to informativity 145
  16. A Modern Greek readability tool 163
  17. Phonological properties as predictors of text success 177
  18. Calculating the victory chances 195
  19. Topological mapping for visualisation of high-dimensional historical linguistic data 209
  20. Book genre and author’s gender recognition based on titles 225
  21. Quantitative analysis of bibliographic corpora 239
  22. Analysis of English text genre classification based on dependency types 257
  23. In memory of Gabriel Altmann 271
  24. Index 277
Downloaded on 27.4.2026 from https://www.degruyterbrill.com/document/doi/10.1075/cilt.356.03mat/html?lang=en
Scroll to top button