To drop or not to drop? Predicting the omission of the infinitival marker in a Swedish future construction

2.1 Omission of the infinitival marker

Swedish has a future tense construction consisting of the auxiliary kommer (present tense of the verb komma with the literal meaning ‘come’) and an infinitive. The infinitive is preceded by the optional infinitival marker att. (1a) illustrates the construction with att present, (1b) shows att-omission, as used in the Swedish online discussion forum Familjeliv (see Section 3.1).

(1a) Ja, vi tror att väntetiden kommer att vara 1-1,5 år framöver.

‘Yes, we think the waiting time is going to be 1–1.5 years in the future.’

(1b) Så kommer det nog vara några veckor till.

‘It probably is going to be like that for some more weeks.’

The kommer (att) construction is not the only way to refer to the future in Swedish. Future tense may also be expressed by present tense inflection or a construction consisting of the future tense auxiliary skola ‘shall’ and an infinitive (Teleman et al. 1999 IV: 243–250).

The kommer (att) construction is part of a larger family of constructions consisting of an auxiliary or auxiliary-like verb plus an infinitive with optional att (Bylin 2013; Lagervall 1999; Mjöberg 1950; Teleman et al. 1999), as illustrated in the constructed examples (2a) to (2c). The parentheses indicate that the infinitival marker is optional.

(2a) Julen börjar (att) närma sig.

‘Christmas is starting to approach.’

(2b) Ibland vägrar hon (att) sitta kvar i vår famn.

‘Sometimes, she refuses to remain sitting in our arms.’

(2c) Jag försöker (att) inte visa min oro inför barnen.

‘I try not to show my worry to the kids.’

Optional infinitival markers are not restricted to Swedish but have also been observed in other Germanic languages. In English, the infinitival marker to may be left out after the verbs help (Kjellmer 1985; Levshina 2018; Lind 1983; Lohmann 2011; Mair 2002; McEnery and Xiao 2005) and try (Kjellmer 2000).

(3a) Sarah helped us (to) edit the book. (Quirk et al. 1985: 1205).

(3b) I have never seen a fish try (to) get warmed by a fire. (Kjellmer 2000: 116).

In Dutch, the infinitival marker te is optional after some aspectual verbs (4a) and the modal verb hoeven ‘have to’ (4b) (Van de Velde 2015, 2017; Coussé forthcoming).

(4a) Volgens mij is het de bedoeling dat we niet keiveel herrie zitten (te) maken .

‘I think the idea is that we should not be making a lot of noise.’ (Coussé forthcoming: §18.2.4)

(4b) Ze heeft er niet veel moeite voor hoeven (te) doen .

‘She did not have to make a lot of effort.’ (Coussé forthcoming: §18.2.4)

In this section, we review the factors that, according to the literature, affect the omission of the infinitival marker in the future kommer (att) construction. We also complement them with related factors suggested for optional infinitival markers in other verb constructions in Swedish, English and Dutch. In this study, we consciously focus mostly on language-internal, structural predictors (but also time and genre), leaving potential language-external predictors for future studies.

2.2 Language-external predictors

2.3 Language-internal predictors

In this subsection, we describe hypotheses about language-internal predictors and their cognitive motivation.

3.1 Corpora

In order to test to what extent the factors outlined in Section 2 enable us to predict which variant of the kommer (att) construction will be used (in a specific utterance or during a given period), we need large, deeply annotated, time-stamped corpora. Since the change has been particularly active during approximately the last two decades, we need corpora that contain recent texts. We use seven Swedish corpora created and maintained by Språkbanken Text (SBX) that satisfy these criteria. The basic information about the corpora is presented in Table 1, the data extraction and annotation processes are described in the rest of this section, more details about the corpora are provided in Appendix A. In addition to those seven corpora, we generated an “All” version for which we aggregated all available observations. All data were collected in January–September 2022.

To generate balanced samples per year, we manually inspected each raw dataset and decided on a corpus-specific time span and sample size (denoted as N _sample), shown for each corpus in columns 2 and 3 in Table 2. Per corpus, we then randomly drew N _sample observations per year (the samples are available in the Supplementary material). Depending on the chosen time span, we used the first N _train years to train our models and last N _test years to evaluate our models, shown in columns 4 and 5 in Table 2. Note that while we measure time in years for the purpose of generating balanced samples, we will measure it in months when training and testing the models. Thus, for DA, for instance, the test set contains not 2, but 2 · 12 = 24 periods (and 2,000 observations).

3.2 Data extraction and annotation

The data were downloaded using the API (https://ws.spraakbanken.gu.se/docs/korp) for the SBX search engine Korp (Borin et al. 2012). All sentences containing at least one relevant observation, that is, the verb komma in the present tense (normally kommer) followed by an infinitive, were extracted, if the distance between kommer and the infinitive was not larger than six tokens. (The distance between two immediately adjacent tokens was taken to be 1; tokens include both words and punctuation marks).

All SBX corpora are annotated by the Sparv pipeline (Hammarstedt et al. 2022), which provides information about part of speech, morphological features, lemma, dependency structure and more. Using this annotation, all observations were automatically labelled either as “noise” (there is no syntactic link between kommer and the infinitive; the example should be discarded), “att” (the infinitive’s head is the word att, the head of which is kommer) or “omission” (the infinitive’s syntactic head is kommer, there is no att). The values of the predictors (see Section 2 for a general description, Section 3.4 for operationalizations) were also automatically calculated.

The accuracy of the part-of-speech and morphological annotation of the SBX corpora is estimated to vary between 0.91 and 0.96, depending on the genre and the version of Sparv (Adesam and Berdicevskis 2021), while the quality of the syntactic annotation is lower: the labeled attachment score (LAS) is estimated to vary between 0.78 and 0.84 (Berdicevskis 2020). In social-media corpora, which often contain unedited non-standard texts, the annotation quality is likely to be lower for both morphology and syntax. The Flashback, Familjeliv, SVT, and DA corpora were recently added or updated, and were annotated with a newer version of Sparv which typically yields higher accuracy.

It is thus not a given that the available annotation, especially the dependency tree structure, is good enough for the current study. In order to test the quality of the data extraction and annotation, we performed a manual spot check, see Section 3.3. The extraction and annotation scripts are provided in Supplementary materials.

3.3 Manual spot check of the annotation quality

We randomly selected 300 observations from the downloaded data for the Familjeliv forum. The classification and all feature values were manually checked by AB, EC and YA. The annotated sample is provided in Supplementary materials.

Out of 300 observations, eight (2.7%) were misclassified. Table 3 shows a confusion matrix of the values generated by our automatic annotation and the values of our manual evaluation.

If the example was classified correctly, most of the feature values in the checked examples were also correct. The only features which contained errors were those related to the syntactic subject of the construction and the length of the infinitive chain (see Section 3.4).

Two systematic limitations were made clear by the analysis of the sample. First, if there are several coordinated infinitives, we focus only on the first one, consider (10):

(10) Just den här eftermiddagen kommer jag att dra ut telefonjacket och låta bli att sätta mig framför datorn

‘But this afternoon, I am going to pull the phone out of its jack and avoid sitting down in front of the computer’ (Familjeliv)

It is difficult to estimate how good the annotation of coordinated structures in the corpora is and whether we can reliably extract relevant structures. Besides, adding non-first conjuncts would arguably introduce another factor which can potentially affect the omission of att.

Second, it is possible that the data contain constructions with kommer which have the same syntactic structure, but a different, non-future tense, meaning (Teleman et al. 1999 II: 511–512). One example where another interpretation is possible is (11), where kommer att tänka på ‘come to think of’ does not signal future, but rather chance.

(11) Såg ett tidigare inlägg och bör nog precisera mig: Jag behöver inga tips på vad jag ska trycka i mig för jag är inte sugen på något av alla saker som jag kommer att tänka på

‘Saw an earlier post and should probably specify: I don’t need tips for what I should chow down on because I’m not keen on any of the things I can think of’ (Familjeliv)

There is no reliable method which could have filtered out these constructions, especially since in many cases they can be argued to be ambiguous. However, they are most probably infrequent (one ambiguous example in the 300-observation sample). Johansson (2006: 139–140) reports a higher proportion of the non-futural constructions (35 out of 576 in her sample, extracted from a corpus of newspaper texts published in 1995–1997), but even with this estimate, their frequency is still rather low.

Overall, with a classification accuracy of 97.3%, we deem the annotation to be reliable enough. No major systematic biases were identified. In the subsequent data processing, we also exclude examples where more than one att occurres between kommer and the infinitive (which is extremely unlikely), as well as those where the syntactic subject of kommer is placed after att (which is ungrammatical). The column "#observations” in Table 1 refers to the number of observations after all these filters were applied.

3.4 Operationalization of the predictors

All examples which were labelled as noise were discarded, that is, we only kept those where there was a syntactic link between kommer and the infinitive (via att, if it was present). We then operationalized the notions outlined in Section 2 by calculating the values of the following predictors (see Appendix A for more details).

Language-external predictors

year, month (see Section 2.2.1: “Time”): the year and the month when the text was written. The distribution of observations over time is very uneven, but we deal with that by drawing balanced samples (see Section 3.1).

genre (see Section 2.2.2: “Genre”): we perform our analysis separately for each corpus.

Language-internal predictors

attraction (see Section 2.3.1: “Predictability”): the conditional probability of a verb given the kommer (att) construction, calculated as the ratio of the frequency of the verb with kommer (att) to the frequency of kommer (att). Note that this measure, unlike all others, is not a property of an observation, but of a corpus. It is calculated using only the samples included in the training set (see Sections 3.1 and 3.6).

distance to att (see Section 2.3.2: “Distance”): the distance from kommer to the infinitival group, that is, to att if present, else to the infinitive. This predictor is considerably skewed: for instance, in the “All” corpus, over 57% of all utterances have a distance of one word. Given this skewness, we made the decision to transform these predictors into binary variables (distance of one word vs. distance of two or more words) to ensure that we have adequate data for model fitting, as suggested by Bresnan et al. (2007).

length of the infinitive chain (see Section 2.3.3: “Length of the infinitive chain”): the infinitive which has kommer as a syntactic head may have another infinitive as a dependent, which, in turn, may have another, and so on. This feature gauges the complexity of the infinitival group by measuring the number of infinitives in it. This predictor is considerably skewed: for instance, in the “All” corpus, nearly 90% of all utterances have a length of one word. Given this skewness, we made the decision to transform these predictors into binary variables (length of one word vs. distance of two or more words).

subject (see Section 2.3.4: “Presence of a syntactic subject”): whether kommer has a syntactic non-clausal subject. If kommer is coordinated with another verb, only the first verb is treated as having a subject (i.e. if kommer is not the first one, it is treated as subjectless). Due to the limitations of the syntactic annotation, this variable is slightly less reliable than others (approximately 2% of the observations in the sample were misannotated).

att before (see Section 2.3.5: “Presence of another att”): a binary variable: whether there is another att in the same sentence before kommer. We do not make any attempt to distinguish between att as an infinitive marker and as a subordinator.

att after: same as att before: whether there is another att in the same sentence after kommer.

voice (see Section 2.3.6: “Voice of the infinitive”): active or s-form. Most s-forms are passive voice (e.g. att påverkas ‘to be influenced’ vs. att påverka ‘to influence’), but some are so-called deponent verbs where the passive form is used with an active meaning (and an active form is missing), e.g. att hoppas ‘to hope’. No attempt was made to distinguish deponent verbs from true passives.

The dataset (available in the Supplementary materials) also contains values of other potentially relevant features as well as detailed metadata.

3.5 Visualization of the effect of the predictors

We perform a simple correlation test, in order to preliminarily check whether our expectations about the relation between the individual predictors and the dependent variable are correct. Two predictors are continuous (time, attraction) and six are binary (or have been simplified to be binary). We calculate the phi coefficient (also known as the Matthews correlation coefficient, see e.g. Chicco and Jurman 2020) between each of the binary predictors and the outcome. For binary variables, phi is equal to the Pearson correlation coefficient. To have a comparable measure for continuous predictors, we calculate the Pearson correlation coefficient between each of them and the outcome (since the outcome is a binary variable, the Pearson coefficient is equivalent to the point-biserial correlation coefficient, for which the assumption of linearity does not apply). We use the training set for each corpus (see Section 3.1). The predictors are coded in a way which should yield a positive correlation if the direction of the effect is in agreement with our expectations and a negative correlation if it is opposite.

The results are visualized in Figure 1.

Figure 1:

The Pearson correlation coefficients between the individual predictors and the outcome (equivalent to the phi coefficient for binary variables). Negative values indicate effects that go against our expectations.

Most predictors do have a certain relation to the outcome, and its direction is usually consistent across different corpora, but its size is always small. Interestingly, for att before and subject the direction of potential effect is almost always opposite to the one we expected. For att after, the direction is inconsistent (which, together with the small values, suggests that there is no effect). Note that while negative values are not what we expected, they are not per se a problem for the predictive approach: if there is a relation, it can be exploited, regardless of its direction.

3.6 Statistical analysis

We focus on the predictive accuracy of statistical models by training them on part of the data (training set) and then measuring their performance on a held-out portion of the data (test set).

Note that at no point do we attempt to estimate statistical significance. This is a conscious choice. First, the very notion of statistical significance has recently been strongly criticized, see Wasserstein et al. (2019) about the limitations and pitfalls of this approach in general and Koplenig (2019) about corpus linguistics in particular.

Second, the predictive approach addresses a question which is somewhat different from questions that are central in many other corpus studies. The question is not “how large is the effect?” (effect size) or “how probable is it that the effect of the same or larger size could have arisen in a sample of a given size if there is no effect in the population?” (statistical significance), but “if we have a theory which claims that the predictors affect the variable in a specific way and know the values of the predictors, can we find out what the value of the variable is?”, which we find equivalent to “how good is our explanatory theory actually?” and thus of utmost importance.

To find out how well we can predict variation and change, we use two different approaches: Approach A that is based on a logistic regression model and Approach B that is based on an autoregressive integrated moving average (ARIMA) model. Logistic regression is a statistical technique that models the relationship between a binary outcome (taking on values of either 0 or 1) and one or more predictor variables. The model uses a logistic function to estimate the probability that an observation belongs to either outcome and a threshold is used to make a final prediction. Logistic regression plays an important role in a wide range of fields and disciplines, such as epidemiology, psychology and sociology or (corpus) linguistics, where researchers try to understand the relationship between an outcome and various predictors. In addition, logistic regression is also widely used in machine learning, where models are trained on large datasets to make predictions about unseen data. This approach has numerous applications, including image or text classification, spam detection, and the prediction of medical outcomes, such as the risk of hospitalization or mortality in patients with chronic diseases. For a comprehensive and accessible introduction to logistic regression, see Hosmer et al. (2013).

ARIMA is a technique that is used to describe and analyse time-series data. The resulting model can then be used to make predictions about future values of the time-series. Due to the temporal nature of the data, most of the methods that are used to analyse cross-sectional data cannot be used to analyse time-series datasets. One of the most important issues arising in this context is that many time series exhibit autocorrelation, i.e. the value of a variable at one moment in time is correlated with values of its own past and its own future. Not accounting for this aspect of time-series will lead to incorrect statistical inference and inaccurate predictions. ARIMA models that are widely used to predict time-dependent values (e.g. food prices, inflation rates, unemployment rates, weather patterns, spread of diseases or language change) solve this problem by incorporating a number of past values of the time-series in order to predict its value at a given time step, which is called an autoregression model (the AR part). In addition, ARIMA models also include a weighted moving average of past forecast errors as predictors (the MA part). Finally, the ‘I’ in ARIMA refers to the process of making a time-series stationary, meaning transforming it to a series whose statistical properties do not change over time. For a comprehensive and accessible introduction to time series analysis and ARIMA models, see Hyndman and Athanasopoulos (2018).

In what follows, both approaches will be explained more formally.

3.6.3 Summary

To sum up, we use Approach A (based on logistic regression) to address both Q1 (predictions for a given utterance, goodness-of-fit measured as G A ) and Q2 (prediction of proportion for a given period, goodness-of-fit measured as G B ). We use Approach B (based on ARIMA) to address Q2.

The data analysis was done in Stata 17.0 and Python 3.6.8. The commented code is available in Supplementary material.

4.1 Approach A (logistic regression)

Figure 2 visualizes the results of the logistic classification (measure G t e s t A , addressing Q1). For each corpus, the percentage of correctly predicted test utterances is visualized per model (also see Appendix C for a table with confusion matrix information for all corpora and all four models). In 2 out of our 8 corpus samples (Flashback and SVT), the prediction accuracies of the models that include predictors (M ^II, M ^III and M ^IV) do not top the baseline prediction accuracy, i.e. G t e s t A ( M I ) . For the other 6 corpus samples, M ^III is the best-fitting model for All, Bloggmix, DA, Familjeliv and GP. For Twitter, M ^IV is the best-fitting model. However, the difference between G t e s t A ( M I I I ) or G t e s t A ( M I V ) and G t e s t A ( M I ) as a measure of prediction improvement is below 1 percentage points in all such cases. This indicates that including predictors hardly helps in predicting the outcome.

Figure 2:

G t e s t A for each model (M ^I, M ^II, M ^III and M ^IV). Per corpus, an asterisk indicates the best-fitting model(s). For Familjeliv, the difference is not obvious from the figure, since the values are rounded to two decimal places.

In Appendix C, we show that if we use a different performance measure (balanced accuracy) which punishes the models that always predict only one variant (as M I does), then one of the linguistically-informed models ( M I I I or M I V ) is always the best, but the improvement over baseline (that is, the difference between the best model and the baseline model) is still small (from 0.25 percentage points for Twitter to 4.12 percentage points for DA).

In Appendix D, we show that the obtained results are somewhat better if we split the data into training/test in a non-consecutive manner, i.e. if utterances are randomly split into 80% training and 20% test data. If we use accuracy, one of the linguistically-informed models is the best for all corpora except for DA (the improvement ranges from 0.17 percentage points for GP to 5.15 for Bloggmix). If we use balanced accuracy, one of the linguistically-informed models is always the best (the improvement ranges from 1.42 for SVT to 9.33 for Bloggmix).

Figure 3 shows time-series line plots for all corpus samples where the monthly proportion of att-omissions is the outcome. In each plot, the training data is depicted in black colour and the test data is coloured in mint green. In 2 out of our 8 corpus samples (Bloggmix and Twitter), the prediction accuracies of the models that include predictors do not top the baseline prediction accuracy, i.e. G t e s t B ( M I ) . While in the remaining 6 cases, the models with the lowest G t e s t B all include language-internal predictors, a visual inspection shows that the fit between the test data and the predicted proportions is rather poor in all cases.

Figure 3:

Proportion of att-omissions as a function of time and corpus. In each plot, G t e s t B for each logistic model is given in brackets. An asterisk indicates the best-fitting model per corpus.

4.2 Approach B (ARIMA)

Figure 4 visualizes the results of the full forecast based on the ARIMA models with P m as the outcome. As above, training data is depicted in black colour and test data is coloured in mint green. In 5 out of our 8 corpus samples (All, DA, Familjeliv, Flashback and SVT), the prediction accuracies of the models that include predictors (M ^II) do not top the baseline prediction accuracy, i.e. G t e s t B ( M I ) .

Figure 4:

Proportion of att-omissions as a function of time and corpus. In each plot, G t e s t B for each ARIMA model is given in brackets. Model predictions are based on a full forecast, where the whole test data set is predicted at once. An asterisk indicates the best-fitting model per corpus.

Figure 5 shows the corresponding results of the one-step-ahead forecast. A visual inspection shows that for all corpora, calculated forecasts based on both M ^I or M ^II closely match the observed test data. The comparison between forecast accuracies of models without predictors (M ^I) and models with predictors (M ^II), show that only in half of our corpus samples (All, Flashback, GP and Twitter), M ^II achieves a better prediction accuracy than M ^I, i.e.  G t e s t B ( M I I ) < G t e s t B ( M I ) .

Figure 5:

Proportion of att-omissions as a function of time and corpus. In each plot, G t e s t B for each ARIMA model is given in brackets. Model predictions are based on a one-step-ahead forecast where, in each month m, the estimated model parameters are used to predict the value of P in the subsequent month, i.e. P ˆ m + 1 . An asterisk indicates the best-fitting model per corpus.

Taken together and in line with the results presented for Approach A, the results for the ARIMA model imply that including the chosen language-internal predictors does not really help to predict the proportion of att-omissions for both the full and the one-step-ahead forecast. In Appendix E, we demonstrate that the lack of prediction improvement has to do with the fact that statistical associations between (i) the outcome and (ii) the predictors are not strong enough.