Dealing with Censored Earnings in Register Data

2.1 In the Lab: a Parametric Earnings Distribution

We illustrate the main procedure (and a proof of concept and performance evidence) using artificial data. So that the model be realistic, we fit the parametric model to monthly earnings data taken from the SOEP for male workers in West Germany in 2018. Specifically, we assume a heavy-tailed GB2 model, the generalized beta distribution of the second kind, with the density function.

The GB2 parameters are estimable by maximum likelihood. It is well-known that the GB2 model is a member of family (1) and that the extreme value index equals γ = 1/(ap). See e.g. Schluter and Trede (2024) for an application of the GB2 model. Maximum likelihood then yields the following estimates: ( a ˆ = 4.037 ; b ˆ = 3633.295 ; p ˆ = 0.757 ; q ˆ = 0.825 ) . Figure 1 reveals that the fitted parametric GB2 model provides a close approximation to the earnings data.

Figure 1:

GB2-fit for earnings of West German men in 2018. Displayed is the empirical earnings distribution in the sample of West German men in 2018 (in bars) and the according gb2 fit (red curve). The dashed vertical line represents the earnings assessment ceiling at € 6,370, which would induce a censoring incidence of 11.22 %. The Stata package gb2fit yields the following parameters: a ̂ = 4.037 , b ̂ = 3633.295 , p ̂ = 0.757 , q ̂ = 0.825 .

We then use the fitted GB2 model as our data generating process (DGP), with implied γ = 0.327 and an earnings share for the top 1 % of 4.7 %.

Figure 2 illustrates the estimator of γ for one such random sample using beyondpareto. Panel (A) depicts the Pareto QQ-plot for the upper earnings tail and shows its approximate linearity. We fit a straight line with slope γ ˆ ; the vertical dashed line depicts the value corresponding to the optimally determined Y_base. In panel (B), we depict γ ˆ as a function of the upper order statistics k, and the optimal k* corresponds to Y_base. Beyond this k*, the estimator is increasingly distorted.

Figure 2:

Diagnostic plots of beyondpareto in GB2 example. Panel A depicts the Pareto QQ-plot for one random sample based on the GB2 model. Panel B depicts the corresponding extreme value index plot. The dashed vertical lines represent the optimal Y_base.

Is the good fit for this one synthetic distribution just a coincidence? To answer this question, we perform a Monte Carlo study, generating 1,000 synthetic distributions of size 10,000. For each synthetic distribution, we estimate the extreme value index γ. Row 1 of Table 1 displays tail index estimate γ ˆ , the optimally AMSE-based chosen earnings threshold Y_base, and the earnings share of the top 1 %, including the standard deviations across all 1,000 samples. In this lab setting, beyondpareto performs very well: the estimated values are very close to the true population values and the empirical standard deviations across simulations suggest tight estimates.

Regarding the top earnings shares, Row 3 of Table 1 reports the empirical sample values that exhibit substantial distortions, illustrating the need for appropriate statistical modelling: The observed downward bias reflects the nature of heavy-tailed distributions, since top earnings in a random sample tend to be under-represented (even, as in our case, for samples of size 10,000).

3.1 German Socio-Economic Panel

The German Socio-Economic Panel, SOEP, is one of the largest and longest-running multidisciplinary household surveys worldwide (Goebel et al. 2019). It is a random draw of all private households in Germany. In every household, all adults are asked to provide, amongst many other items, incomes from all sources. Most importantly for our purposes, it asks for individual gross labor earnings from dependent employment, the determinant of pension contributions and entitlements. Accordingly, the data provide, in contrast to German register data, information on the distribution of earnings below and also above the assessment ceiling.

3.2 SOEP-RV

SOEP-RV is based on a record linkage project with the German Pension Insurance (Deutsche Rentenversicherung Bund), in which the survey data of SOEP respondents have been linked on a 1:1 basis with their individual social security biographies.

The insurance data have a very broad coverage, as the vast majority of employees in Germany are mandatorily insured. The pension insurance keeps an account for each of these employees – for both the employment phase and retirement phase. As contributions are closely linked with pension entitlements in the German system, contributors’ earnings biographies are carefully recorded and contained in individual insurance accounts at the German Pension Insurance. However, social insurance contributions in Germany are capped at an annually adjusted earnings threshold, leading to right-censored (topcoded) earnings in all administrative social security datasets.

The SOEP data have been linked with administrative records from the German Pension Insurance, creating the SOEP-RV dataset (Lüthen et al. 2022). For all SOEP respondents who gave explicit consent,^[7] SOEP-RV links, on an individual level, the SOEP data with the pension account biographies. The present study is based on the longitudinal dataset, called SOEP-RV.VSKT2020.^[8] The longitudinal data comprise exact information on pension-relevant status, such as pension recipient, education, or dependent employment, and in case of the latter the associated earnings points and gross earnings for up to 624 months starting from January in the calendar year in which an individual turns 14 years old (Forschungsdatenzentrum der Rentenversicherung 2024). Throughout, we will use interchangeably ‘administrative earnings’ and ‘Insurance Accounts (IA) earnings.’ In the next section, we use SOEP-RV to assess the differences between self-reported earnings in the SOEP and the administrative records below the censoring threshold. Their close correspondence then enables our research design in a lab-style setting: Above the censoring threshold, we use uncensored self-reported earnings to assess the performance of imputation techniques for censored administrative earnings.

3.3 Comparability of Earnings in SOEP and Register Data

As with all surveys, respondents’ self-reported earnings may be subject to measurement errors, whereas register earnings are, presumably, reliable. Another difference between the two data sources concerns the accounting period: SOEP earnings refer to the month before the interview, while register earnings refer to employment spells based on which monthly earnings are derived. Another issue are one-time payments: Such payments are subject to social security and are part of register earnings – unless these earnings already exceeded the assessment ceiling. In SOEP, one-time payments are contained in a separate variable that refers to the full year and that is surveyed retrospectively. Therefore, for comparability reasons, we divide annual one-time payments by 12 and add it to the reported regular earnings.

To use the SOEP as an external validation tool, SOEP and register earnings must be sufficiently “close,” allowing credible comparisons between the top-tail of SOEP earnings and the tail of censored register data. Figure 3 provides an initial descriptive assessment. It is a scatter plot of self-reported SOEP earnings versus administrative IA earnings in the year 2018 for record-linked men in West Germany with IA earnings below the assessment ceiling of € 6,370. The black line is the 45-degree bisector, the red functional form shows the result of a locally weighted regression – and thus a highly flexible form. The figure shows that both earnings variables are highly correlated with a correlation coefficient (ρ) of 0.92. The in-depth analysis in Schröder et al. (2023) supports that the SOEP data are very suitable as comparative statistic for the register data.

Figure 3:

Scatter plot of IA and SOEP earnings below the censoring threshold. Linked SOEP-RV data for men in West Germany with IA earnings below the assessment ceiling. The correlation coefficient is ρ = 0.92. The black line marks the 45-degree line. The blue vertical line indicates the assessment ceiling. The red line results from a locally weighted regression. Four outliers with self-reported earnings above 8,000 Euro are excluded in the scatter plot but included in the regression. Cases with statistically imputed SOEP earnings are excluded. Furthermore, individuals who work less than half of the month are excluded, as part of these individuals receive wage subsidies that might be included in the SOEP but not in the register data.

3.4 Working Samples Restrictions

The literature on earnings distributions in Germany using administrative data usually focuses on male workers in West Germany (see Section 4.3 below for examples). We follow this practice, but present in Appendix A the analyses for women in West and all workers in East Germany.

We focus on 2018, the year with the largest number of linked observations. The 2018 West German assessment ceiling is € 6,500 per month and we follow common practice of scaling it by 0.98 to account for imprecision in the register earnings. Thus, the effective censoring threshold is € 6,370 and, based on the weighted SOEP data, the censoring incidence among West German men is 12.2 %.

To construct the SOEP-RV sample, we proceed as follows: As SOEP earnings refer to the pre-interview month, we check the employment status in the register data in that exact month. If the status indicates employment subject to social security in West Germany for at least half of that month, we consider the registered earnings for that month. For the larger SOEP sample, we follow the same restrictions, except for two minor deviations: 1) For the SOEP sample, we cannot rely on the pension status of employment subject to social security in West Germany and, thus, consider all dependent employees earnings above € 450 living in West Germany (excluding Berlin). Thus, instead of the workplace location, the sample restriction is based on residence. 2) Since the exact pension records of days employed in a month are not available for the large SOEP sample, we cannot restrict the sample to individuals who worked for at least half of the reference month. Both of these small differences in sample construction have a negligible – if any – impact for our empirical exercises. Table 2 summarizes these selection restrictions, and reports the resulting sample sizes.

4.1 Top Earnings: the SOEP Sample

We use the SOEP sample as a first benchmark, before considering the smaller SOEP-RV subsample.

Table 3 summarizes the results for the SOEP sample. The target value of the extreme value index based on the uncensored earnings data and an optimal choice of Y_base (Row 1) is γ ˆ = 0.24 . The implied estimate of the top 1 % earnings share is 4.01 % and, for those above the administrative censoring limit, 26.57 %. These results also substantiate our claim that the parametric GB2 approach (see Row 1 of Table 1) is outperformed. The optimally chosen Y_base = € 7,992 exceeds the assessment ceiling of € 6,370. Figure 5 depicts the Pareto QQ-plot of the upper order statistics of these uncensored SOEP earnings and the fitted line with slope γ ˆ = 0.24 at the optimal Y_base = 7, 992 choice. Panel (b) depicts γ ˆ as a function of the upper order statistics k, while the optimal k* corresponds to Y_base. Row 2 of Table 3 shows the effect of restricting Y_base to be below censoring ceiling, leading to a restricted Y_base of 6,200. Consequently, γ ˆ increases slightly to 0.26 and the top 1 % earnings share increases marginally to 4.16 %. This modest increase in γ ˆ is further illustrated in panel (b) of Figure 5, where the restricted choice of k associated with Y_base = 6,200 is identified by the solid grey vertical line. Finally, in Row 3, we artificially censor the SOEP earnings data at the assessment ceiling and incorporate the resulting mass point in our estimation. Thanks to the weighting procedure of our estimator, the estimate γ ˆ is slightly above the target value of 0.24. However, since Y_base is smaller, the implied earnings share of the top 1 % is now 4.21 %.

Figure 5:

Diagnostic plots of beyondpareto in the SOEP sample. Panel A depicts the Pareto QQ-plot in the SOEP sample of West German men. Panel B depicts the corresponding extreme value index plot. The dashed vertical lines represent the optimal Y_base. The dotted line (panel B) represents the optimal Y_base in the restricted setting.

Thus, we conclude that censoring of the data at the assessment ceiling does not prevent us from achieving a very good approximation of the tail of the earnings distribution.

Finally, we verify that the results of our analysis of top earnings in the focal year of 2018 are qualitatively representative for other recent years. To this end, Table 4 summarizes the results for the years 2014–2017 and 2019. For all years, the estimates of γ using the censored data get close to the target values, slightly overestimating the latter and implying a slight overestimation of the 1 % and 5 % top earnings shares. Overall, as regards the evolution of top earnings, the 2014-19 period is a period without major changes or trends: measured earnings concentration fluctuates without direction within a narrow band.^[10]

4.2 Top Earnings: The SOEP-RV Sample

We now consider the smaller SOEP-RV sample. Table 5 reports the results. Row 1 considers again the uncensored SOEP survey earnings to establish the target values. The extreme value index estimate is γ ˆ = 0.27 , which slightly exceeds the value for the complete SOEP sample. Hence, potentially selective consent into the record-linkage has little effect. The earnings share of the top 1 % is 4.27 %. Y_base is at € 6,200 and thus below the administrative censoring threshold. As a consequence, Row 2 where Y_base is forced to be below the administrative censoring threshold equals Row 1. Row 3 censors the reported survey earnings at the social security assessment ceiling. In line with the results of Table 3 (and the simulation setting of Table 1), our estimation approach accommodates the censoring very well, with only minor effects on γ ˆ or the top earnings share. Specifically, γ ˆ = 0.26 . The earnings share of the top 1 % is somewhat reduced to 4.10 %. Finally, in Row 4 we consider the matched administrative but right-censored earnings reports, i.e. the usual setting a researcher using administrative data would confront. The table reveals that our estimation approach gets close to the target value of γ ˆ = 0.27 ! The estimated top earnings shares are reduced by small amount.

We conclude that our estimation approach enables us to provide a credible estimate for the right tail of the earnings distribution based on censored administrative earnings. Figure 6 visualizes the setting and the results: The histogram (brown bars) depicts the earnings density below the administrative censoring ceiling alongside the mass point of the latter (transparent brown bar). We then smoothly paste a Pareto tail (blue line) based on our estimate of γ ˆ = 0.24 .

Figure 6:

IA earnings distribution: censored and fitted tail. Displayed is the empirical distribution of IA earnings in the SOEP-RV sample (probability density function), with the mass point at the assessment ceiling (transparent), and the fitted tail. Source: SOEP-RV.VSKT.2020-v2.

4.3 Tobit Imputations

A popular way of dealing with top-censored earnings in German administrative data is to impute based on individual-level Tobit regressions. In the regressions’ predictions, practitioners generally add an error term drawn from a group-specific (log) normal distribution (see e.g. Dustmann et al. (2009) and Card et al. (2013) in case of the well-known Integrated Labour Market Biographies (SIAB), provided by the Institute for Employment Research in Germany (IAB)). Dauth and Eppelsheimer (2020) provide the de facto standard data wrangling code for IAB data, including the Tobit-based imputation procedure. Drechsler and Ludsteck (2025), assessing the performance of these imputations, observe a sharp-discontinuity of the earnings density at the censoring threshold for prime-aged university educated male workers (for whom the censoring incidence is above 30 %) and offer several strategies how to alleviate the deficiencies of Tobit-based imputations.

For the implementation of the Tobit procedure, we stick to a more simple approach and follow Dauth and Eppelsheimer (2020) as closely as our SOEP-RV data allow us to.^[11] For our experiments, we consider the setting of Table 5 and focus on the top earnings shares. The Tobit model implies a tail decay that is faster than the power function of equation (1), so γ in this case equals the limit value of 0 (see e.g. Schluter and Trede (2019) for a discussion and tests of power function behavior against subexponentiality).

Figure 7 compares the resulting empirical distribution function (EDF) for Tobit-imputed earnings to the target EDF of uncensored SOEP earnings. The vertical line marks the censoring threshold. Also included is the cumulative distribution function (CDF) based on beyondpareto’s extreme value estimate. While the Tobit imputation does produce an approximation for the missing right tail of the censored earnings distribution that captures its heaviness fairly well, our CDF is always closer to the target EDF. The Tobit imputation systematically assigns earnings that are too large.

Figure 7:

IA earnings distribution: censored and fitted tail. Displayed are the distribution functions based on beyondpareto, Tobit estimations and the empirical survey earnings in the SOEP-RV sample. Source: SOEP-RV.VSKT.2020-v2.

This is further illustrated in Figure 8, which compares the implied top earnings shares with the target shares based on the SOEP sample (Table 3, row 1). For maximum comparability, we imputeartificially censored SOEP earnings, to prevent deviations between target and imputed shares due to differing earnings concepts. For the vast majority of top earnings shares, beyondpareto’s estimates are closer to the target than the empirical shares of the Tobit imputation. For example, the target earnings share of the top 5 % (10 %) is 13.71 % (23.12 %). The estimated share based on beyondpareto, is very close at 13.70 % (22.78 %), while Tobit imputation yields slightly higher shares of 15.03 % (24.52 %). Generally, the beyondpareto procedure slightly under-predicts, while the Tobit imputation over-predicts. Only in the extreme part of the tail do all three shares become, by necessity, very close.

Figure 8:

Comparison of top earnings shares based on different imputation methods. Displayed are the cumulative earnings shares of top percentiles as well as the share of all earnings above the censoring threshold in the target estimates (see Table 3, row 1), based on the beyondpareto procedure and based on Tobit estimations. For the years 2017 and 2019, the Pareto estimates are even closer to the target, see Figures A.2 and A.3 in the Appendix. Source: SOEPv38.1.