The Missing Window of Opportunity and Quasi-Experimental Effects of Institutional Integration: Evidence from Ukraine

2.1 Approach

Our approach to measure institutional quality in Ukrainian regions is constrained by the lack of observable and measurable characteristics to provide insights into the variation in institutional quality. To address these concerns, our empirical strategy to measure subnational institutional quality relies on the extraction of the residual component of institutional quality from the observable aggregate institutional quality series using the variation in pre-determined geographic characteristics (Focacci et al. 2023). By using a well-established series of governance indicators (Kaufmann et al. 2011), our aim is to extract a latent variable component from higher-level aggregation and project it to the subnational-level using the set of pre-determined characteristics that cannot be empirically manipulated. The extraction of the residual component of institutional quality conveys an important intuitive advantage compared to the existing quality measures. A zero residual level indicates the level of institutional quality A positive residual component indicates the level of institutional quality that is better than the one plausibly expected in geographically similar areas in the control sample. A zero residual component invariably suggests that the level of institutional quality is on par with the level expected in geographically similar areas. By contrast, a negative residual component suggests that the implied level of quality is worse than what would be expected in the areas with geographically similar endowment. Nevertheless, it should be acknowledged that our proposed measure of institutional quality is a residual rather than directly observed measure. We also collect region-level data on observable outcomes for the entire sample of Ukrainian regions on the election outcomes in 2004 and 2010, voter turnout in 2015, as well as protest activity during the Maidan Revolution and civilian causalities during the Russian-Ukrainian war since 2014, and correlate our residual measures with the observed counterparts. Against this backdrop, the correlation between our residual measures and real-time observed institutional and political variables is between +0.61 and +0.82, and is statistically highly significant (i.e. p-value = 0.000). Supplementary Appendix 1 provides a more elaborate and extensive discussion of the residual component and its validity.

2.2 Sample

Our treatment sample consists of 24 Ukrainian provinces (i.e. oblasts)^[5] and two cities with special status.^[6] Our control sample comprises 195 regions from the Central and Eastern European countries that were admitted to the European Union in 2004 and later, which includes Bulgaria, Croatia, Czech Republic, Hungary, Latvia, Lithuania, Poland, Romania, Slovakia and Slovenia^. Our respective period of investigation comprises the years between 1996 and 2020. Our overall sample covers 5525 region-year observations pooled into a strongly balanced panel. The size of the treatment sample comprises 650 region-year paired observations whilst the size of the control sample comprises 4875 paired observations.

3.1 Setup

Our goal is to examine the effect of remaining outside of the European integration process to the quality of institutions consistently. In this respect, our aim is to estimate the missing counterfactual scenario and construct the trajectories of institutional quality of Ukrainian regions in the hypothetical absence of the missing integration process. To this end, we apply the synthetic control estimator (Abadie and Gardeazabal 2003; Abadie et al. 2015; Gobillon and Magnac 2016; Gilchrist et al. 2023) to estimate the effect of missing European integration on the quality of institutions. By adopting the framework of potential outcomes, we simulate the region-level trajectories of institutional quality by making use of the implicit attributes of region-level residualized institutional quality from Central and Eastern European countries that have joined and completed the European integration process in the respective period of investigation. This particular approach allows us to build the respective counterfactual realization of institutional quality using weighted averages of outcome-specific, region-level attributes and characteristics obtained by making use of several machine-learning features before the stay-out period, that best track and reproduce the path of residualized institutional quality of Ukrainian regions. Using the benchmark path of pre-treatment outcomes and auxiliary geographic covariates as the battery of predictive variables and dynamic nested optimization with a Newton-Raphson algorithm to find the convex root of the differentiable objective function, we show that our modification of the synthetic control estimator provides an excellent quality of the pre-treatment fit in the residualized trajectories of institutional quality further reinforced by low imbalance between Ukrainian regions and their synthetic counterparts. Our modification of the synthetic control estimator is described in greater detail in Supplementary Appendix 2.

A potential caveat against the validity of the treatment effect arises from the choice of treatment year. Since admission to the European Union is contingent on a series of institutional and policy reforms,^[7] most of the countries included in the donor pool implemented these reforms through a variety of financial mechanisms. Some countries in the donor pool also joined Eurozone and Schengen area that ended internal border controls and established a common visa policy. To partially mitigate these concerns, it is therefore crucial that these reforms are more or less completed by the time of the treatment year. In this respect, beginning the stay-out treatment in 2004 may be problematic since a strong anticipatory component is present in the donor pool. To address this particular caveat, we base our treatment year in 2007 when the European Commission’s reform screening process was largely completed in all countries included in our donor pool.

3.2 Inference

To evaluate the significance of the effect of remaining outside of the European integration, we rely on the standard treatment permutation test, and ask whether the estimated gap in institutional quality is obtained by chance. Although large-sample asymptotic inference is not permittable when making use of a synthetic control estimator, the use of the treatment permutation method for inferential purposes may detect whether or not the estimated institutional quality gaps are relatively unique to Ukraine or are also perceptible in other regions in the donor pool. To fill the void in the analysis, we perform a simple in-space placebo analysis, by iteratively shifting the stay-out to the other regions in the donor pool. Our interpretation is simple and straightforward. If the estimated post-2008 gaps are also perceivable in other regions across the Central and Eastern European donor pool, then the notion of a significant effect associated with staying-out of the integration process may not be credible. By contrast, if the estimated gaps are unique to the Ukrainian regions and cannot be detected elsewhere, then the supposition of a significant effect becomes more credible and noteworthy. Through the iterative assignment of the stay-out to the unaffected regions, the p-values on the null hypothesis are computed for each outcome-year pair in the post-treatment period (Cavallo et al. 2013) which yields a very large number of placebo gaps to be estimated. Further details behind the placebo analysis are elicited in Supplementary Appendix 3.

4.1 Baseline

Figure 1 presents the average treatment effect of remaining out of the European Union for the full province-level treatment sample. Our synthetic control estimates suggest that Ukrainian provinces have developed substantially worse institutional quality by their exclusion from the European Union compared to the regions and counties of countries that have entered the European Union in 2004 and later. The estimated set of average treatment effects also indicates an invariably worse trajectory of institutional quality after 2008, a notable outcome-specific heterogeneity. More specifically, the largest drop in institutional quality outcome is perceptible for the level of political stability and absence of violence which appears to deteriorate, on average, by 40 percent (i.e. p-value = 0.000) vis-á-vis the synthetic Central and Eastern European control groups. In a similar vein, our synthetic control estimates indicate a pervasive deterioration in the control of corruption, which seems to worsen by 20 percent (i.e. p-value = 0.000) and in regulatory quality by 11 percent (i.e. p-value = 0.000) whereas rule of law tends to retrograde by 15 percent (i.e. p-value = 0.000) relative to the condensed synthetic control groups. The estimated retrogression of the rule of law, political stability and the control of corruption appears to be permanent in that it does not seem to disappear up to early 2020. By contrast, we find evidence of a temporary deviation in the trajectories of regulatory quality and government effectiveness which seem to worsen sharply by 23 percent by 2015 whilst the negative effect tends to disappear up to the end-of-sample period and is not statistically significant at conventional levels.

Figure 1:

Average region-level treatment effect of remaining out of the European integration process on institutional quality trajectories across Ukraine, 1996–2020.

Hence, our results imply that a hypothetical admission of Ukraine to the EU would substantially improve the institutional quality, whereas the admission to the EU does not appear to be a necessary condition for improving the institutional quality of government administration where plausible policy alternatives can equally improve the efficacy of the administration. Given the previously recognized importance of the rule of law, political stability and the control of corruption for sustained economic growth and social development, our conjecture is that the admission into a more inclusive institutional integration would generate marked, substantial and permanent gains in institutional quality. A more exhaustive and detailed discussion of results is elaborated in Supplementary Appendix 3.

4.2 Large-Sample In-Space Placebo Analysis

Our approach to tackle the statistical significance of the effect of staying out of the European Union on the institutional quality of Ukrainian provinces consists of the in-space placebo analysis. In particular, the stay-out of the European Union is assigned to the regions in the donor pool that have not stayed out of the European Union which effectively shifts Ukrainian provinces to the donor pool. Through an iterative application of the synthetic control estimator to the unaffected regions, we estimate a large vector of placebo gaps and evaluate the estimated effects for Ukrainian provinces compared to the effect sequence from placebo runs. If the institutional quality effects estimated for Ukrainian provinces are particularly large and unique, the notion of significant effects associated with remaining outside of the European Union becomes more credible. By contrast, if the gaps estimated for Ukrainian provinces do not seem to be substantially different from the placebo gaps, then our conclusion is that the analysis fails to provide evidence of a significant effect of staying out of the European Union. For each institutional quality outcome under consideration, we compute two-tailed p-values and proceed with the benchmark rejection rule implying no effect whatsoever through inversely weighted placebo gaps. Given the size of the treatment sample and the donor pool and the length of the pre-treatment period, we compute more than 34 billion placebo gap averages. A relatively large number of placebo averages allows us to use an almost full random sampling approach towards treatment permutation and improves the consistency of the estimated p-values denoting the probability of no effect whatsoever. Apart from the standard p-values, we also undertake a simple difference-in-differences analysis of placebo gap coefficients to determine parametrically whether the gaps estimated for Ukrainian provinces are statistically significantly different from the placebo gaps. Each specification also contains the full set of region-fixed effects and time-fixed effects to absorb the confounding influence of the heterogeneity bias from the estimated post-treatment coefficient.

Table 1 reports difference-in-differences analysis of in-space placebo gaps of remaining out of the European Union. Our key parameter of interest is the average post-2007 institutional quality gap across Ukrainian provinces for each institutional outcome under consideration. The evidence from the placebo analysis uncovers a high degree of uniqueness of the institutional quality gap. Controlling for the unobserved region-specific and time-varying heterogeneity, the estimated coefficients for the Ukrainian province-level post-treatment gaps are large and statistically significant at the 1 percent level. Consistent with the estimated magnitudes, the largest post-treatment coefficients are found with respect to the political stability and absence of violence, control of corruption and the strength of the rule of law. The difference-in-differences analysis thus largely confirms the significance of the estimated gaps for Ukrainian provinces and highlights relatively high losses associated with remaining outside of the European Union. A more granular and detailed inference on the treatment effect of the stay-out is reported in Supplementary Appendix 4. Furthermore, Supplementary Appendix 5 provides a more extensive empirical discussion of the correlates behind the institutional quality gaps.

4.3 Synthetic Difference-in-Differences Estimates

One potential caveat arises from the feasibility and potential existence of a parallel trend assumption. To fill the void in the literature, Arkhangelsky et al. (2021) propose a hybrid synthetic control and difference-in-differences estimation combining the attractive features of both methods.^[8] Since the details behind the synthetic difference-in-differences estimator extend beyond the goals of this paper, we discuss its properties and counterfactual weights in greater detail in Supplementary Appendix 6.

Table 2 reports the synthetic difference-in-differences estimated institutional quality impact of staying out of the European integration on the full sample of Ukrainian provinces. The table reports the differences in each institutional quality outcome denoted by δ adjusted for region- and time-specific weights. The point estimates confirm the evidence of widespread deterioration in the trajectories of institutional quality compared to the Central and Eastern European donor pool. In particular, synthetic difference-in-differences estimates indicate pervasive deterioration in democratic accountability and freedom of the press as well as substantially more deleterious efficacy of the government administration alongside significantly diminished rule of law and weaker enforcement of anti-corruption framework. For each respective outcome, the null hypothesis of zero treatment effect of exclusion from the European Union is rejected at 1 %, respectively (i.e. p-value = 0.000). The trajectory of regulatory framework for private-sector development appears to be the only component unaffected by the missing window of European integration (i.e. p-value = 0.488). Overall, the deterioration in institutional quality trajectories appears to be permanent. Figure 2 exhibits outcome-specific institutional quality gaps associated with the missing window of European integration, and reiterates the evidence of relatively large losses associated with missing the European integration process. The estimated losses are particularly large and confirm pervasive deterioration of political stability, corrosive weakening of the rule of law, and substantial annihilation of the control of corruption. Region-specific weights in the composition of synthetic control groups are reported and detailed in Supplementary Appendix 7.

Figure 2:

Synthetic difference-in-differences estimated institutional quality gaps of exclusion from the European Union across Ukrainian provinces, 1996–2020.