Startseite Wirtschaftswissenschaften Matching on Noise: Finite Sample Bias in the Synthetic Control Estimator
Artikel
Lizenziert
Nicht lizenziert Erfordert eine Authentifizierung

Matching on Noise: Finite Sample Bias in the Synthetic Control Estimator

  • Joseph Cummins , Douglas L. Miller ORCID logo EMAIL logo , Brock Smith und David Simon
Veröffentlicht/Copyright: 30. Oktober 2023
Veröffentlichen auch Sie bei De Gruyter Brill

Abstract

We investigate the properties of a systematic bias that arises in the synthetic control estimator in panel data settings with finite pre-treatment periods, offering intuition and guidance to practitioners. The bias comes from matching to idiosyncratic error terms (noise) in the treated unit and the donor units’ pre-treatment outcome values. This in turn leads to a biased counterfactual for the post-treatment periods. We use Monte Carlo simulations to evaluate the determinants of the bias in terms of error term variance, sample characteristics and DGP complexity, providing guidance as to which situations are likely to yield more bias. We also offer a procedure to reduce the bias using a direct computational bias-correction procedure based on re-sampling from a pilot model that can reduce the bias in empirically feasible implementations. As a final potential solution, we compare the performance of our corrections to that of an Interactive Fixed Effects model. An empirical application focused on trade liberalization indicates that the magnitude of the bias may be economically meaningful in a real world setting.

JEL Classification: C23; C52

Corresponding author: Douglas L. Miller, Cornell University and NBER, Ithaca, USA, E-mail:

We thank the Associate Editor and referees for their guidance and helpful comments.


References

Abadie, A., A. Diamond, and J. Hainmueller. 2010. “Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of california’s Tobacco Control Program.” Journal of the American Statistical Association 105 (490): 493–505. https://doi.org/10.1198/jasa.2009.ap08746.Suche in Google Scholar

Abadie, A., A. Diamond, and J. Hainmueller. 2015. “Comparative Politics and the Synthetic Control Method.” American Journal of Political Science 59 (2): 495–510. https://doi.org/10.1111/ajps.12116.Suche in Google Scholar

Abadie, A., and J. Gardeazabal. 2003. “The Economic Costs of Conflict: A Case Study of the Basque Country.” The American Economic Review 93 (1): 113–32. https://doi.org/10.1257/000282803321455188.Suche in Google Scholar

Acemoglu, D., S. Johnson, A. Kermani, J. Kwak, and T. Mitton. 2016. “The Value of Connections in Turbulent Times: Evidence from the united states.” Journal of Financial Economics 121 (2): 368–91. https://doi.org/10.1016/j.jfineco.2015.10.001.Suche in Google Scholar

Advani, A., T. Kitagawa, and T. Słoczyński. 2019. “Mostly Harmless Simulations? Using Monte Carlo Studies for Estimator Selection.” Journal of Applied Econometrics 34 (6): 893–910. https://doi.org/10.1002/jae.2724.Suche in Google Scholar

Arkhangelsky, D., S. Athey, D. A. Hirshberg, G. W. Imbens, and S. Wager. 2021. “Synthetic Difference-In-Differences.” The American Economic Review 111 (12): 4088–118. https://doi.org/10.1257/aer.20190159.Suche in Google Scholar

Athey, S., and G. W. Imbens. 2017. “The State of Applied Econometrics: Causality and Policy Evaluation.” The Journal of Economic Perspectives 31 (2): 3–32. https://doi.org/10.1257/jep.31.2.3.Suche in Google Scholar

Bai, J. 2009. “Panel Data Models with Interactive Fixed Effects.” Econometrica 77 (4): 1229–79.10.3982/ECTA6135Suche in Google Scholar

Ben-Michael, E., A. Feller, and J. Rothstein. 2021. “The Augmented Synthetic Control Method.” Journal of the American Statistical Association 116 (536): 1789–803. https://doi.org/10.1080/01621459.2021.1929245.Suche in Google Scholar

Billmeier, A., and T. Nannicini. 2013. “Assessing Economic Liberalization Episodes: A Synthetic Control Approach.” Review of Economics and Statistics 95 (3): 983–1001. https://doi.org/10.1162/rest_a_00324.Suche in Google Scholar

Bohn, S., M. Lofstrom, and S. Raphael. 2014. “Did the 2007 Legal arizona Workers Act Reduce the State’s Unauthorized Immigrant Population?” Review of Economics and Statistics 96 (2): 258–69. https://doi.org/10.1162/rest_a_00429.Suche in Google Scholar

Busso, M., J. DiNardo, and J. McCrary. 2014. “New Evidence on the Finite Sample Properties of Propensity Score Reweighting and Matching Estimators.” Review of Economics and Statistics 96 (5): 885–97. https://doi.org/10.1162/rest_a_00431.Suche in Google Scholar

Cavallo, E., S. Galiani, I. Noy, and J. Pantano. 2013. “Catastrophic Natural Disasters and Economic Growth.” Review of Economics and Statistics 95 (5): 1549–61. https://doi.org/10.1162/rest_a_00413.Suche in Google Scholar

Chen, A., A. Graves, M. Resnick, and R. Michael. 2018. “Does Spending More Get More? Health Care Delivery and Fiscal Implications from a Medicare Fee Bump.” Journal of Policy Analysis and Management 37 (4): 706–31. https://doi.org/10.1002/pam.22084.Suche in Google Scholar

Courtemanche, C. J., and D. Zapata. 2014. “Does Universal Coverage Improve Health? the massachusetts Experience.” Journal of Policy Analysis and Management 33 (1): 36–69. https://doi.org/10.1002/pam.21737.Suche in Google Scholar PubMed

Cunningham, S., and M. Shah (2017). “Decriminalizing Indoor Prostitution: Implications for Sexual Violence and Public Health.” The Review of Economic Studies 85 (3): 1683–715. https://doi.org/10.1093/restud/rdx065.Suche in Google Scholar

Donohue, J. J., A. Aneja, and K. D. Weber. 2019. “Right-to-carry Laws and Violent Crime: A Comprehensive Assessment Using Panel Data and a State-Level Synthetic Control Analysis.” Journal of Empirical Legal Studies 16 (2): 198–247. https://doi.org/10.1111/jels.12219.Suche in Google Scholar

Doudchenko, N., and G. W. Imbens. 2016. Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis, Technical report. National Bureau of Economic Research.10.3386/w22791Suche in Google Scholar

Eren, O., and I. S. Ozbeklik. 2016. “What Do Right-To-Work Laws Do? Evidence from a Synthetic Control Method Analysis.” Journal of Policy Analysis and Management 35 (1): 173–94. https://doi.org/10.1002/pam.21861.Suche in Google Scholar

Ferman, B., and C. Pinto. 2016. Revisiting the Synthetic Control Estimator.Suche in Google Scholar

Ferman, B., and C. Pinto. 2017. Placebo tests for Synthetic Controls, Technical report. Munich: University Library of Munich.Suche in Google Scholar

Ferman, B., and C. Pinto. 2021. “Synthetic Controls with Imperfect Pretreatment Fit.” Quantitative Economics 12 (4): 1197–221. https://doi.org/10.3982/qe1596.Suche in Google Scholar

Ferman, B., C. Pinto, and V. Possebom. 2020. “Cherry Picking with Synthetic Controls.” Journal of Policy Analysis and Management 39 (2): 510–32. https://doi.org/10.1002/pam.22206.Suche in Google Scholar

Fitzpatrick, M. D. 2008. “Starting School at Four: The Effect of Universal Pre-kindergarten on Children’s Academic Achievement.” The B.E. Journal of Economic Analysis & Policy 8 (1): 1–40, https://doi.org/10.2202/1935-1682.1897.Suche in Google Scholar

Frölich, M. 2004. “Finite-sample Properties of Propensity-Score Matching and Weighting Estimators.” Review of Economics and Statistics 86 (1): 77–90. https://doi.org/10.1162/003465304323023697.Suche in Google Scholar

Gautier, P. A., A. Siegmann, and A. Van Vuuren. 2009. “Terrorism and Attitudes towards Minorities: The Effect of the Theo Van Gogh Murder on House Prices in Amsterdam.” Journal of Urban Economics 65 (2): 113–26. https://doi.org/10.1016/j.jue.2008.10.004.Suche in Google Scholar

Gurantz, O. 2020. “What Does Free Community College Buy? Early Impacts from the oregon Promise.” Journal of Policy Analysis and Management 39 (1): 11–35. https://doi.org/10.1002/pam.22157.Suche in Google Scholar

Hinrichs, P. 2014. “Affirmative Action Bans and College Graduation Rates.” Economics of Education Review 42: 43–52. https://doi.org/10.1016/j.econedurev.2014.06.005.Suche in Google Scholar

Hu, L., R. Kaestner, B. Mazumder, S. Miller, and A. Wong. 2018. “The Effect of the Affordable Care Act Medicaid Expansions on Financial Wellbeing.” Journal of Public Economics 163: 99–112. https://doi.org/10.1016/j.jpubeco.2018.04.009.Suche in Google Scholar PubMed PubMed Central

Huber, M., M. Lechner, and C. Wunsch. 2013. “The Performance of Estimators Based on the Propensity Score.” Journal of Econometrics 175 (1): 1–21. https://doi.org/10.1016/j.jeconom.2012.11.006.Suche in Google Scholar

Jones, D., and L. Marinescu. 2022. “The Labor Market Impacts of Universal and Permanent Cash Transfers: Evidence from the Alaska Permanent Fund.” American Economic Journal: Economic Policy 14 (2): 315–40.10.1257/pol.20190299Suche in Google Scholar

Kaul, A., S. Klößner, G. Pfeifer, and M. Schieler. 2022. “Standard Synthetic Control Methods: The Case of Using All Preintervention Outcomes Together with Covariates.” Journal of Business & Economic Statistics 40 (3): 1362–76. https://doi.org/10.1080/07350015.2021.1930012.Suche in Google Scholar

Kiesel, K., and S. B. Villas-Boas. 2013. “Can Information Costs Affect Consumer Choice? Nutritional Labels in a Supermarket Experiment.” International Journal of Industrial Organization 31 (2): 153–63. https://doi.org/10.1016/j.ijindorg.2010.11.002.Suche in Google Scholar

Klasik, D. 2013. “The Act of Enrollment: The College Enrollment Effects of State-Required College Entrance Exam Testing.” Educational Researcher 42 (3): 151–60. https://doi.org/10.3102/0013189x12474065.Suche in Google Scholar

Kreif, N., R. Grieve, D. Hangartner, A. J. Turner, S. Nikolova, and M. Sutton. 2016. “Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units.” Health Economics 25 (12): 1514–28. https://doi.org/10.1002/hec.3258.Suche in Google Scholar PubMed PubMed Central

Larramona, G., and M. Sanso-Navarro. 2016. “Do Regularization Programs for Illegal Immigrants have a Magnet Effect? Evidence from Spain.” The Manchester School 84 (2): 296–311.10.1111/manc.12099Suche in Google Scholar

Lee, W.-S. 2010. “Comparative Case Studies of the Effects of Inflation Targeting in Emerging Economies.” Oxford Economic Papers 63 (2): 375–97. https://doi.org/10.1093/oep/gpq025.Suche in Google Scholar

Nannicini, T., and R. Ricciuti. 2010. Autocratic Transitions and Growth. Munich: CESifo.10.2139/ssrn.1563996Suche in Google Scholar

Peri, G., and V. Yasenov. 2019. “The Labor Market Effects of a Refugee Wave Synthetic Control Method Meets the Mariel Boatlift.” Journal of Human Resources 54 (2): 267–309. https://doi.org/10.3368/jhr.54.2.0217.8561r1.Suche in Google Scholar

Pinotti, P. 2015. “The Economic Costs of Organised Crime: Evidence from Southern italy.” The Economic Journal 125 (586): F203–F232. https://doi.org/10.1111/ecoj.12235.Suche in Google Scholar

Smith, B. 2013. Cross-Country Determinants of Growth: A Microeconometric Approach. Davis: University of California.Suche in Google Scholar

Smith, B. 2015. “The Resource Curse Exorcised: Evidence from a Panel of Countries.” Journal of Development Economics 116: 57–73. https://doi.org/10.1016/j.jdeveco.2015.04.001.Suche in Google Scholar

Sun, J., F. Wang, H. Yin, and B. Zhang. 2019. “Money Talks: The Environmental Impact of china’s Green Credit Policy.” Journal of Policy Analysis and Management 38 (3): 653–80. https://doi.org/10.1002/pam.22137.Suche in Google Scholar

Xu, Y. 2017. “Generalized Synthetic Control Method: Causal Inference with Interactive Fixed Effects Models.” Political Analysis 25 (1): 57–76. https://doi.org/10.1017/pan.2016.2.Suche in Google Scholar


Supplementary Material

This article contains supplementary material (https://doi.org/10.1515/jem-2021-0019).


Received: 2021-07-29
Accepted: 2023-09-25
Published Online: 2023-10-30

© 2023 Walter de Gruyter GmbH, Berlin/Boston

Heruntergeladen am 5.2.2026 von https://www.degruyterbrill.com/document/doi/10.1515/jem-2021-0019/pdf?lang=de
Button zum nach oben scrollen