Cognitive Biases in Government Procurement – An Experimental Study
-
Omer Dekel
and
Amos Schurr
Abstract
Competitive bidding (CB) is the dominant governmental contracting mechanism by which hundreds of billions of dollars are allocated annually. We claim that when bid evaluators assess the qualitative components of competing bids while being exposed to the bid prices, a systematic bias occurs that gives an unjust advantage to the lower bidder. We term this the Lower-Bid Bias. It is then shown that this bias can be neutralized by splitting the evaluation process into two stages, whereby bid price is revealed only after the evaluation process has culminated (two-stage CB). This is demonstrated through the findings of a survey and three controlled experiments, the first to be conducted with procurement officials. We also explain why this bias is undesirable and suggest a mandatory rule, requiring two-stage CB for any competitive public procurement based on evaluation criteria other than price. Further applications of the experiments’ findings are also discussed.
Appendices
Appendix A: Experiment 2a form
The municipality of Ramat-Gan conducted a Request for Proposals (RFP) for the transportation of the city’s schoolchildren during the 2011–2012 school year. A weight of 60% was given to the bid price and 40% was given to quality criteria. Two bids were submitted.
[1st version: The bid prices were submitted in separate sealed envelopes, to be opened after examination of the other parts of the bids (two-stage sealed bidding).]
[2nd version: “The Carrier Ltd.” submitted the lowest bid, at $800,000, and received the full 600 points (out of 1,000) for the price criterion. The other bid, submitted by “The Transporter Ltd.,” for $875,000, scored only 550 points for the price criterion.]
| Criterion9 | Max. points | “The Carrier” | Points | “The Transporter” | Points |
| [2nd version only: Price] | 600 | $800,000 | 600 | $875,000 | 550 |
| Years of operation | 60 | 8 years | 12 years | ||
| Bus model | 30 | Mercedes “VIP” 2011 | Mercedes “Dolphin” 2010 | ||
| Recommendations | 70 | 2 good, 1 moderate | 5 good | ||
| No. of buses owned by the bidder | 60 | 12 buses | 20 buses | ||
| Average experience of the drivers | 70 | 12 years | 20 years | ||
| Training hours per driver | 30 | 35 hours | 20 hours | ||
| ISO certification | 50 | Not certified, but in the final stages of the certification process | Has certification | ||
| Distance from R.G. | 30 | Bnei-Brak (5 km) | Ra’anana (15 km) | ||
| Total | 1,000 |
Experiment 2a results: the mean scores for the bids’ components at each stage
| Criterion | Max. points | “The Carrier” | Average points “The Carrier” | “The Trans.” | Average points “The Trans.” | ||
| 2-stage CB | 1-stage CB | 2-stage CB | 1-stage CB | ||||
| 2nd version only: Price | 600 | $800,000 | – | 600 | $875,000 | – | 550 |
| Years of operation | 60 | 8 years | 43.33(12.31) | 49.72(8.53) | 12 years | 55.41(9.81) | 58.47(4.28) |
| Bus model | 30 | Mercedes “VIP” 2011 | 28.56(4.07) | 29.31(2.44) | Mercedes “Dolphin” 2010 | 24.39(5.12) | 25.72(4.31) |
| Recommendations | 70 | 2 good, 1 moderate | 37.64(12.15) | 44.58(11.29) | 5 good | 67.03(6.00) | 66.81(7.85) |
| No. of buses owned by the bidder | 60 | 12 buses | 36.56(10.90) | 39.28(9.24) | 20 buses | 54.25(8.00) | 56.25(9.88) |
| Average experience of the drivers | 70 | 12 years | 49.58(12.56) | 56.94(12.21) | 20 years | 67.03(6.12) | 68.19(5.09) |
| Training hours per driver | 30 | 35 hours | 28.50(3.28) | 28.42(3.29) | 20 hours | 20.42(5.53) | 21.81(3.72) |
| ISO certification | 50 | Not certified, but in the final stages of the certification process | 21.94(18.02) | 20.69(19.71) | Has certification | 49.72(1.67) | 46.66(10.95) |
| Distance from R.G. | 30 | Bnei-Brak (5 km) | 27.50(3.68) | 28.75(3.85) | Ra’anana (15 km) | 19.61(6.48) | 23.17(7.76) |
| Total (quality) | 400 | 273.61(43.00) | 297.42(36.08) | 357.86(30.04) | 367.08(22.32) | ||
| Total | 1,000 | 873.61 | 897.42 | 907.86 | 917.08 | ||
Appendix B: Experiment 2b form
The preface is exactly the same as in Appendix A.
| Criterion | Max. points | “The Carrier” | Points | “The Trans.” | Points |
| [2nd version only: Price] | 600 | $800,000 | 600 | $875,000 | 550 |
| Years of operation | 75 | 8 years | 12 years | ||
| Recommendations | 80 | 2 good, 1 moderate | 5 good | ||
| No. of buses owned by the bidder | 70 | 12 buses | 20 buses | ||
| Average experience of the drivers | 75 | 12 years | 20 years | ||
| Distance from R.G. | 100 | Bnei-Brak (5 km) | Ra’anana (15 km) | ||
| Total | 1,000 |
Experiment 2b results: the mean scores for the bids’ components at each stage
| Criterion10 | Max. points | “The Carrier” | Average points “The Carrier” | “The Trans.” | Average points “The Trans.” | ||
| 2-stage CB | 1-stage CB | 2-stage CB | 1-stage CB | ||||
| 2nd version only: Price | 600 | $800,000 | – | 600 | $875,000 | – | 550 |
| Recommendations | 80 | 2 good, 1 moderate | 42.96(8.83) | 47.65(11.86) | 5 good | 78.79(3.18) | 78.97(5.57) |
| No. of buses owned by the bidder | 70 | 12 buses | 44.41(11.63) | 47.71(10.33) | 20 buses | 66.88(8.06) | 69.13(2.70) |
| Average experience of the drivers | 75 | 12 years | 49.95(13.78) | 53.73(14.59) | 21 years | 72.59(5.77) | 72.97(5.09) |
| Distance from R.G. | 100 | Bnei-Brak (5 km) | 91.72(12.48) | 90.86(22.83) | Ra’anana (15 km) | 79.31(16.73) | 76.38(21.99) |
| Years of operation | 75 | 8 years | 56.59(9.68) | 55.57(9.68) | 12 years | 73.34(3.30) | 73.93(2.45) |
| Total (quality) | 400 | 283.57(34.41) | 295.52(47.62) | 370.93(21.99) | 371.38(25.26) | ||
| Total | 1,000 | 883.57 | 895.52 | 920.93 | 921.38 | ||
Acknowledgments
We would like to thank for helpful comments to Judith Avrahami, Constantine Boussalis, Yoav Dotan, Yuval Feldman, Einav Hart, Yaakov Kareev, Barak Medina, Ma’ayan Menashe, Simone Moran, Ariel Porat, Ilana Ritov, Boaz Shnoor, Doron Teichman, Sarit Yakuti, and Eyal Zamir, and to the participants of the session on Public Goods at the 2013 CELS at Penn University.
References
Ask, K., and P. A.Granhag. 2007. “Motivational Bias in Criminal Investigators’ Judgments of Witness Reliability,” 37Journal of Applied Social Psychology561–591.10.1111/j.1559-1816.2007.00175.xSearch in Google Scholar
Backwith, N. E., and D. R.Lehmann. 1975. “The Importance of Halo Effects in Multi-Attribute Attitude Models,” 12Journal of Marketing Research265–275.Search in Google Scholar
Bazerman, M., D. A.Moore, A. E.Tenbrunsel, K.Wade-Benzoni, and S.Blount. 1999. “Explaining How Preferences Change Across Joint Versus Separate Evaluation,” 39Journal of Economic Behavior & Organization41–58.10.1016/S0167-2681(99)00025-6Search in Google Scholar
Bertrand, M., and S.Mullainathan. 2004. “Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination,” 94The American Economic Review991–1013.10.1257/0002828042002561Search in Google Scholar
Blank, R. M. 1991. “The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence From the American Economic Review,” 81The American Economic Review1041–1067.Search in Google Scholar
Blume, J. H. 2008. “The Dilemma of the Criminal Defendant with a Prior Record – Lessons from the Wrongfully Convicted,” 5Journal of Empirical Legal Studies477–505.10.1111/j.1740-1461.2008.00131.xSearch in Google Scholar
Bornmann, L., and H.-D.Daniel. 2006. “Potential Sources of Bias in Research Fellowship Assessments: Effects of University Prestige and Field of Study,” 15Research Evaluation209–219.10.3152/147154406781775850Search in Google Scholar
Bornmann, L., R.Mutz, and H.-D.Daniel. 2007. “Gender Differences in Grant Peer Review: A Meta-Analysis,” 1Journal of Informetrics226–238.10.1016/j.joi.2007.03.001Search in Google Scholar
Brucks, M., V. A.Zeithaml, and G.Naylor. 2000. “Price and Brand Name as Indicators of Quality Dimensions for Consumer Durables,” 28Journal of the Academy of Marketing Science359–374.10.1177/0092070300283005Search in Google Scholar
Carlson, K. A., and J. E.Russo. 2001. “Biased Interpretation of Evidence by Mock Jurors,” 7Journal of Experimental Psychology91–103.10.1037/1076-898X.7.2.91Search in Google Scholar
Cramton, P. C. 1995. “Money Out of Thin Air: The Nationwide Narrowband PCS Auction,” 4Journal of Economics and Management Strategy267–343.10.1111/j.1430-9134.1995.00267.xSearch in Google Scholar
Danziger, S., J.Levav, and L.Avnaim-Pesso. 2011. “Extraneous Factors in Judicial Decisions,” 108Proceedings of the National Academy of Sciences USA6889–6892.10.1073/pnas.1018033108Search in Google Scholar
Edwards, K., and E. E.Smith. 1996. “A Disconfirmation Bias in the Evaluation of Arguments,” 71Journal of Personality and Social Psychology5–24.10.1037/0022-3514.71.1.5Search in Google Scholar
Eerland, A., and E.Rassin. 2012. “Biased Evaluation of Incriminating and Exonerating (Non)Evidence,” 18Psychology, Crime and Law351–358.Search in Google Scholar
Eisenberg, T., and V. P.Hans. 2009. “Taking a Stand on Taking the Stand: The Effect of a Prior Criminal Record on the Decision to Testify and on Trial Outcomes,” 94Cornell Law Review1353–1390.Search in Google Scholar
Federal Procurement Data System – Next Generation. https://www.fpds.gov/Reports/manage/jsp/myReportsController.jsp (last updated June 2013).Search in Google Scholar
Feldblum, C. R. 1991. “Medical Examinations and Inquiries Under the Americans with Disabilities Act: A View From the Inside,” 64Temple Law Review521–549.Search in Google Scholar
Felton, J., J. B.Mitchell, and M.Stinson. 2004. “Web-Based Student Evaluations of Professors: The Relations Between Perceived Quality, Easiness, and Sexiness,” 29Assessment and Evaluation in Higher Education91–108.10.1080/0260293032000158180Search in Google Scholar
Goldin, C., and C.Rouse. 2000. “Orchestrating Impartiality: The Impact of ‘Blind’ Auditions on Female Musicians,” 90The American Economic Review715–741.10.1257/aer.90.4.715Search in Google Scholar
Guthrie, C., J. J.Rachlinski, and A. J.Wistrich. 2009. “The ‘Hidden Judiciary’: An Empirical Examination of Executive Branch Justice,” 58Duke Law Journal1477–1530.10.2139/ssrn.1374346Search in Google Scholar
Hans, V. P., and A. N.Doob. 1976. “Section 12 of the Canada Evidence Act and the Deliberations of Simulated Juries,” 18Criminal Law Quarterly235–253.Search in Google Scholar
Hsee, C. K. 1996. “The Evaluability Hypothesis: An Explanation for Preference Reversals between Joint and Separate Evaluations of Alternatives,” 67Organizational Behavior and Human Decision Processes247–257.10.1006/obhd.1996.0077Search in Google Scholar
Hsee, C. K., G. F.Loewenstein, S.Blount, and M. H.Bazerman. 1999. “Preference Reversals Between Joint and Separate Evaluations of Options: A Review and Theoretical Analysis,” 125Psychological Bulletin576–590.10.1037/0033-2909.125.5.576Search in Google Scholar
Hsee, C. K., and J.Zhang. 2010. “General Evaluability Theory,” 5Perspectives of Psychological Science343–355.10.1177/1745691610374586Search in Google Scholar
Jayasinghe, U. W., W. H.Marsh, and N. W.Bond. 2006. “A New Reader Trial Approach to Peer Review in Funding Research Grants: An Australian Experiment,” 69Scientometrics591–606.10.1007/s11192-006-0171-4Search in Google Scholar
Jolls, C., and C. R.Sunstein. 2006. “Debiasing Through Law,” 35Journal of Legal Studies199–241.10.1086/500096Search in Google Scholar
Kahneman, D. 2011. Thinking Fast and Slow. New York: Farrar, Straus and Giroux.Search in Google Scholar
Klemperer, P. 2002. “What Really Matters in Auction Design,” 16Journal of Economic Perspectives169–189.10.1257/0895330027166Search in Google Scholar
Kunda, Z. 1987. “Motivation and Inference: Self-Serving Generation and Evaluation of Evidence,” 53J. Pers. Soc. Psychol. 636–647.Search in Google Scholar
Kunda, Z. 1990. “The Case for Motivated Reasoning,” 108Psychological Bulletin480–498.10.1037/0033-2909.108.3.480Search in Google Scholar
Kwerel, E. R., and G. L.Rosston. 2000. “An Insider’s View of FCC Spectrum Auctions,” 17Journal of Regulatory Economics253–289.Search in Google Scholar
Lichtenstein, S., and P.Slovic. 1971. “Reversal of Preferences Between Bids and Choices in Gambling Decisions,” 89Journal of Experimental Psychology46–55.10.1037/h0031207Search in Google Scholar
Lord, C. G., L.Ross, and M. R.Lepper. 1979. “Biased Assimilation and Attitude Polarization: The Effects of Prior Theories on Subsequently Considered Evidence,” 37Journal of Personality and Social Psychology2098–2019.10.1037/0022-3514.37.11.2098Search in Google Scholar
Marsh, H. W., U. W.Jayasinghe, and N. W.Bond. 2008. “Improving the Peer-Review Process for Grant Applications: Reliability, Validity, Bias, and Generalizability,” 63American Psychologist160–168.10.1037/0003-066X.63.3.160Search in Google Scholar
McAfee, P. R., and J.McMillan. 1996. “Analyzing the Airwaves Auction,” 10Journal of Economic Perspectives159–175.10.1257/jep.10.1.159Search in Google Scholar
Milgrom, P. 2004. Putting Auction Theory to Work. Cambridge: Cambridge University Press.10.1017/CBO9780511813825Search in Google Scholar
Newman, J. M. 1978. “Discrimination in Recruitment: An Empirical Analysis,” 32Industrial and Labor Relations Review15–23.Search in Google Scholar
Nickerson, R. S. 1998. “Confirmation Bias: A Ubiquitous Phenomenon in Many Guises,” 2Review of General Psychology175–220.10.1037/1089-2680.2.2.175Search in Google Scholar
Nisbett, R. E., and T. D.Wilson. 1977. “The Halo Effect: Evidence for Unconscious Alteration of Judgments,” 35Journal of Personality and Social Psychology250–256.10.1037/0022-3514.35.4.250Search in Google Scholar
Nowlis, S. M., B. E.Kahn, and R.Dhar. 2002. “Coping with Ambivalence: The Effect of Removing a Neutral Option on Consumer Attitude and Preference Judgments,” 29Journal of Consumer Research319–334.10.1086/344431Search in Google Scholar
Nowlis, S. M., and I.Simonson. 1997. “Attribute-Task Compatibility as a Determinant of Consumer Preference Reversals,” 34Journal of Marketing Research205–218.Search in Google Scholar
O’Brian, B. 2009. “Prime Suspect: An Examination of Factors That Aggravate and Counteract Confirmation Bias in Criminal Investigations,” 15Psychology, Public Policy, and Law315–334.Search in Google Scholar
Pronin, E., and M. B.Kugler. 2007. “Valuing Thoughts, Ignoring Behavior: The Introspection Illusion as a Source of the Bias Blind Spot,” 43Journal of Experimental Social Psychology565–578.10.1016/j.jesp.2006.05.011Search in Google Scholar
Rachlinski, J. J. 2012. “Judicial Psychology,” 2Rechtstreeks15–34.Search in Google Scholar
Rachlinski, J. J., C.Guthrie, and A. J.Wistrich. 2006. “Inside the Bankruptcy Judge’s Mind,” 86Boston University Law Review1227–1265.Search in Google Scholar
Rao, A. R., and K. B.Monroe. 1989. “The Effect of Price, Brand Name, and Store Name on Buyers Perceptions of Product Quality: An Integrative Review,” 26Journal of Marketing Research351–357.Search in Google Scholar
Rassin, E., A.Eerland, and I.Kuijpers. 2010. “Let’s Find the Evidence: An Analogue Study of Confirmation Bias in Criminal Investigations,” 7Journal of Investigative Psychology and Offender Profiling231–246.10.1002/jip.126Search in Google Scholar
Ross, L., M. R.Lepper, and M.Hubbard. 1975. “Perseverance in Self Perception and Social Perception: Biased Attributional Processes in the Debriefing Paradigm,” 32Journal of Personality and Social Psychology880–892.10.1037/0022-3514.32.5.880Search in Google Scholar
Shafir, E. 1993. “Choosing Versus Rejecting: Why Some Options Are Both Better and Worse Than Others,” 21Memory & Cognition546–556.10.3758/BF03197186Search in Google Scholar
Shanteau, J. 1988. “Psychological Characteristics and Strategies of Expert Decision Makers,” 68Acta Psychologica203–215.10.1016/0001-6918(88)90056-XSearch in Google Scholar
Shiv, B., Z.Carmon, and D.Ariely. 2005. “Placebo Effects of Marketing Actions: Consumers May Get What They Pay for,” 42Journal of Marketing Research383–393.Search in Google Scholar
Simonson, I. 2008. “Will I Like a ‘Medium’ Pillow? Another Look at Constructed and Inherent Preferences,” 18Journal of Consumer Psychology155–169.10.1016/j.jcps.2008.04.002Search in Google Scholar
Smith, N. C., D.Read, and S.López-Rodríguez. 2010. “Consumer Perceptions of Corporate Social Responsibility: The CSR Halo Effect.” INSEAD Working Paper No. 2010/16/INSEAD Social Innovation Centre, http://ssrn.com/abstract=1577000 or http://dx.doi.org/10.2139/ssrn.1577000.10.2139/ssrn.1577000Search in Google Scholar
Taylor, C. R., and H.Yildirim. 2011. “Subjective Performance and the Value of Blind Evaluation,” 78The Review of Economic Studies762–794.10.1093/restud/rdq005Search in Google Scholar
Thorndike, E. L. 1920. “A Constant Error in Psychological Ratings,” IVJournal of Applied Psychology25–29.10.1037/h0071663Search in Google Scholar
Tversky, A., and D.Kahneman. 1983. “Extensional Versus Intuitive Reasoning: The Conjunction Fallacy in Probability Judgment,” 90Psychological Review293–315.10.1037/0033-295X.90.4.293Search in Google Scholar
Tversky, A., and D.Kahneman. 1986. “Rational Choice and the Framing of Decisions,” 59The Journal of Business251–278.10.1086/296365Search in Google Scholar
Tversky, A., S.Sattath, and P.Slovic. 1988. “Contingent Weighting in Judgment and Choice,” 95Psycological Review371–384.10.1037/0033-295X.95.3.371Search in Google Scholar
Tversky, A., P.Slovic, and D.Kahneman. 1990. “The Causes for Preference Reversal,” 80The American Economic Review204–217.Search in Google Scholar
Tversky, A., and R. H.Thaler. 1990. “Anomalies: Preference Reversals,” 4The Journal of Economic Perspectives201–211.10.1257/jep.4.2.201Search in Google Scholar
Wistrich, A. J., C.Gutherie, and J. J.Rachlinski. 2005. “Can Judges Ignore Inadmissible Information? The Difficulty of Deliberately Disregarding,” 153University of Pennsylvania Law Review1251–1345.Search in Google Scholar
- 1
To illustrate, according to the Federal Procurement Data System, in 2012 alone, the U.S. federal government procured approximately $536 billion worth of goods, services, and infrastructure. Of this, $334.5 billion was expended through competitive procedures (Federal Procurement Data System – Next Generation (last updated December 2013), https://www.fpds.gov/Reports/manage/jsp/myReportsController.jsp). Millions of additional competitive procedures, worth billions of dollars, are conducted each year by administrative agencies and state and local governments.
- 2
For the regulation of the U.S. federal government acquisition system, see the Federal Acquisition Regulation (FAR) (48 C.F.R. § 1 (2003)). For the European Community public procurement regulations, see Directive 2004/18/EC of the European Parliament and of the Council of 31 March 2004 on the coordination of procedures for the award of public works contracts, public supply contracts and public service contracts, 2004 O.J. (L 134) 114-240. For the regulation of public procurement by the World Trade Organization (WTO), see the Agreement on Government Procurement (April 15, 1994), available athttp://www.wto.org/english/docs_e/legal_e/gpr-94_01_e.htm. Similar regulations exist in the State of Israel, where the experiments used for the current research were conducted. See the Public Tenders Act, 1992 (Israel).
- 3
TRW Environmental Safety Systems, Inc. v. United States, 18 Cl. Ct. 33, 43 (1989) Glenn Def. Marine Asia, PTE Ltd. v. United States, 720 F.3d 901, 907–908 (Fed. Cir. 2013).
- 4
Banknote Corp. of Am. v. United States, 56 Fed. Cl. 377, 386 (2003) (“It is hornbook law that agencies must evaluate proposals and make awards based on the criteria stated in the solicitation.”); 360Training.com, Inc. v. United States, 106 Fed. Cl. 177, 184 (2012). (“… if the agency’s evaluation of proposals significantly differs from the evaluation process disclosed in the solicitation, the agency’s decision can be found to lack a rational basis.”)
- 5
A distinction must be made between “two-stage competitive bidding” and “two-step sealed bidding” (48 C.F.R. § 14.5 (2003)). In the latter, the submitted bids do not include the proposed price. Only once the bids have been reviewed, are the bidders whose bids were found to be suitable in the first stage asked to submit price proposals. For more details, see Section 5 below.
- 6
Latecoere Int’l, Inc. v. United States Dep’t of the Navy, 19 F.3d 1342, 1359 (11th Cir. Fla. 1994). See also Dubinsky v. United States, 43 Fed. Cl. 243 (Fed. Cl. 1999).
- 7
360 Training.com, Inc. v. United States, 106 Fed. Cl. 177, 184 (2012). See also L-3 Communs. Eotech, Inc. v. United States, 83 Fed. Cl. 643, 654 (2008).
- 8
- 9
The order of presentation of the evaluation criteria was randomly altered between the two rounds of the experiment, in order to prevent participants from remembering their responses in the first round of the experiment.
- 10
The order of presentation of the evaluation criteria was randomly altered between the two rounds of the experiment, in order to prevent participants from remembering the answers they gave in the first round.
©2014 by De Gruyter
Articles in the same Issue
- Frontmatter
- Articles
- Litigation with Default Judgments
- Addressing Federal Conflicts: An Empirical Analysis of the Brazilian Supreme Court, 1988–2010
- Cognitive Biases in Government Procurement – An Experimental Study
- Endogenous Detection of Collaborative Crime: The Case of Corruption
- The Power of the Racketeer: An Empirical Approach
Articles in the same Issue
- Frontmatter
- Articles
- Litigation with Default Judgments
- Addressing Federal Conflicts: An Empirical Analysis of the Brazilian Supreme Court, 1988–2010
- Cognitive Biases in Government Procurement – An Experimental Study
- Endogenous Detection of Collaborative Crime: The Case of Corruption
- The Power of the Racketeer: An Empirical Approach
