Using Journal Impact Factor to Assess Scholarly Records: Overcorrecting for the Potter Stewart Approach to Promotion and Tenure
-
Elizabeth A. Oldmixon
and
J. Tobin Grant
Abstract
Promotion and tenure decisions frequently require an assessment of the quality of a candidate’s research record. Without carefully specifying what constitutes a tenurable and promotable record, departments frequently adopt the Potter Stewart approach – they know it when they see it. The benefit of such a system is that it allows for multiple paths to tenure and promotion and encourages holistic review, but the drawback is that it allows for the promotion and tenure process to be more easily manipulated by favoritism and bias. Incorporating transparent metrics such as journal impact factor (JIF) would seem like a good way to standardize the process. We argue, however, that when JIF becomes determinative, conceptual disadvantages and systematic biases are introduced into the process. JIF indicates the visibility or utility of a journal; it does not and cannot tell us about individual articles published in that journal. Moreover, it creates inequitable paths to tenure on the basis of gender and subfield, given gendered patterns of publications and the variation in journal economies by subfield.
Acknowledgements
The authors thank Victor Asal, Bethany Blackstone, and Amy Erica Smith for their helpful comments and suggestions for revision.
References
Borchardt, Rachel, and Matthew R. Hartings. 2018. “The Academic Papers Researchers Regard as Significant Are Not Those That Are Highly Cited.” The LSE Impact Blog. Accessed May 14, 2018. http://blogs.lse.ac.uk/impactofsocialsciences/2018/05/14/the-academic-papers-researchers-regard-as-significant-are-not-those-that-are-highly-cited/.Search in Google Scholar
Carpenter, Christopher R., David C. Cone, and Cathy C. Saril. 2014. “Using Publication Metrics to Highlight Academic Productivity and Research Impact.” Academic Emergency Medicine 21 (10): 1160–1172.10.1111/acem.12482Search in Google Scholar
Chandra, Kanchan, Jennifer Gandhi, Gary King, Arthur Lupia, and Edward Mansfield. 2006. Report of APSA Working Group on Collaboration. Washington, DC: American Political Science Association.Search in Google Scholar
Collins, Paul M. 2018. “Reputational Rankings of Peer-Reviewed Law Journals: A Survey Approach.” PS: Political Science & Politics 51 (2): 377–386.10.1017/S1049096517002529Search in Google Scholar
Delgado, Richard. 1984. “The Imperial Scholar: Reflections on a Review of Civil Rights Literature.” University of Pennsylvania Law Review 132: 561–578.10.4324/9781315633732-20Search in Google Scholar
Djupe, Paul A., Amy Erica Smith, and Anand Edward Sokhey. 2018. “Explaining Gender in the Journals: How Submission Practices Affect Publication Patterns in Political Science.” PS: Political Science & Politics. doi:10.1017/S104909651800104X.Search in Google Scholar
DORA. No date. San Francisco Declaration on Research Assessment. Accessed May 14, 2018. https://sfdora.org/read/.Search in Google Scholar
Elliott, David B. 2013. “The Impact Factor: A Useful Indicator of Journal Quality or Fatally Flawed?” Opthalmic & Physiological Optics 34: 4–7.10.1111/opo.12107Search in Google Scholar
Fisher, Bonnie S., Craig T. Cobane, Thomas M. Vander Van, and Francis T. Cullen. 1998. “How Many Authors Does It Take to Publish an Article? Trends and Patterns in Political Science.” PS: Political Science & Politics 31 (4): 847–856.10.2307/420730Search in Google Scholar
Garand, James C., and Michael W. Giles. 2003. “Journals in the Discipline: A Report on a New Survey of American Scientists.” PS: Political Science & Politics 36: 293–308.10.1017/S1049096503002245Search in Google Scholar
Garfield, Eugene. 1955. “Citation Indexes to Science: A New Dimension in Documentation Through Association of Ideas.” Science 122: 108–111.10.1126/science.122.3159.108Search in Google Scholar
Garfield, Eugene. 2001. “Impact Factors, and Why They Won’t Go Away.” Nature 411 (6837): 522.10.1038/35079156Search in Google Scholar
Garfield, Eugene. 2006. “The History and Meaning of the Journal Impact Factor.” Journal of the American Medical Association 295 (1): 90–93.10.1001/jama.295.1.90Search in Google Scholar
Hicks, Diana, Paul Wouters, Ludo Waltman, Sarah de Rijcke, and Ismael Rafols. 2015. “The Leiden Manifesto for Research Metrics.” Nature 520: 429–531.10.1038/520429aSearch in Google Scholar
Hoeffel, C. 1998. “Journal Impact Factors.” Allergy 53: 1225.10.1111/j.1398-9995.1998.tb03848.xSearch in Google Scholar
Jensenius, Francesca R., Mala Htun, David J. Samuels, David A. Singer, Adria Lawrence, and Michael Chwe. 2018. “The Benefits and Pitfalls of Google Scholar.” PS: Political Science & Politics. Advanced online publication. doi:10.1017/S104909651800094X.Search in Google Scholar
König, Thomas, and Guido Ropers. 2018. “Gender and Editorial Outcomes at the American Political Science Review.” PS: Political Science & Politics. Advanced online publication. doi:10.1017/S1049096518000604.Search in Google Scholar
Mitchell, Sara McLaughlin, Samantha Langley, and Holly Brus. 2013. “Gendered Citation Patterns in International Relations Journals.” International Studies Perspectives 14: 485–492.10.1111/insp.12026Search in Google Scholar
Moen, H.F., and Th. N. van Leeuwen. 1996. “Correspondence: Impact Factors Can Mislead.” Nature 381: 186.10.1038/381186a0Search in Google Scholar
Moher, David, Florian Naudet, Ioana A. Cristea, Frank Miedema, John P. Ioannidis, and Steven N. Goodman. 2018. “Assessing Scientists for Hiring, Promotion, and Tenure.” PLoS Biology 16 (3): 1–20.10.1371/journal.pbio.2004089Search in Google Scholar
Nedal, Dami K., and Daniel H. Nexon. 2018. “Gender in the International Studies Quarterly Review Process.” PS: Political Science & Politics. Advanced online publication. doi:10.1017/S1049096518000628.Search in Google Scholar
Pudovkin, Alexander I. 2018. “Comments on the Use of the Journal Impact Factor for Assessing the Research Contributions of Individual Authors.” Frontiers in Research Metrics and Analytics. doi:10.3389/frma.2018.00002.Search in Google Scholar
Saha, Somnath, Sanjay Saint, and Dimitri A. Christakis. 2003. “Impact Factor: A Valid Measure of Journal Quality?” Journal of the Medical Library Association 91 (1):42–46.Search in Google Scholar
Seglen, Per O. 1997a. “Citations and Journal Impact Factors: Questionable Indicators of Research Quality.” Allergy 52: 1050–1056.10.1111/j.1398-9995.1997.tb00175.xSearch in Google Scholar
Seglen, Per O. 1997b. “Why the Impact Factor of Journals Should Not Be Used for Evaluating Research.” British Medical Journal 314: 498–502.10.1136/bmj.314.7079.497Search in Google Scholar
Smith, Amy Erica (@amyericasmith). 2018. “The More Consequential the Measurement Task for the Fate of Something We Should Care About (Mauritius; a Junior Professor), the Greater the Normative Burden to Get the Measurement Right. HRW and FH Scores Aren’t Usually Used in Isolation to Withdraw International Aid, Whereas 1/2.” Twitter, May 7, 6:13am, https://twitter.com/amyericasmith/status/993448616667860993.Search in Google Scholar
Tachibana, Chris. 2017. “New Tools for Measuring Academic Performance.” Science. Accessed: June 4, 2018. http://www.sciencemag.org/features/2017/02/new-tools-measuring-academic-performance.10.1126/science.opms.r1700173Search in Google Scholar
Teele, Dawn Langan, and Kathleen Thelen. 2017. “Gender in the Journals: Publication Patterns in Political Science.” PS: Political Science and Politics 50: 433–447.10.1017/S1049096516002985Search in Google Scholar
The PLoS Medicine Editors. 2006 “The Impact Factor Game.” PLoS Medicine 3 (6): e291.10.1371/journal.pmed.0030291Search in Google Scholar
Van Noorden, Richard. 2010. “A Profusion of Measures.” Nature 465: 864–866.10.1038/465864aSearch in Google Scholar
Waltman, Ludo, and Vincent A. Traag. 2017. “Use of the Journal Impact Factor for Assessing Individual Articles Need not be Wrong.” Unpublished manuscript: https://arxiv.org/abs/1703.02334.Search in Google Scholar
Wilsdon, James, Liz Allen, Eleonora Belfiore, Philip Campbell, Stephen Curry, Steven Hill, Richard Jones, Roger Kain, Simon Kerridge, Mike Thelwall, Jane Tinkler, Ian Viney, Paul Wouters, Jude Hill, and Ben Johnson. 2015. The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. doi:10.13140/RG.2.1.4929.1363.10.13140/RG.2.1.4929.1363Search in Google Scholar
Zhang, L., R. Rousseau, and G. Sivertsen. 2017. “Science Deserves to Be Judged by Its Contents, Not by Its Wrapping: Revisiting Seglen’s Work on Journal Impact and Research Evaluation.” PLoS One 12 (3): e0174205.10.1371/journal.pone.0174205Search in Google Scholar
©2019 Walter de Gruyter GmbH, Berlin/Boston
Articles in the same Issue
- Frontmatter
- Introduction
- Articles
- Moral Foundations, System Justification, and Support for Trump in the 2016 Presidential Election
- Explaining and Predicting Midterm Congressional Election Outcomes: Factoring in Opposition Party Strategy
- Dumping Trump and Electoral Bumps: The Causes and Consequences of Republican Officeholders’ Endorsement Decisions
- Using Journal Impact Factor to Assess Scholarly Records: Overcorrecting for the Potter Stewart Approach to Promotion and Tenure
- Deconstructing Popular Mythologies about Millennials and Party Identification
- Ready for Hillary?: Explicit and Implicit Sexism in the 2016 Presidential Election
- “One Difficulty…of a Serious Nature”: The Overlooked Racial Dynamics of the Electoral College
- The Myth of the Bipartisan National Popular Vote Plan
- Book reviews
- Review of Choosing the Leader: Leadership Elections in the U.S. House of Representatives
- Review of Identity Crisis: The 2016 Presidential Campaign and the Battle for the Meaning of America
Articles in the same Issue
- Frontmatter
- Introduction
- Articles
- Moral Foundations, System Justification, and Support for Trump in the 2016 Presidential Election
- Explaining and Predicting Midterm Congressional Election Outcomes: Factoring in Opposition Party Strategy
- Dumping Trump and Electoral Bumps: The Causes and Consequences of Republican Officeholders’ Endorsement Decisions
- Using Journal Impact Factor to Assess Scholarly Records: Overcorrecting for the Potter Stewart Approach to Promotion and Tenure
- Deconstructing Popular Mythologies about Millennials and Party Identification
- Ready for Hillary?: Explicit and Implicit Sexism in the 2016 Presidential Election
- “One Difficulty…of a Serious Nature”: The Overlooked Racial Dynamics of the Electoral College
- The Myth of the Bipartisan National Popular Vote Plan
- Book reviews
- Review of Choosing the Leader: Leadership Elections in the U.S. House of Representatives
- Review of Identity Crisis: The 2016 Presidential Campaign and the Battle for the Meaning of America
