Band 10, Heft 1

ABSTRACT Purpose Evaluating the quality of academic journal articles is a time consuming but critical task for national research evaluation exercises, appointments and promotion. It is therefore important to investigate whether Large Language Models (LLMs) can play a role in this process. Design/methodology/approach This article assesses which ChatGPT inputs (full text without tables, figures, and references; title and abstract; title only) produce better quality score estimates, and the extent to which scores are affected by ChatGPT models and system prompts. Findings The optimal input is the article title and abstract, with average ChatGPT scores based on these (30 iterations on a dataset of 51 papers) correlating at 0.67 with human scores, the highest ever reported. ChatGPT 4o is slightly better than 3.5-turbo (0.66), and 4o-mini (0.66). Research limitations The data is a convenience sample of the work of a single author, it only includes one field, and the scores are self-evaluations. Practical implications The results suggest that article full texts might confuse LLM research quality evaluations, even though complex system instructions for the task are more effective than simple ones. Thus, whilst abstracts contain insufficient information for a thorough assessment of rigour, they may contain strong pointers about originality and significance. Finally, linear regression can be used to convert the model scores into the human scale scores, which is 31% more accurate than guessing. Originality/value This is the first systematic comparison of the impact of different prompts, parameters and inputs for ChatGPT research quality evaluations.

ABSTRACT Purpose Citations can be used in evaluative bibliometrics to measure the impact of papers. However, citation analysis can be extended by a multi-dimensional perspective on citation impact which is intended to receive more specific information about the kind of received impact. Design/methodology/approach Bornmann, Wray, and Haunschild (2019) introduced citation concept analysis (CCA) for capturing the importance and usefulness certain concepts have in subsequent research. The method is based on the analysis of citances – the contexts of citations in citing papers. This study applies the method by investigating the impact of various concepts introduced in the oeuvre of the world-leading French sociologist Pierre Bourdieu. Findings We found that the most cited concepts are ‘social capital’ (with about 34% of the citances in the citing papers), ‘cultural capital’, and ‘habitus’ (both with about 24%). On the other hand, the concepts ‘doxa’ and ‘reflexivity’ score only about 1% each. Research limitations The formulation of search terms for identifying the concepts in the data and the citation context coverage are the most important limitations of the study. Practical implications The results of this explorative study reflect the historical development of Bourdieu’s thought and its interface with different fields of study. Originality/value The study demonstrates the high explanatory power of the CCA method.

ABSTRACT Purpose We aimed to measure the variation in researchers’ knowledge and attitudes towards bibliometric indicators. The focus is on mapping the heterogeneity of this metric-wiseness within and between disciplines. Design/methodology/approach An exploratory survey is administered to researchers at the Sapienza University of Rome, one of Europe’s oldest and largest generalist universities. To measure metric-wiseness, we use attitude statements that are evaluated by a 5-point Likert scale. Moreover, we analyze documents of recent initiatives on assessment reform to shed light on how researchers’ heterogeneous attitudes regarding and knowledge of bibliometric indicators are taken into account. Findings We found great heterogeneity in researchers’ metric-wiseness across scientific disciplines. In addition, within each discipline, we observed both supporters and critics of bibliometric indicators. From the document analysis, we found no reference to individual heterogeneity concerning researchers’ metric wiseness. Research limitations We used a self-selected sample of researchers from one Italian university as an exploratory case. Further research is needed to check the generalizability of our findings. Practical implications To gain sufficient support for research evaluation practices, it is key to consider researchers’ diverse attitudes towards indicators. Originality/value We contribute to the current debate on reforming research assessment by providing a novel empirical measurement of researchers’ knowledge and attitudes towards bibliometric indicators and discussing the importance of the obtained results for improving current research evaluation systems.

ABSTRACT Purpose This study investigates whether publication-centric incentive systems, introduced through the National Scientific Accreditation (ASN: Abilitazione Scientifica Nazionale) for professorships in Italy in 2012, contribute to adopting “salami publishing” strategies among Italian academics. Design/methodology/approach A longitudinal bibliometric analysis was conducted on the publication records of over 25,000 Italian science professors to examine changes in publication output and the originality of their work following the implementation of the ASN. Findings The analysis revealed a significant increase in publication output after the ASN’s introduction, along with a concurrent decline in the originality of publications. However, no evidence was found linking these trends to increased salami slicing practices among the observed researchers. Research limitations Given the size of our observation field, we propose an innovative indirect approach based on the degree of originality of publications’ bibliographies. We know that bibliographic coupling cannot capture salami publications per se, but only topically-related records. On the other hand, controlling for the author’s specialization level in the period, we believe that a higher level of bibliographic coupling in his scientific output can signal a change in his strategy of disseminating the results of his research. The relatively low R-squared values in our models (0.3-0.4) reflect the complexity of the phenomenon under investigation, revealing the presence of unmeasured factors influencing the outcomes, and future research should explore additional variables or alternative models that might account for a greater proportion of the variability. Despite this limitation, the significant predictors identified in our analysis provide valuable insights into the key factors driving the observed outcomes. Practical implications The results of the study support those who argue that quantitative research assessment frameworks have had very positive effects and should not be dismissed, contrary to the claims of those evoking the occurrence of side effects that do not appear in the empirical analyses. Originality/value This study provides empirical evidence on the impact of the ASN on publication behaviors in a huge micro-level dataset, contributing to the broader discourse on the effects of quantitative research assessments on academic publishing practices.

ABSTRACT Purpose In this paper, we use author clustering based on journal coupling (i.e., shared academic journals) to determine researchers who have the same scientific interests and similar conceptual frameworks. The basic assumption is that authors who publish in the same academic journals are more likely to share similar conceptual frameworks and interests than those who never publish in the same venues. Therefore, they are more likely to be part of the same invisible college (i.e., authors in this subgroup contribute materially to research on the same topic and often publish their work in similar publication venues). Design/methodology/approach Test in a controlled exercise the grouping of authors based on journal coupling to determine invisible colleges in a research field using a case study of 302 authors who had published in the Information Science and Library Science (IS&LS) category of the Web of Science Core Collection. For each author, we retrieved all the scientific journals in which this author had published his/her articles. We then used the cosine measure to calculate the similarity between authors (both first and second order). Findings In this paper, using journal coupling of IS&LS authors, we found four main invisible colleges: “Information Systems”, “Business and Information Management”, “Quantitative Information Science” and “Library Science.” The main journals that determine the existence of these invisible colleges were Inform Syst Res, Inform Syst J, J Bus Res, J Knowl Manage, J Informetr, Pro Int Conf Sci Inf, Int J Geogr Inf Sci, J Am Med Inform Assn, and Learn Publ. However, the main journals that demonstrate that IS&LS determine a field were J Am Soc Inf Sci Tec/J Assoc Inf Sci Tech, Scientometrics, Inform Process Manag, and J Inf Sci. Research limitations The results shown in this article are from a controlled exercise. The analysis performed using journal coupling excludes books, book chapters, and conference papers. In this article, only academic journals were used for the representation of research results. Practical implications Our results may be of interest to IS&LS scholars. This is because these results provide a new lens for grouping authors, making use of the authors’ journal publication profile and journal coupling. Furthermore, extending our approach to the study of the structure of other disciplines would possibly be of interest to historians of science as well as scientometricians. Originality/value This is a novel approach based on journal coupling to determine authors who are most likely to be part of the same invisible college.

ABSTRACT Purpose This study investigates the physics of annual fractional citation growth and its impact on journal bibliographic metrics, focusing on the interplay between journal publication growth and citation dynamics. Design/methodology/approach We analyze bibliometric data from three prominent fluids journals— Physics of Fluids, Journal of Fluid Mechanics , and Physical Review Fluids —over the period 1999–2023. The analysis examines the relations among annual fractional journal publication growth, citation growth, and bibliographic metric suppressions. Findings Our findings reveal that the suppression of impact factor growth is significantly influenced by annual fractional journal publication growth rather than citation growth. All three journals exhibit similar responses to publication growth with minimal scatter, following a consistent functional relation. We also identify narrow, nearly Gaussian distributions for annual fractional journal publication growth. Furthermore, we introduce a new growth-independent dimensionless bibliometric metric, journal urgency , the ratio of annual fractional citation growth to the 4-year running average immediacy index. This metric captures effectively the dependency of citation growth on urgency and reveals consistent distributions across the journals analyzed. Research limitations The study is limited to three major fluids journals and to the availability of bibliometric data from 1999 to 2023. Future work could extend the analysis to other disciplines and journals. Practical implications Understanding the relation between publication growth and bibliometric suppressions can inform editorial and strategic decisions in journal management. The proposed journal urgency metric offers a novel tool for assessing and comparing journal performance independent of growth rates. Originality/value This study introduces a new bibliometric metric—journal urgency—that provides fresh insights into citation dynamics and bibliographic metric behavior. It highlights the critical role of publication growth in shaping journal impact factors and CiteScores, offering a unified framework applicable across multiple journals.

Purpose For a set of 1,561 Open Access (OA) and non-OA journals in business and economics, this study evaluates the relationships between four citation metrics—five-year Impact Factor (5IF), CiteScore, Article Influence (AI) score, and SCImago Journal Rank (SJR)—and the journal ratings assigned by expert reviewers. We expect that the OA journals will have especially high citation impact relative to their perceived quality (reputation). Design/methodology/approach Regression is used to estimate the ratings assigned by expert reviewers for the 2021 CABS (Chartered Association of Business Schools) journal assessment exercise. The independent variables are the four citation metrics, evaluated separately, and a dummy variable representing the OA/non-OA status of each journal. Findings Regardless of the citation metric used, OA journals in business and economics have especially high citation impact relative to their perceived quality (reputation). That is, they have especially low perceived quality (reputation) relative to their citation impact. Research limitations These results are specific to the CABS journal ratings and the four citation metrics. However, there is strong evidence that CABS is closely related to several other expert ratings, and that 5IF, CiteScore, AI, and SJR are representative of the other citation metrics that might have been chosen. Practical implications There are at least two possible explanations for these results: (1) expert evaluators are biased against OA journals, and (2) OA journals have especially high citation impact due to their increased accessibility. Although this study does not allow us to determine which of these explanations are supported, the results suggest that authors should consider publishing in OA journals whenever overall readership and citation impact are more important than journal reputation within a particular field. Moreover, the OA coefficients provide a useful indicator of the extent to which anti-OA bias (or the citation advantage of OA journals) is diminishing over time. Originality/value This is apparently the first study to investigate the impact of OA status on the relationships between expert journal ratings and journal citation metrics.

ABSTRACT Purpose In this paper, we develop a heterogeneous graph network using citation relations between papers and their basic information centered around the “Paper mills” papers under withdrawal observation, and we train graph neural network models and classifiers on these heterogeneous graphs to classify paper nodes. Design/methodology/approach Our proposed citation network-based “Paper mills” detection model (PDCN model for short) integrates textual features extracted from the paper titles using the BERT model with structural features obtained from analyzing the heterogeneous graph through the heterogeneous graph attention network model. Subsequently, these features are classified using LGBM classifiers to identify “Paper mills” papers. Findings On our custom dataset, the PDCN model achieves an accuracy of 81.85% and an F1-score of 80.49% in the “Paper mills” detection task, representing a significant improvement in performance compared to several baseline models. Research limitations We considered only the title of the article as a text feature and did not obtain features for the entire article. Practical implications The PDCN model we developed can effectively identify “Paper mills” papers and is suitable for the automated detection of “Paper mills” during the review process. Originality/value We incorporated both text and citation detection into the “Paper mills” identification process. Additionally, the PDCN model offers a basis for judgment and scientific guidance in recognizing “Paper mills” papers.

ABSTRACT Purpose This study investigates key factors contributing to research impact and their interactions with the Research Impact Quintuple Helix Model by Arsalan et al. (2024). Design/methodology/approach Using data from a global survey of 630 scientists across diverse disciplines, genders, regions, and experience levels, Structural Equation Modelling (SEM) was employed to assess the influence of 29 factors related to researcher characteristics, research attributes, publication strategies, institutional support, and national roles. Findings The study validated the Quintuple Helix Model, uncovering complex interdependencies. Institutional support significantly affects research impact by covering leadership, resources, recognition, and funding. Researcher attributes, including academic experience and domain knowledge, also play a crucial role. National socioeconomic conditions indirectly influence research impact by supporting institutions, underscoring the importance of conducive national frameworks. Research limitations While the study offers valuable insights, it has limitations. Although statistically sufficient, the response rate was below 10%, suggesting that the findings may not fully represent the entire global research community. The reliance on self-reported data may also introduce bias, as perceptions of impact can be subjective. Practical implications The findings have a significant impact on researchers aiming to enhance their work’s societal, economic, and cultural significance, institutions seeking supportive environments, and policymakers interested in creating favourable national conditions for impactful research. The study advocates for a strategic alignment among national policies, institutional practices, and individual researcher efforts to maximise research impact and effectively address global challenges. Originality/value By empirically validating the Research Impact Quintuple Helix Model, this study offers a holistic framework for understanding the synergy of factors that drive impactful research.