Volume 4, Issue 3

Purpose This paper describes an alternative way of assessing journals considering a broader perspective of its impact. The Area-based connectedness (ABC) to society of journals applied here contributes to the assessment of the dissemination task of journals but with more data it may also contribute to the assessment of other missions. Design/methodology/approach The ABC approach assesses the performance of research actors, in this case journals, considering the characteristics of the research areas in which they are active. Each paper in a journal inherits the characteristics of its area. These areas are defined by a publication-based classification. The characteristics of areas relate to 5 dimensions of connectedness to society (news, policy, industrial R&D, technology and local interest) and are calculated by bibliometric indicators and social media metrics. Findings In the paper, I illustrate the approach by showing the results for a few journals. They illustrate the diverse profiles that journals may have. We are able to provide a profile for each journal in the Web of Science database. The profiles we present show an appropriate view on the journals’ societal connectedness. Research limitations The classification I apply to perform the analyses is a CWTS in house classification based on Web of Science data. As such the application depends on the (updates of) that system. The classification is available at www.leidenranking.com Practical implications The dimensions of connectedness discussed in this paper relate to the dissemination task of journals but further development of this method may provide more options to monitor the tasks/mission of journals. Originality/value The ABC approach is a unique way to assess performance or impact of research actors considering the characteristics of the areas in which output is published and as such less prone to manipulation or gaming.

Purpose The main goal of this paper is to show that international peer review can work in China’s context with satisfactory outcomes. Moreover, this paper also provides a reference for the practice of science and technology management. Design/methodology/approach This paper starts with a discussion of two critical questions about the significance and design of international peer review. A case study of international peer review of CAS Centers for Excellence is further analyzed. Findings International peer review may provide a solution to address the problem of quantitative oriented research evaluation in China. The case study of research evaluation of CAS Centers for Excellence shows that it is possible and feasible to conduct an international peer review in China’s context. When applying this approach to other scenarios, there are still many issues to consider including individualized design of international peer review combined with practical demands, and further improvement of theories and methods of international peer review. Research limitation 1) Only the case of international peer review of CAS Centers for Excellence is analyzed; 2) A relatively small number of respondents were surveyed in the questionnaire. Practical implications The work presented in this study can be used as a reference for future studies. Originality/value Currently, there are no similarly detailed studies exploring the significance and methodology of international peer review in China.

Purpose This study expands on the results of a stakeholder-driven research project on quality indicators and output assessment of art and design research in Flanders—the Northern, Dutch-speaking region of Belgium. Herein, it emphasizes the value of arts & design output registration as a modality to articulate the disciplinary demarcations of art and design research. Design/methodology/approach The particularity of art and design research in Flanders is first analyzed and compared to international examples. Hereafter, the results of the stakeholder-driven project on the creation of indicators for arts & design research output assessment are discussed. Findings The findings accentuate the importance of allowing an assessment culture to emerge from practitioners themselves, instead of imposing ill-suited methods borrowed from established scientific evaluation models (Biggs & Karlsson, 2011)—notwithstanding the practical difficulties it generates. They point to the potential of stakeholder-driven approaches for artistic research, which benefits from constructing a shared metadiscourse among its practitioners regarding the continuities and discontinuities between “artistic” and “traditional” research, and the communal goals and values that guide its knowledge production (Biggs & Karlsson, 2011; Hellström, 2010; Ysebaert & Martens, 2018). Research limitation The central limitation of the study is that it focuses exclusively on the “Architecture & Design” panel of the project, and does not account for intra-disciplinary complexities in output assessment. Practical implications The goal of the research project is to create a robust assessment system for arts & design research in Flanders, which may later guide similar international projects. Originality/value This study is currently the only one to consider the productive potential of (collaborative) PRFSs for artistic research.

Purpose To develop and test a mission-oriented and multi-dimensional benchmarking method for a small scale university aiming for internationally first-class basic research. Design/methodology/approach An individualized evidence-based assessment scheme was employed to benchmark ShanghaiTech University against selected top research institutions, focusing on research impact and competitiveness at the institutional and disciplinary levels. Topic maps opposing ShanghaiTech and corresponding top institutions were produced for the main research disciplines of ShanghaiTech. This provides opportunities for further exploration of strengths and weakness. Findings This study establishes a preliminary framework for assessing the mission of the university. It further provides assessment principles, assessment questions, and indicators. Analytical methods and data sources were tested and proved to be applicable and efficient. Research limitations To better fit the selective research focuses of this university, its schema of research disciplines needs to be re-organized and benchmarking targets should include disciplinary top institutions and not necessarily those universities leading overall rankings. Current reliance on research articles and certain databases may neglect important research output types. Practical implications This study provides a working framework and practical methods for mission-oriented, individual, and multi-dimensional benchmarking that ShanghaiTech decided to use for periodical assessments. It also offers a working reference for other institutions to adapt. Further needs are identified so that ShanghaiTech can tackle them for future benchmarking. Originality/value This is an effort to develop a mission-oriented, individually designed, systematically structured, and multi-dimensional assessment methodology which differs from often used composite indices.

Purpose Communicating scientific results to the public is essential to inspire future researchers and ensure that discoveries are exploited. News stories about research are a key communication pathway for this and have been manually monitored to assess the extent of press coverage of scholarship. Design/methodology/Approach To make larger scale studies practical, this paper introduces an automatic method to extract citations from newspaper stories to large sets of academic journals. Curated ProQuest queries were used to search for citations to 9,639 Science and 3,412 Social Science Web of Science (WoS) journals from eight UK daily newspapers during 2006–2015. False matches were automatically filtered out by a new program, with 94% of the remaining stories meaningfully citing research. Findings Most Science (95%) and Social Science (94%) journals were never cited by these newspapers. Half of the cited Science journals covered medical or health-related topics, whereas 43% of the Social Sciences journals were related to psychiatry or psychology. From the citing news stories, 60% described research extensively and 53% used multiple sources, but few commented on research quality. Research Limitations The method has only been tested in English and from the ProQuest Newspapers database. Practical implications Others can use the new method to systematically harvest press coverage of research. Originality/value An automatic method was introduced and tested to extract citations from newspaper stories to large sets of academic journals.