Fifty years of the Journal of Perinatal Medicine: an altmetric and bibliometric study

Introduction

The Journal of Perinatal Medicine (JPM) is an international journal that publishes original research related to perinatal medicine, obstetrics, and neonatology [1]. The Journal, which is the official journal of the World Association of Perinatal Medicine, the International Academy of Perinatal Medicine, and the New York Perinatal Society, was first published in January 1973 [1]. As the Journal approaches its fiftieth year of continuous publication, several academic questions arise: What has been the impact of the Journal? What articles published by the Journal have had the most impact? How has the journal evolved within the changing academic and research standards of the last half-century?

These questions, among others, may be answered with bibliometrics, which is the application of statistical methods to quantitatively analyze research [2]. Citation analysis is a conventional type of bibliometric methodology based on citation rates that assumes frequently cited articles and the journals that publish them are impactful. Citation rates are heavily influenced by several biases, however, such as time bias (older articles have more time to be cited) [3] and author self-citations (authors generate citations by repeatedly citing their own works) [4]. The Relative Citation Ratio (RCR) is an novel metric of research influence that was developed by intramural scientists at the National Institutes of Health (NIH) Office of Portfolio Analysis to address some of these limitations [5]. The RCR is calculated from the number of citations that an article receives divided by an expected citation rate derived from the performance of articles within a specific field and benchmarked to a peer comparison group [5]. Although it may provide better insight into the influence of articles within the same medical field, the RCR still relies on citation rates and may be similarly biased.

There is growing awareness that the extent of research engagement with public and social media platforms may be an alternative way to gauge research influence that has advantages over citation-based metrics. The Altmetric Attention Score (AAS) is one such metric that is based on research engagement with these platforms [6]. The AAS is determined by an automated algorithm that is derived from the number of posts on various public and social media platforms, but it also considers the quality of the posts’ sources in determining the relative weight of their contributions to the score (e.g. mentions in national newspapers count more than blog posts) [7].

In this cross-sectional study of all articles published in JPM since the journals’ inception, we performed an altmetric and bibliometric analysis with two main objectives. First, we sought to identify the most influential articles published in the Journal based on three metrics of research influence (two based on citation rates [the absolute citation number and the RCR] and one based on alternative metrics [the AAS]). Second, we sought to compare influential articles based on citation rates to articles that had highest AAS. We hypothesized that this analysis would provide insight into the contribution of the Journal of Perinatal Medicine in the last 50 years, but would also provide insight into where the Journal is heading. Further, we anticipated that the study would serve as an example for other researchers who wish to evaluate the impact of academic journals.

Materials and methods

We performed a cross-sectional study of all articles published by the Journal of Perinatal Medicine from January 1973 to September 2022. Data from the study was downloaded from the Clarivate Web of Science (WOS) Core Collection, the NIH Open Citation Collection (OCC), and the Altmetric Explorer on 6 Sept 2022. The WOS is an online, paid-access bibliometric resource that contains citations data from academic journals, conference proceedings, and other documents [8]. The OCC is a publicly available database that includes citations data from federally funded resources, including PubMed Central, MEDLINE, Entrez, CrossRef, and reference data from open access articles via a machine-learning pipeline that extracts citation data from online full-text resources [9]. The Altmetric Explorer is an online, paid-access altmetric aggregation service that quantifies research engagement with social networking platforms, news outlets, blogs, and reference managers, among other platforms [10, 11]. All data used in this study are publicly available and include no protected health information. Therefore, we did not seek Institutional Review Board approval. The study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cross-sectional studies [12].

We identified all articles published in JPM (1973–2022) and indexed in the WOS. We downloaded available bibliometric data, including author affiliations and countries of origin, times cited in all WOS databases, publication year, and open access status. We subsequently queried the OCC for all articles published in the Journal and indexed in the database (1980–2020), including RCR, citations per year, NIH percentile, and the Approximate Potential to Translate (APT) score. The NIH percentile reflects the ranking of RCRs compared to NIH publications. The APT score is an estimate of the likelihood that an article will be cited in the future (0.5 indicates very low likelihood and 0.95 indicates very high likelihood) [13]. Finally, we queried the Altmetric Explorer to identify AASs and mentions in news sources, blog sources, policy sources, patent sources, Google+, Wikipedia, LinkedIn, social media (including Twitter and Facebook), and reference managers (including Mendeley). Data from the OCC and Altmetric Explorer were merged with bibliometric data from the WOS by PubMed ID.

Articles cited 100 or more times in the WOS, meeting the definition for Citation Classics proposed by Eugene Garfield [14], were identified. We then identified the same number of articles with highest RCRs and the same number of articles with highest AAS. We selected the same number of articles in each group to facilitate comparisons. These influential articles represented the study cohort and were selected for further review.

In order to characterize the influential articles, we reviewed online abstracts and, when necessary, obtained the full text of the manuscripts. We characterized each article by country of origin, institution, study design, topic, and open access status. The country of origin was determined by the author affiliations listed in the WOS. Topics were determined by review of keywords. Study design was defined according to prior bibliometric studies [3, 15], [16], [17] as basic/translational, observational, reviews, consensus statements, systematic review/meta-analysis, randomized clinical trials and non-randomized clinical trials, new procedure/case reports, editorials, and other.

Results

Altmetric and bibliometric data were downloaded from the WOS, OCC, and Altmetric Explorer on 6 Sept 2022 (Figure 1). We identified 4,095 articles that were indexed in the WOS. The bibliometric data were merged with OCC and Altmetric data by PubMed ID. Of those WOS articles, 3,959 (96.7%) had RCRs in the OCC and 939 (22.9%) had AAS in the Altmetric Explorer. 34 (0.8%) Citation Classics with ≥100 citations in the WOS were identified. These articles as well as the n=34 articles with highest RCRs (top-RCR articles) and the n=34 articles with highest AAS (top-AAS articles), representing a total of 83 unique articles, were selected for further review (Supplementary Table 1).

Figure 1:

Study flow diagram. PMID, PubMed ID; WOS, ISI web of science; OCC, NIH open citations collection; RCR, relative citation ratio; AAS, altmetric attention score.

Most influential articles in JPM

In Table 1, we characterize the most influential articles published in the Journal of Perinatal Medicine from 1973 to 2022. The median year of article publication was 2010. These articles were cited median of 67 times, had median RCR of 3.4, and had median AAS of 14. The majority of articles were observational studies and reviews.

Table 1:

Characteristics of the most influential articles published in the Journal of Perinatal Medicine by three metrics of research influence, 1973–2022.

	All influential articles
	(n=83)
Citations in WOS	67 (17–114)
Citations per year	5.3 (2.1–9.8)
Publication year	2010 (2003–2018)
RCR	3.4 (1.7–5.0)
NIH percentile	87.6 (54.8–93.0)
APT	0.8 (0.8–1.0)
AAS	14 (3–28)
Origin a
US	17 (20.5)
Non-US	50 (60.2)
Both	16 (19.3)
Study design
Basic/translational	3 (3.6)
Observational	24 (41.0)
Review	29 (34.9)
Consensus statement	3 (3.6)
Systematic review/metaanalysis	3 (3.6)
RCT	2 (2.4)
New procedure/case report	4 (4.8)
Nonrandomized clinical trials	1 (1.2)
Editorial/letter	3 (3.6)
Other	1 (1.2)
Open access	27 (32.5)

1. The most influential articles were cited because they had ≥100 citations in Clarivate Web or Science or had highest RCR in the NIH Open Citation Collection or had highest AAS in the Altmetric Explorer. Data is median (interquartile range) and n (percent). ^aDetermined by author affiliation in the WOS. RCR, relative citation ratio; AAS, altmetric attention score; WOS, clarivate web of science; NIH, National Institutes of Health; APT, approximate potential to translate score; US, United States; RCT, randomized clinical trial.

The article that received the most citations in WOS was a practice guideline about fatty acids in pregnancy, lactation, and infancy that was published in 2008 by members of the World Association of Perinatal Medicine Dietary Guidelines Working Group [18]. The articles with highest RCR and highest AAS, published in 2020 and 2022 respectively, focused on the impact of coronavirus disease 2019 (COVID-19) in pregnancy [19, 20].

More than half of the influential articles originated from collaborations that included the US and other countries. There were global contributions from collaborators in 37 countries and 178 institutions. The countries and institutions that contributed three or more articles are described in Figures 2 and 3. The institutions that contributed the most articles were the NIH (n=11), the NIH Eunice Kennedy Shriver National Institute of Child Health & Human Development (NICHD, n=11), and Wayne State University (n=11). The countries that contributed the most articles were the US (n=34), Germany (n=15), and Italy (n=12).

Figure 2:

Institutions that published three or more influential articles in the Journal of Perinatal Medicine, 1973–2022.

Figure 3:

Countries that published three or more influential articles in the Journal of Perinatal Medicine, 1973–2022.

The most common topic for influential articles was obstetric ultrasound (n=12). COVID-19 (n=10), preterm labor/prelabor rupture of membranes (PPROM, n=8), infections in pregnancy (n=5), and inflammation (n=5) also featured prominently.

Altmetric vs. bibliometric analysis of influential articles

Top-cited and top-RCR articles were characterized and compared with top-AAS articles. In these analyses, we excluded three articles that were featured on both the top-cited and top-AAS lists as well as three articles that were on the top-RCR and top-AAS lists. Top-cited articles had higher median citations and citations per year compared to top-AAS articles (Table 2). These articles also had substantially higher NIH Percentiles and APT scores. The most common study type for articles in each group was review, but top-cited articles had higher representation of observational studies, and top-RCR and top-AAS articles had more consensus statements. Compared to top-AAS articles, top-cited articles were older (median year of publication 2006 vs. 2017, median difference −11.0, 95% CI −15.9 to −6.1). The breakdown of engagement with public and social media platforms for top-AAS is described in Table 3. The decade of publication by metric of research influence is further described in Figure 4.

Table 2:

Comparison of influential articles published in the Journal of Perinatal Medicine by three metrics of research influence, 1973–2022.

	Top-cited	Top-RCR	Top-AAS	Top-cited vs. top-AASa	Top-RCR vs. top-AASb
	(n=34)	(n=34)	(n=34)
				Median difference (95% CI)c
Citations in WOS	117 (111–147)	84 (47–147)	13 (5–62)	104.0 (86.6–121.4)	74.0 (35.3–112.7)
Citations per year	7.3 (4.9–10.6)	11.1 (6.9–19.3)	2.3 (0.7–4.6)	5.5 (3.1–7.9)	9.2 (6.0–12.4)
Publication year	2006 (2001–2010)	2013 (1999–2019)	2017 (2009–2020)	−11.0 (−15.9 to −6.1)	−5.0 (−11.6 to 1.6)
RCR	3.6 (3.2–5.5)	5.4 (4.6–8.0)	0.9 (0.3–2.3)	2.9 (1.8–4.1)	4.5 (3.3–5.8)
NIH percentile	88.7 (86.5–94.2)	93.9 (92.3–97.0)	41.3 (15.0–75.5)	56.5 (30.8–82.2)	61.2 (37.6–84.8)
APT	1.0 (0.8–1.0)	1.0 (0.8–1.0)	0.6 (0.3–0.8)	—d	—d
AAS	3.0 (3.0–9.0)	3.5 (2.5–10.0)	26.5 (17.0–35.0)	−23.0 (−29.9 to −16.1)	−23.0 (−30.5 to −15.5)
	n=17	n=20
Origin e
US	6 (17.7)	5 (14.7)	9 (26.5)	—	—
Non-US	18 (52.9)	17 (50.0)	20 (58.8)	—	—
Both	10 (29.4)	12 (35.3)	5 (14.7)	—	—
Study design
Basic/translational	3 (8.8)	0	0	—	—
Observational	8 (23.5)	0	0	—	—
Review	17 (50.0)	16 (47.1)	16 (47.1)	—	—
Consensus statement	2 (5.9)	12 (35.3)	10 (29.4)	—	—
Systematic review/metaanalysis	1 (2.9)	1 (2.9)	2 (5.9)	—	—
RCT	1 (2.9)	1 (2.9)	2 (5.9)	—	—
New procedure/case report	1 (2.9)	1 (2.9)	1 (2.9)	—	—
Non-randomized clinical trials	1 (2.9)	2 (5.9)	2 (5.9)	—	—
Editorial/letter	0	1 (2.9)	1 (2.9)	—	—
Open access	9 (26.5)	16 (47.1)		—	—

1. Data is median (interquartile range) and n (percent). ^aIncludes 31 unique articles on both lists and excludes three articles that were featured on both lists. ^bIncludes 30 unique articles on both lists and excludes four articles that were featured on both lists. ^c95% confidence intervals were based on 500 bias-corrected bootstrap resampling method. ^dConvergence not achieved. ^eDetermined by author affiliation in the WOS. RCR, Relative Citation Ratio; AAS, altmetric attention score; WOS, clarivate web of science; NIH, National Institutes of Health; APT, approximate potential to translate score; US, United States; RCT, randomized clinical trial.

Table 3:

Characteristics of engagement with public and social media platforms for articles with highest altmetric attention scores in the Journal of Perinatal Medicine .

	Top-AAS
	(n=34)
News mentions	1.5 (1.0–4.0)
Blog mentions	0 (0–1.0)
Policy mentions	0 (0–0)
Patent mentions	0 (0–0)
Twitter mentions	3.0 (1.0–10.0)
Facebook mentions	0 (0–1.0)
Wikipedia mentions	0 (0–0)
Google+ mentions	0 (0–0)
Linkedin mentions	0 (0–0)
Mendeley readers	51.5 (21.0–105.0)

1. Data is median (interquartile range). AAS, altmetric attention score.

Figure 4:

Decade of publication of influential articles published in the Journal of Perinatal Medicine by three metrics of research influence, 1973–2022.

Although there was some overlap among topics, there were notable differences in the topics of top-cited articles, top-RCR articles, and top-AASs articles. For top-cited articles, seven articles were about preterm labor/PPROM, five were about ultrasound, four were about inflammation, and three were about preeclampsia. For top-RCR articles, there were eight articles about COVID-19 and eight articles about ultrasound. Other important topics were inflammation (n=4) and infections in pregnancy (n=3). For top-AAS articles, the topics of greatest interest were COVID-19 (n=3), nutrition during pregnancy (n=3), and labor and delivery care (n=3).

Discussion

In this cross-sectional altmetric and bibliometric study of articles published in the Journal of Perinatal Medicine, we identified and characterized the journal’s most influential articles. The study identified 83 influential articles based on three metrics of research influence, including two metrics based on citation rates and one metric based on engagement with public and social media platforms. The articles covered a range of topics in perinatal medicine, but top-cited articles were generally related to fundamental clinical topics of perinatal medicine (such as ultrasound and preterm delivery) whereas top-RCR and top-AAS articles prominently featured articles about pressing current events, namely COVID-19.

There are various way to evaluate the influence of a journal. The journal impact factor is one such method, but its calculation is based on citation rates [21], can be manipulated [22, 23], and takes a minimum of two years to generate. Citation analysis may provide complementary insights into the contributions of specific journals and has been performed for specialty journals such as the American Journal of Obstetrics & Gynecology [15] and in journals of other fields [24], [25], [26]. Although these analyses recognize influential articles and quantify their reach, they are based on characteristics of top-cited articles and are prone to biases. To our knowledge, the analysis performed in this study is unique in that it employed multiple metrics of research influence to evaluate the impact of all articles published in a specific journal. For example, 31 (37.3%) articles were included in this analysis by RCR and AAS scores that had absolute citations less than 100 and would not have been identified in conventional bibliometric studies.

Several observations warrant further discussion. First, the study period covered 50 years, yet the median year of top-cited article publication was 2006. While time bias may provide some advantage of older articles, it is interesting that the majority of top-cited articles (n=27, 79.4%) identified in this study were published in 2000 or later. It is possible that this reflects the natural maturation of the Journal and/or the work of the editorial staff and reviewers in enhancing the reach of the Journal.

Second, there was overlap among top-cited and top-RCR articles, which has been previously described [16], but the analysis of top-RCR articles that were not on the top-cited list identified influential articles with high APT scores, NIH Percentiles, and citations per years. These articles are highly influential at this time and will likely become Citation Classics in the future. Although top-cited articles are historically important and the majority remain relevant for clinicians and researchers in the field, a bibliometric analysis based on RCR identifies articles that are making contemporaneous impact.

Third, the top-AAS articles warrant a closer look. Although we identified the 34 articles with highest AASs in this study, the articles had relative limited engagement with public and social media platforms. Although most of the top-AAS articles had AAS that were in the top-5% of published articles, the median AAS in our study was 26.5, which was substantially lower compared to the mean AAS of top-AAS articles in a study that looked at influential articles in obstetrics and gynecology (mean 763.1, standard deviation [SD] 520.8). The mean AAS of top-cited articles (mean 49.9, SD 81.6]) and top-RCR articles (116.2, SD 415.9) in that study were also higher than the median AAS in our study. Further, the top-AAS in our study had more readers on Mendeley and fewer posts on Facebook and Twitter, which was different from what was observed in the previously mentioned study. In that study, top-cited articles had much greater numbers of Mendeley readers (mean 447.3, SD 274.7) compared to top-AAS articles (mean 101.3, SD 94.3) and significantly fewer posts on Facebook and Twitter. The reasons for these differences are not clear, but could reflect that the Journal of Perinatal Medicine does not yet have a mature social media presence and/or the authors are more broadly international and use different social media platforms.

Conclusions

In this cross-sectional altmetric and bibliometric study of articles published in the Journal of Perinatal Medicine, we identified and characterized the Journal’s most influential articles in the last 50 years. The articles, which were identified by conventional bibliometric analysis employing citation rates, as well a novel metric of research influence called the RCR and an alternative metric of research influence based on engagement with public and social media platforms, covered a range of topics in perinatal medicine. The analysis provided insight into the types of articles that have been historically relevant to the Journal’s readership, but also identified articles of contemporary relevance. The study suggests that the Journal’s profile of engagement with public and social media platforms is not yet mature, with opportunities to build, and underscores the need for a metric that incorporates quality into its assessment of impact. Finally, the study may – and should – be used as framework for researchers who are looking to evaluate the impact of other academic journals.