Towards OER Statistics: Data Requirements for the Scientometric Analysis of Open Educational Resources

2.1 OER Statistics Framework

The OER statistics framework is a tool for the scientometric consideration of OER (Kullmann & Weimer, 2024; Weimer & Kullmann, 2025, in press). It builds the basis of the following work. Thus, the data requirements discussed in the following are derived from it (see Section 4). The aim of the OER statistics framework is to recognise teaching performance in higher education by expanding the recognition and rewards system of science to include freely available and possibly openly licensed teaching/learning resources as special scientometric measurement objects. These are categorised as achievements worthy of recognition alongside other scientific outputs such as publications. Due to their special characteristics and their diversity, OER represents a very complex object of measurement. Thus, OER statistics are based on a specific understanding of OER which is summarised in the following OER definition for scientometric purposes (Kullmann & Rasulzade, 2025; Kullmann, 2025):

OER are publicly available, freely accessible materials that have been created specifically for teaching/learning purposes and are of sufficient quality and level of personal contribution. OER are divided into the categories of Dedicated Learning Content for learning materials primarily intended for learners, and Learning Design Content for supporting materials for teachers. OER also include contributions to a supportive OER ecosystem that facilitates the creation, use and dissemination of OER. This includes, for example, infrastructure elements developed for open teaching/learning purposes, such as OER repositories, OER-supportive working environments, editorial work, consulting services, training and other support services for OER authors.

To create the broadest possible basis for the recognition of teaching achievements, the concept of openness is not only related to legal openness and far-reaching possibilities for subsequent use but is extended to include the possibility of free access, i.e. public availability of free-of-charge teaching/learning materials. In order to achieve a distinction from other scientific output, only artefacts that were created specifically for teaching/learning purposes or that contribute to the promotion of OER through various organisational measures within an OER ecosystem are eligible for OER statistics. Other important characteristics of OER in the context of the OER statistics framework are a sufficient level of personal contribution by OER authors and sufficient content-related and didactical quality. The problem that an author’s OER can be edited by other authors as part of the OER lifecycle (Figure 1) and published as a new version under their name without major efforts worth being rewarded, is specifically relevant with regard to the consideration of performances. A sufficient quality of OER plays a crucial role especially for learners who rely on the provided open artefacts. Apart from that, from a scientometric view, only high-quality material is eligible for consideration as it is the case with scientific publications. Ensuring that these two entry requirements are met is no easy task. For this reason, before the OER statistics are compiled, it must be determined what a sufficient level of personal contribution and sufficient quality look like in concrete terms and how their existence is to be checked in each case. Due to the wealth of possible representations of OER (worksheets, slide sets, videos, courses, etc.), instead of considering every single type and format, the OER statistics framework classifies resources into three categories: Dedicated Learning Content, Learning Design Content, and OER ecosystem. OER falling into the Dedicated Learning Content category represents OER created specifically for teaching/learning purposes, which can be freely and ideally openly reused by other teachers or used by learners to acquire knowledge. The Learning Design Content category includes OER that supports teachers in the creation of their own teaching/learning materials. The OER ecosystem category is used to present services for establishing an OER-friendly environment, which includes counselling services as well as the operation of OER infrastructures, for example.

Figure 1

OER lifecycle (Glahn, Kalz, Gruber, & Specht, 2010).

The OER statistics framework consists of indicators at the level of individuals (individual level) and the institutional level for the assessment of organisational units like universities. Table 1 illustrates these two levels and its indicators.

There is partial congruence between the two levels, but there are also significant differences to consider. For example, the categorisation of OER into granularity classes at the institutional level is not the case at the individual level. This is due to the fact that a more qualitative assessment of teaching/learning performance is sought at the individual level in order to counteract the typical criticisms of scientometric assessments of individuals (cf. Leiden Manifesto; Hicks et al., 2015). Thus, the person to be evaluated should be integrated into the process and the output should be evaluated individually. The indicators also differ between the levels in other areas, which is due to the different scientometric evaluation of individuals and institutions.

2.2 OER Infrastructures

Publication, findability, and accessibility are important prerequisites for the reception of OER in the educational field. To ensure this, around the globe specific OER infrastructures have emerged in recent years that can be regarded as a special case of Digital Knowledge infrastructures. OER infrastructures include open courseware (OCW), thematic websites, Wikimedia platforms, and, in particular, open textbook libraries and OER repositories (Marín & Villar-Onrubia, 2023; Figure 2).

Figure 2

OER Infrastructures as specialised Digital Knowledge Infrastructures (Marín & Villar-Onrubia, 2023).

OER infrastructures all serve to make OER discoverable and to disseminate them among interested reusers. Due to their aggregative nature, OER repositories are particularly noteworthy. As specialised infrastructures, in many cases, they provide OER from a range of different subject areas, in various representations (e.g. slides, exercises, and videos) created at various universities. With regard to the provided functions and services, besides enabling OER authors to publish their materials, they ensure that the resources are adequately described by appropriate metadata and often also provide quality assurance measures in terms of content and formal aspects. In addition, they enable a qualified search for OER and the provision of the OER themselves for download. Another important reason for placing OER infrastructures at the centre of investigations into their suitability for the OER statistics framework is that OER does not currently belong to the collection mandate of literature databases.

OER infrastructures are discussed in the literature from various perspectives. The desirable and actually provided functions (Heck et al., 2020; Hiebl, Kullmann, Heck, & Rittberger, 2023; Santos-Hermosa, Ferran, & Abadal, 2017) and quality aspects (Atenas & Havemann, 2013; Romero-Pelaez, Segarra-Faggioni, Piedra, & Tovar, 2019) are important topics. OER metadata also play a central role (Menzel, 2023; Segarra-Faggioni & Romero-Pelaez, 2022; Simão de Deus & Barbosa, 2020; Tavakoli, Elias, Kismihók, & Auer, 2020; Tischler, Heck, & Rittberger, 2022). The results of these metadata studies make different statements about the quality, comprehensiveness, and completeness of the metadata used, with different focuses, for example, on findability, didactic description, or quality. In the context of the scientometric analysis of OER, which is relevant for the present study, they can only be used to a very limited extent, since the OER metadata requirements posed by scientometric analyses are not discussed. Since large-scale OER infrastructures aggregate OER and can thus be compared in many ways to scientific literature databases or Open Access repositories, they represent a suitable starting point for investigating the current availability of data for creating OER statistics based on the OER statistics framework. For this reason, in the following, the data requirements from the OER statistics framework will first be operationalised and their feasibility will be checked in a second step using selected OER infrastructures. The focus is not on the general quality of OER metadata, but rather on identifying the data requirements of the OER statistics framework and the extent to which these are already provided by selected OER infrastructures.

4.1 Data Requirements Derived from the OER Definition

The OER statistics are based on a specific understanding of OER, which results in entry requirements for every teaching/learning artefact to be considered scientometrically. Firstly, it must be checked whether it has been created specifically for teaching/learning purposes (requirement 001). A sufficient level of personal contribution by the authors involved in an OER (requirement 002) and of sufficient quality of the OER (requirement 003) must be ensured. Furthermore, information about the resource type of an OER (requirement 004) is important to enable the classification into one of the two categories Dedicated Learning Content or Learning Design Content. In addition, the artefact under consideration has to be either openly licensed or, as a minimum requirement, publicly available and accessible, and free of charge. With regard to the examination of the degree of openness, there is an overlap with the openness indicators, which is why the data requirements regarding this issue are discussed there (requirements 012, 013, 014, 015).

4.2 Data Requirements of the Productivity Indicators

The productivity indicators consist of the total OER count in the categories Dedicated Learning Content, which contains materials created for teaching/learning purposes, and the category Learning Design Content, which contains artefacts that support teachers in the creation of teaching materials or lessons. At the institutional level of the OER statistics, the Dedicated Learning Content is categorised according to four levels of granularity. In addition, the OER publication dynamics and the indexed growth rate are calculated, which make it possible to recognise trends regarding the publication of OER. The compilation of productivity indicators is rounded off by the determination of a subject profile.

For the total counts, the number of published OER by or in cooperation with an institution must be determinable. To make this possible, all OER under consideration must be explicitly assigned to a uniquely identifiable institution (requirement 005). The materials should be categorised in terms of granularity (requirement 006). The resource type, which is essential for the categorisation of Dedicated Learning Content or Learning Design Content, is also necessary. This is already required by the OER definition (requirement 004).

The publication dynamics at the institutional level can be illustrated by the average annual growth rate of OER in a given observation period. The publication dynamics is calculated using the following formula:

The values t ₀ and t _T refer to the first and last years of the observation period. |T| indicates the number of observation years. For the OER publication dynamics, information on the publication year of all OER under consideration is required (requirement 007). It should be noted that the OER statistics do not use a fractional but work with full counting (DZHW, 2021).

In order to determine the indexed growth rate of OER, the publication figures determined for the first year under consideration are equated with 100. It is then determined for every subsequent year of interest how these compare to the first year respectively. Values above 100 are interpreted as growth, while values below 100 are interpreted as a decline. For the indexed growth rate, the publication years of the OER under consideration are crucial (requirement 007).

For the assignment of OER to a subject area, appropriate categorisation via suitable metadata is necessary (requirement 008).

4.3 Data Requirements of the Resonance Indicators

The resonance of OER is measured via attributions, which can be compared with citations in the research publication sector. At the institutional level, the indicators MNAS, highly attributed OER, and unattributed OER are relevant.

As a fieldnormalised rate the MNAS is based on the average attribution rate in the scientific community under consideration. The calculation is based on the following formula, analogous to the MNCS method for scientific publications (DZHW, 2021):

where P _o represents the number of OER of the institution o in a selected period, a _i the attributions of the ith OER in this timeframe, and EAR i field the expected attribution rate in the field the OER belongs to, in each case for the publication year of the individual OER. Data on publication years (requirement 007), categorisations by subject area (requirement 008), and attributions per OER and year are required (requirement 009).

Highly attributed OER indicates how many OER of an institution belong to the most frequently attributed 10% of OER of a subject area in a defined observation period. To determine this, the OER must be assigned to an institution (requirement 005) and subject area (requirement 008). It is also necessary to know the respective publication year (requirement 007). Furthermore, the current attributions for each OER of a subject area must be available for the corresponding observation period (requirement 009). The same data are necessary to determine the unattributed OER.

4.4 Data Requirements of the Cooperation Indicators

The OER statistics measure cooperation between different sectors (sector cooperation) as well as institutions and countries (most important co-publishing institutions/partner countries). First of all, the allocation of the OER to an institution is fundamentally important (requirement 005). In addition, the affiliation of an institution to a specific sector is crucial. Besides universities, cooperation partners can, for example, include non-university research institutes, and partners from industry or the public sector (administration, government). Information on sector (requirement 010) and country affiliation (requirement 011) are therefore necessary for the cooperation indicators at the level of the institutions involved in the creation of OER.

4.5 Data Requirements of the Openness Indicators

The openness indicators for OER provide support at the institutional level in visualising the openness of teaching services to the outside world at a specific point in time as well as over time. The measurement is based on open licences and their allocation to the categories outlined in Table 3. The allocation to the first three categories is based on the licence information provided for an OER under consideration (requirement 012). The OER statistics focuses on Creative Common (CC) licenses which are very usual in the OER sector. If OER is provided with other licences, a mapping between the licence systems is required.

While the classification of openly licensed OER is easy if the respective licence is known, category (4) requires explicit information on the modus of usability. Here, it must be ensured that the OER is actually freely available and accessible and can be used free of charge. Metadata that provides information on availability, accessibility, and is free of charge (requirements 013, 014, 015) is therefore required for materials with this level of openness.

4.6 Data Requirements of the Transfer Indicators

The transfer indicators attribution in an OER to research material and citation in a research material to OER can be understood as an extension of the resonance indicators due to their reference to attributions and citations. In the first case, attributions of OER in a research output are recorded. In the second case, referencing of research output in OER artefacts is depicted. By linking output from the areas of research and teaching via attributions or citations, a clear assignment of artefacts to one of the two areas of teaching or research is required. This can be solved by a separate document type for OER (requirement 016). With regard to the indicator OER material resulting from research projects, a direct and clearly documented provenience of OER is crucial (requirement 017). In addition, the publication years (requirement 007), subject areas (requirement 008) as well as attributions/references per OER and year are required (requirement 009).

4.7 Data Requirements of the Support Indicators

The support indicators aim to present activities and services for the creation and design of an OER-promoting ecosystem. Thus, the support indicators do not focus on OER material but on activities and initiatives of institutions that support OER and its typical lifecycle as outlined in Figure 1. This means, for example, to provide administrative and consulting services as well as technical infrastructures to help OER authors to create, publish, find and access, reuse in unadapted or enriched form, and republish OER artefacts. Accordingly, the support indicators deal with OER policies (requirement 018), OER-related certifications (requirement 019), the development and operation of OER infrastructures (requirement 020), funding in the context of OER (requirement 021), the provision of support services to strengthen OER authors (requirement 022) or measures to educate, promote and support OER communities (e.g. legal advice, provision of human or technical resources to create OER; requirement 023). In the OER statistics themselves, the existence of these measures is recorded dichotomously (“available” or “not available”). One exception is the OER funding indicator, where the amount of funding granted can be specified as a total.

4.8 General Data Requirements

In addition to the requirements needed specifically for the determination of the indicators on the institutional level, general requirements have to be fulfilled. It is very important that OER can be assigned to an institution (requirement 005). At a minimum, the individual OER should be uniquely identifiable. Thus, at least a title (requirement 024) and a uniquely identifiable persistent identifier such as the Digital Object Identifier (DOI; requirement 025) is crucial.

Table 4 summarises all identified data requirements.

5.1 Requirement 001: Teaching/Learning as the Purpose of Creation

All OER infrastructures analysed clearly state their collection mandate for OER created for teaching/learning purposes. In the case of MERLOT, the vocabulary comprises the value “Open Access Journal Article,” which reflects a broader understanding of OER that includes openly licenced scientific publications. The explicit labelling of OA articles in the MERLOT collection makes it easy to sort them out and only include dedicated teaching and learning material. In summary, due to the well-described collection mandates of all analysed OER infrastructures, the requirement that the artefact under consideration must be created specifically for teaching/learning purposes can be fulfilled by every OER infrastructure under consideration.

5.2 Requirement 002: Sufficient Level of Personal Contribution by OER Authors

Requirement 002 is very complex, as the first step would be to determine the criteria for a sufficient level of personal contribution. Another prerequisite for scrutinising the personal contribution of reusing OER authors is the identification of connections between OER versions. The direct mapping of a relationship between different OER versions via metadata modelling is not strictly necessary to find the source version of an OER derivative. However, this is very helpful. OER versioning already plays a role in some of the OER infrastructures analysed. OER Commons connects versions via relationships (CR_isAdaptationOf, CR_hasAdaption). In addition, it is possible to indicate via the CR_Parent_Modified element if an OER from another author has been changed. In the LOM for Higher Education OER Repositories metadata profile, versions of OER can be identified via the Lifecycle element. The AMB metadata profile uses the relationships isBasedOn, isPartOf, and hasPart to provide a reference to another resource under consideration. The establishment of relationships between OER that are provided in different OER infrastructures is not realised anywhere yet. In view of the sufficient level of personal contribution of OER authors, no content-related information is provided to answer the question if the author of the new version has made a sufficient personal contribution to enrich the previous version. The requirement 002 can therefore not be fulfilled by any OER infrastructure analysed yet.

5.3 Requirement 003: Sufficient Level of Quality

The documentation of the quality of the provided OER plays a particularly important role in the US OER infrastructures. MERLOT has various options for assessing the quality of OER. These include formal peer- and editor-reviews, as well as user ratings and the option for users to leave comments. The materials with quality ratings can be filtered out using the search mask. However, the documentation of the quality assessment results is not an overarching mandatory requirement. OER Commons takes a user-based approach to quality assessment. On the one hand, there are artefacts that have been evaluated by one or more users according to the Educators Evaluating the Quality of Instructional Products (EQuIP) or AchieveOER criteria. On the other hand, users can rate the OER they use by stars. It is also possible for users to leave comments. However, the quality assessment is not documented within the metadata profile yet. The German metadata profiles LOM for Higher Education OER Repositories and AMB as well as of the OTL and Open Stax do not provide quality characteristics of OER. However, for most of the books in OTL and all books in Open Stax, it is stated, that they have been formally peer-reviewed by educators in the field. Overall, the metadata schemes do not record formal quality assessment results on individual materials. However, there are often processes that serve the formal quality assurance of materials which are documented alongside the metadata profiles.

5.4 Requirement 004: Resource Type

With regard to the resource type of the OER provided, all the OER infrastructures analysed have suitable vocabularies for categorisation. Although these vocabularies overlap, they are not identical. The metadata profiles LOM for Higher Education OER Repositories and AMB use the same terminology. In the case of the Open Textbook Libraries OTL and Open Stax, a resource type is not explicitly provided but could be added easily as additional metadata. Overall, requirement 004 is fulfilled by all OER infrastructures analysed.

5.5 Requirement 005: Institutional Affiliation

A fundamental requirement is the documentation of a relationship between an OER and the issuing institution. This can be done directly or by assigning the authors to an institution. Of great importance here is the unique identifiability of the institution. This can be implemented via unique identifiers like the Research Organisation Registry (ROR). Basic institutional allocation is provided by all OER infrastructures analysed. In the case of MERLOT, it is possible to identify the institution via the author. However, it should be noted that it is not mandatory for institutions to specify an author and fill in the field. Furthermore, MERLOT does not work with persistent identifiers for authors and institutions. OER Commons requires the mandatory entry of an institution (CR_Provider field) but also refrains from using unique identifiers. The German metadata profile LOM for Higher Education OER Repositories demands the documentation of the relationship between contributors (authors, other contributors, and institutions). The use of unique identifiers is also provided for in principle. An URI reference to the Common Authority File (GND), Wikidata, or the use of a ROR are recommended for use. The mandatory documentation of institutional affiliation in the metadata is also planned for AMB. The use of unique identifiers analogous to the LOM-based metadata profile is also recommended. For OTL the books must be in use at multiple higher education institutions, or affiliated with a higher education institution, scholarly society, or professional organisation. The optional requirement and the resulting relationship of a book to an affiliated institution are only partially reflected in the metadata. In the case of Open Stax, the associated institution is usually specified in addition to the author. However, unique identifiers are not used. Overall, information on affiliated institutions is largely available, but from a scientometric point of view, there is clear potential for improvement with regard to the presentation of independent metadata and the use of persistent identifiers.

5.6 Requirement 006: Granularity of OER

Information on granularity can currently only be found in the metadata profile LOM for Higher Education OER Repositories, where this is explicitly mapped via the aggregation-level element. However, by specifying the resource type, it is possible to categorise the granularity using the available metadata in the case of OER Commons and MERLOT. This also applies to OTL and Open Stax, as they only offer open textbooks and the granularity can be derived from the book form. Overall, not all OER infrastructures provide a granularity categorisation. However, due to the information already available on the individual OER, such a categorisation is basically possible.

5.7 Requirement 007: Publication Year

The publication years of the OER under consideration are required to calculate the productivity indicators. Firstly, the creation year of an OER must be distinguished from the publication date in a repository, which is naturally more recent and will differ from the actual creation year. In the case of MERLOT, the publication date in the repository is specified instead of the creation year. OER Commons allows to document the creation year of an artefact but does not make this mandatory. As with MERLOT, on the individual pages of the artefact, the publication date in OER Commons can be found. The metadata profile LOM for Higher Education OER Repositories contains an option for specifying OER-related data within the contribute element in the metadata section. The contribute element can be used to specify persons with different roles in relation to the metadata for the respective OER. In this context, it has been specified that the date in the case of the creator role contains the date of the last change of the OER which represents the creation date. For the validator role, on the other hand, the publication date of the OER in a repository is documented. The AMB metadata profile provides the creation date (dateCreated), modification date (dateModified), and publication date (datePublished). The information is not mandatory. The Open Textbook Library records the publication year of a book. In the case of Open Stax, the publication date and the date of the last update of the version of a book published on the Internet are given. Overall, not all OER infrastructures provide the original creation date within the applied metadata profile yet.

5.8 Requirement 008: Subject Area

For the productivity indicators, it is also necessary to specify the subject area to which the OER under consideration belongs. The picture here is uniformly positive. In all OER infrastructures analysed, the OER is assigned to specific subject areas. It should be noted that the German metadata profile based on LOM and the German AMB both use the subject classification system for students at higher education institutions used in Germany by the Federal Statistical Office.^[17] However, overall, the vocabularies used are different.

5.9 Requirement 009: Attributions/References per OER and Year

The attributions of OER in other OER and/or scientific publications represent important information that is comparable to citations in scientific publications. Mapping the relationships between attributing and attributed OER is essential for determining the resonance indicators. Access to the references of the OER under consideration is crucial in order to be able to analyse these relationships. References in scientific publications can be evaluated via citation databases such as Web of Science, Scopus, or Open Citations for the majority of artefacts. However, this is not the case for references in OER. In none of the OER infrastructures analysed are the references stored separately and thus made evaluable.

5.10 Requirements 010 and 011: Sector and Country Affiliation of Institutions

With regard to the cooperation indicators, the sector and country affiliation of an institution are necessary. None of the OER infrastructures analysed currently provide explicitly corresponding data for the inclusion of this information.

5.11 Requirement 012, 013, 014 and 015: Information on Openness

The OER statistics not only considers openly licensed teaching/learning artefacts, but also OER with free access, free availability, and no cost. The recording of the degree of openness of artefacts via metadata varies between the different OER infrastructures. MERLOT records whether a material is licensed with a CCs licence of the CC-0 type or another CC licence with or without permission to use the material commercially. If a material has a different licence, it is recorded whether it can be used free of charge. In OER Commons, the individual CC licences are stored in detail in the metadata of an artefact. Other possible values are ‘Public Domain’ and information on more restricted usage rights in the categories ‘Read the fine Print’ and ‘Copyright Restricted’. Information on the free usability of materials without an open licence is not recorded. In the LOM for Higher Education OER Repositories and AMB metadata profiles, detailed legal information on the usability of an artefact is provided. With AMB, additional information on free access and free usability is provided. In general, the two Open Textbook Libraries only provide for the collection of openly licensed books in their guidelines. In the OTL metadata record, the information on the copyright status is explicitly documented. In the case of Open Stax, the licence information is stored in the GitHub repository with every single textbook. Overall, the information for requirement 012 is available in all infrastructures. However, information on requirements 013 to 015 is not available everywhere.

5.12 Requirement 016: OER as Document Type in Bibliographic Databases

The identification of OER in reference lists of publications or other scientific output is not yet possible without problems, as the systematic recording of OER as an artefact/document type in citation databases is not yet comprehensive. To determine the resonance indicators of the OER statistics, separate documentation of the attributions/references of OER has to be made. In addition, the document type OER has to be introduced in common citation databases.

5.13 Requirement 017: Provenience of OER

The provenience of the OER provided at the level of smaller units (e.g. projects) within institutions is not currently documented by any of the OER infrastructures analysed.

5.14 Requirements 018–023: OER Ecosystem

The information required to determine the support indicators is currently not mapped by any OER infrastructure.

5.15 Requirement 024–025: OER Identification

A title is always provided (requirement 024). The assignment of persistent identifiers is explicitly provided in the metadata profiles of OER Commons (CR_Native_ID) as well as LOM for Higher Education OER Repositories and AMB. In particular, the identification systems DOI, Handle, International Standard Book Numbers (ISBN), and Uniform Resource Names are proposed for use. In the case of OTL and Open Stax, the books have ISBNs. Thus, requirement 024 can already be met by the majority of the OER infrastructures analysed.

Table 5 breaks down the requirements and current data availability by indicator.