Data format standards in analytical chemistry

Introduction

The amount of research data is growing in all disciplines, and chemistry is no exception. Modern methods and instruments effortlessly produce large amounts of data, stored within the data infrastructure of research institutes. Some of that data might be published supplementing scientific articles, but the majority of data accessible currently does not lead to facile reuse of the data by other scientists or is readily available for computational analysis such as machine learning. The reasons for this situation are manifold: Even if most of the journal articles are published in digital form, important information is often hidden in text and images or even left out entirely for publication. If accessible, the required information is frequently given as free-form textual description [1]. Tables or schematic representations of e.g. chemical reactions in the article PDF files are often challenging to convert back to tabular formats for reuse. Their contextualization can be difficult due to missing common standards for metadata. The situation is even worse regarding data files of instrumental measurements, which are often recorded in proprietary vendor formats and impossible to be opened without the vendors’ software.

To overcome this situation, digital assets can (and should) be shared based on the FAIR principles for scientific data management [2]. These principles define that digital objects should be findable, accessible, interoperable and reusable to be of value for (different) potential consumers – human beings or computers. Findability and accessibility can be achieved by depositing data in public repositories and databases, which also ensures that all data is accompanied by a defined set of metadata. Interoperability and reusability depend on data standards to describe which data and metadata should be recorded and how they are made available. In other words, standards are prerequisites of FAIR data. Other studies already underlined the importance of data standards to prevent information decay [3], curation of old datasets with new methods [4] or the reuse of experimental raw data [5], [6]. Some publishers also started to encourage authors to submit data following the FAIR principles.

The availability and use of open standards to obtain FAIR data is key (Fig. 1). Thus, it is necessary to recommend and apply existing open standards or, if they do not suffice, to extend them or create new open standards for archival, exchange and reuse of experimental data and metadata. Standardized open formats allow reading, analyzing, and reusing data collected from different sources and enable integrated, comprehensible and reusable workflows. Lowering the technical barriers to access shared data will improve the quality of data, by simplifying the processes of review, quality control, comparison and reproduction [7]. Not only the quality of data itself, but also the quality of software applied in digital workflows can be enhanced, if software developers can rely on standards when implementing and maintaining servers, databases or tools.

Fig. 1:

The role of data standards to share research data. Any interaction coupled to sharing research data via data files, databases or repositories requires standards to define data formats, minimal information metadata, ontologies, identifiers, guidelines and tools. These facilitate findability in conjunction with persistent identifiers (PIDs) as well as accessibility and interoperability for humans and computers to enable reuse.

In this article, we will review the state of data standards and file formats in chemistry for selected analytical techniques including mass spectrometry, nuclear magnetic resonance spectroscopy, infrared and UV/Vis spectroscopy as well as X-ray diffraction and X-ray absorption and emission spectroscopy. Additionally, we summarize standardization efforts and bodies in chemistry and start with definitions of terms, which chemists might not be familiar with, but which are frequently used in this review.

Relationship between data models, specifications, file formats and software implementations

For successful data standards, several components are required: the data model and the data representation. A specification of a representation may contain the model implicitly.

The Data model describes how data is organized, which information they contain, the data types (e.g. text, numbers, lists), the relationship between these components, and rules of component and data integrity. Conceptually, the model and its components can be organized in different ways, e.g. flat, multidimensional, as network or hierarchical. The meaning of the applied components has to be well described in an unambiguous way. Ontologies can be consulted to ensure a consistent usage of the components in the context of the model. As the data model is abstract, it is necessary to implement representations for it, such as a database or data file on a computer.

File formats are one way to represent a data model. File formats as models can be categorized by different criteria: proprietarily vs. openly specified, binary vs. text, simple vs. complex or flat vs. n-dimensional (Fig. 2). For long-term storage, a compact binary format may be the first choice, while for further processing with cheminformatics tools a format based on standards like Comma Separated Values (CSV), Extensible Markup Language (XML) or JavaScript Object Notation (JSON) may be more favourable due to support by most programming languages. When thinking about data strategies, open formats should be considered. Proprietary formats may prevent further reuse due to issues connected to licensing, poor documentation and support by the vendor. Complex data often need more complex formats to express the meaning and relation of data.

Fig. 2:

Examples for different data file formats. XML and JSON are similar in their logical structure, but use different syntax to encode the data model. Binary files may also have an internal structure, but reading and access requires dedicated software libraries.

To represent data distributed over various files or to bundle files from different stages of the workflow, container file formats like ZIP can maintain the folder structure and provide compression. Index files or cross references may link these files in the container context.

The specification of a data file standard may describe the whole format from scratch or can utilize generic formats, which are standardized anywhere else. Such generic formats include CSV [8], XML [9] or JSON [10]. For some of these generic formats, a formal specification format is available, called the schema format, which streamlines the specification of format derivatives. XML or JSON files can be described by schemas. If the format cannot be described formally by a general or existing schema, it is important to create an unambiguous new specification, clear in description of syntax, and grammar, as well as in definition of the components of the underlying data model. File formats should also be extendable to allow the usage of the format for new techniques and with future requirements. Data file standards not fulfilling these criteria may fork into incompatible formats or will be outdated soon and replaced by new standards.

Any software importing or exporting a file format has to include readers and/or writers for this format, often accompanied by a validator for checking documents for adherence to the standard. Integration of any format into software is more likely if developers may rely on an implementation, available as a library, package or module for their programming language or environment. Thus, acceptance of a format also depends on the availability and support of implementations. Formats are more sustainable if they have a free licence and are open source. For formats described by schemas there are often well-maintained implementations for the generic format including validation. So only the schema file has to be provided. Implementations for formats, which are not openly specified and lack vendor support, are sometimes only feasible by reverse engineering, which is debatable concerning legality. Only a complete and unambiguous specification allows well-crafted implementations.

In this paper, we do not deal with the question of how to package and store data. This is a separate field, which has recently been addressed e.g. by RO-Crate [11], which describes ways how to link data stored in diverse locations.

Standardization efforts and -bodies in chemistry

To be adopted by the community of scientists, any data format or model is required to be well-defined and to include necessary and valuable information. This applies to standards developed under the auspice of an official standardization body and to community-driven efforts. Nonetheless, standardization bodies might guarantee long-term maintenance as well as further developments to adapt to new techniques and demands. Data standards can be created by community initiatives or through official generic or discipline focused standardization bodies. Some of these organizations and initiatives are listed in Table 1.

The International Union of Pure and Applied Chemistry (IUPAC) with its numerous working groups and committees [12] is a major player in the field of standardization in chemistry. Among other topics, the Subcommittee on Cheminformatics Data Standards is the maintainer of the JCAMP-DX format standard [13].

The Allotrope Foundation is “a group of pharmaceutical, device vendor, and software companies that develops and releases technologies […] to simplify the exchange of electronic data.” [14], established nearly 10 years ago. The foundation is working on 1) the Allotrope Data Format (ADF) which is based on the widely used domain independent hierarchical data format 5 (HDF5) combined with semantic metadata described via the resource description framework (RDF), 2) Allotrope Data Models (ADM), and 3) Allotrope Foundation Ontologies (AFO). Only the latter is licensed under Creative Commons terms. ADF is used in several industrial environments, since several instrument providers are members of the foundation and provide Allotrope compatible software interfaces for their devices. With the Allotrope Simple Model (ASM) there is a way to make data more accessible by using the popular JSON format.

The Pistoia Alliance is also a non-profit organization of more than 100 member companies, working in several subprojects. One of those is the Unified Data Model (UDM) [15] which covers experimental information on the synthesis of compounds and their testing, including referencing of analytical data, and the possibility to use the AFO. The UDM licence was recently changed to the liberal MIT licence, which should facilitate the wider adoption of this model.

The consortium for Standardization in Lab Automation (SiLA) is a non-profit initiative of software and device suppliers to standardize software interfaces for laboratory automation. Alongside their communication protocol for instruments, they promote the AnIML format (described below) for data storage and interchange [16].

NFDI4Chem is a national consortium in Germany [17], embedded in the interdisciplinary NFDI (Nationale Forschungsdateninfrastruktur [National Research Data Infrastructure]), which will contribute in developing and specifying recommendations for analytical data standards in chemistry for users but also for the instrument and software providers, as well as providing metadata and publication standards.

There are several other organizations to support the science community to make their data more FAIR and more open. GO FAIR is a worldwide acting initiative supporting the community in establishing FAIR principles by raising awareness, providing training and giving recommendations for technical standards and infrastructure [18]. The Committee on Data (CODATA) promotes Open Science [19], and thus, also FAIRness of data and open standards as requirements. The community-driven Research Data Alliance (RDA) has a similar mission of encouraging scientists to share and reuse their data [20].

There is currently no general agreement by the publishers on how to publish and cite data sets. Communities such as FORCE11, with currently about 3500 members [21], support initiatives in harmonizing publication standards including data citation. One of the prerequisites to make data citable is the existence of persistent identifiers (PIDs) [22]. The most widely adopted PIDs are Digital Object Identifiers (DOIs), with CrossRef as the main DOI registration agency for publications and DataCite as a provider of DOIs for datasets [23, 24].

Although these initiatives can help to raise awareness for the challenges connected to RDM and provide strategic advice in improving RDM, chemists need domain-specific information on existing standards, data models and file formats. Moreover, resources on repositories for datasets or databases with domain-specific information are required. NFDI4Chem will provide recommendations to this domain specific needs and will contribute to standard development for data exchange and archival by using, extending and creating open data formats.

There are some registries present which can be used as a starting point. FAIRsharing provides information on policies, standards and databases for many subjects including chemistry [25]. The Registry of Research Data Repositories (Re3data) also offers searchable information for domain-specific and discipline-agnostic repositories and databases [26].

Data formats for analytical data

When developing infrastructures for chemistry, one of the major tasks is to adopt or define data models and formats for exchange, storage and archival of analytical data. Vendors often provide proprietary formats to store instrumental and experimental metadata as well as data readings. With the growing set of workflows, software tools, databases and repositories there is demand for open and well specified models and format standards. It has also become evident that broad adoption of a standard data format benefits from reference software implementations, i.e. software modules for common programming languages with the main purpose to read and write a specific format, that can easily be reused in larger software projects.

A simple practice to export and exchange analytical data is conversion to text based and tabular formats, like CSV. These are human-readable and can usually be interpreted by the user. However, with the lack of metadata and specification, these formats are less appropriate for automatic processing and storage. Nevertheless, implementation of text based and tabular file format support is straightforward. Therefore, there are several approaches to create extensible formats to store an unified set of metadata together with experiment-specific metadata and experimental results. Most of them are joint efforts of scientific institutions and/or researching companies (Table 2).

JCAMP-DX is a format which can be applied for a wide range of analytical data. It was developed by the Joint Committee on Atomic and Molecular Physical Data (JCAMP) since 1988 [27] and is now maintained under the auspices of IUPAC. A general standard is proposed which can be used for different spectroscopic and spectrometric methods. Additionally, defined special standards for electron paramagnetic resonance (EPR) [13] and nuclear magnetic resonance (NMR) [28], chromatography and mass spectrometry [29] were published. Because the standard does not provide native support for ontology or controlled vocabulary use, and each implementation may use its own extensions, files from different sources can be incompatible. There is a Java reference implementation for this format available and there are libraries for other programming languages applicable like Python, R JavaScript and MATLAB. As JCAMP-DX is accepted as the exchange format for many analytical methods, there is a wide support by software for spectral analytics.

The XML-based Analytical Information Markup Language (AnIML) has been created to be an ASTM International standard and covers different analytical techniques. The standard comprises schema definitions for a generic core and technique-specific documents. Thus, it is possible to define technique documents for various analytical measurements. As AnIML is fully specified by its XML schemas, it can be effortlessly implemented in any language with XML support. No reference implementation is available, but among the (few) open-source implementations Jmol/JSmol (formerly JSpecView) can import and visualize AniML documents. For developers working with the python programming language, a library is under development to create, parse and validate AniML files [30]. There is also support by BSSN Software (now Merck) that promotes the format, often in combination with the device interface SiLA (Standardization in Lab Automation).

NetCDF is a binary file format and software interface, mainly defined by its implementations by the Unidata community [31]. It is an abstract model, which can be extended by self-describing objects. Thus, this model can be flexibly adopted to specific use cases. A family of ANDI (ANalytical Data Interchange) formats is specified by the ASTM, which are based on netCDF (see also ANDI-MS below).

The ISA (Investigation-Study-Assay) framework, originated from the bioscience community, defines the hierarchical ISA data model to store metadata on project context and study details, and analytical measurement data [32]. As the abstract ISA data model already encourages the user to annotate any parameter or value with ontology terms, it assures well described datasets. Implementations are available as tab separated value files (ISA-Tab) or as JSON (ISA-JSON). The ISA API is a Python library implementing the model for usage with the ISA formats [33]. The model is also applied for repositories such as MetaboLights. Moreover, journals such as ScientificData or GigaScience use the ISA data model to describe complex experimental setups covered in the manuscripts.

Conclusion

Data standards are key to attain interoperable and reusable data as these standards do not solely facilitate data analysis and long-term archival but also interactions between scientists by publications and direct exchange. Versatile, robust and widely adopted data standards require an ecosystem of well-defined specifications, data models, examples and implementations. Efforts on standardization were and are driven by numerous organizations, consortia and communities. For a few methods specific data standards exist. For other methods, standards still need to be developed or existing standards need to be extended to also include essential method specific metadata.