Skip to main content

Data Guide

This page contains requirements and recommendations related to the content of a dataset. It shall help DataSubmitters to prepare their data and metadata according to the FAIR principles.

Aspects covered are:

  • Metadata
  • Files
    • File format
    • File description, file path (folder structure)
    • File size and number of files, ZIP files
    • Readme file

Quick summary​

  • Restrictions and limitations:
    • No sensitive data, cf here.
    • Number of files: Too many files should be avoided, thus recommendation to collect many small files in ZIP files.
    • Large data amounts: Please contact Edmond support team aforehand.
  • File formats:
    • Avoid proprietary file formats, use open file formats.
    • Use standard file formats which are suitable for long-term preservation.
  • File names:
    • Should only contain letters, digits, underscore, hyphen, and one dot before the file ending
    • Keep file names short.
  • Metadata
    • Authors: Provide ORCID and affiliation with ROR-ID.
    • Title: Should be concice and not identical to the title of a related paper.
    • Description: Shall provide an overview of the dataset. Who produced what kind of data, when and where?
    • Give references to related papers and datasets.
  • Provide Readme file with further details.

Metadata​

Please fill all the relevant metadata as complete as possible. Metadata fields marked with a red asterisk are mandatory, e.g. the field "Title".

For the description of the metadata fields see here. For some of them, further notes are given in the following subsections.

Licence​

It is important to choose the right licence for a dataset if you want others to be able to reuse it. The licence informs potential reusers about how they are allowed to deal with the data. Edmond supports several licences. In most cases, we recommend using CC0, which is a very open licence. See here for more information.

Author​

  • Field Name:
    Should be filled in format "family name, given name", e.g. Planck, Max. During typing, proposals are made based on an ORCID lookup mechanism. If you find the name with the correct ORCID, you can choose it. In that case, both fields, "name" and "ORCID", are taken from the ORCID profile.
  • Field ORCID:
    As just mentioned, this field is filled automatically when choosing one of the proposals at field "Name". Otherwise you can add it manually - in that case, please add the full link, e.g. https://orcid.org/0000-0001-1234-5678. We strongly recommend to provide the ORCID: This avoids ambiguities (different persons with same name, but also different spellings of the same person, e.g. "Planck, Max" versus "M. Planck"). In the ORCID profile, further information about the author can be given like contact email, affiliations and publications.
  • Fields Organization, ROR ID and Department/Group
    Field "Organization" shall contain the name of the organization of the author, and field "ROR ID" the ROR ID of that organization. Similar to authorʼs name and ORCID, there is a lookup functionality, making it easy to provide the ROR ID of the institute. In particular for Max Planck Institutes, please stick to the proposed name, e.g. use the proposed name Max Planck Institute for Chemistry instead of alternatives like "MPI for Chemistry" or "Max-Planck-Institut für Chemie". Optionally you can add the name of the department or group in field "Department/Group", e.g. Multiphase Chemistry.
    Fields 2. Organization, 2. ROR ID and 2. Department/Group allow for providing a second affiliation.
    Fields 3. Organization, 3. ROR ID and 3. Department/Group allow for providing a third affiliation.

Description​

The Description field shall describe the dataset and its content.

Structure, format and general notes​

  • The description has to be written in English.
  • The first part of the description should be a clear and concise overview of the datasat, similar to the abstract of a paper. It should be written in general terms (avoid jargon unless necessary) so that it is also understandable to researchers in other fields as well.
  • Afterwards, more detailed explanations should follow (see below).
  • Acronyms, abbreviations and technical terms shall be explained, potentially providing links to further information.
  • Currently, Edmond allows for some html formatting of the description field, e.g. <b>This text is bold.</b>. However, we recommend to avoid this option at least in the first part and to use it only modestly, because:
    • Metadata crawler and search engines usually do not interprete these html commands. For example: Dataset doi:10.17617/3.ES5ZSC and its description shown at DataCite Commons
    • We cannot guarantee that Edmond will support these feature on a long-term bases, because the rendering of the html formatting is depending on the software we use for Edmond.
  • For the length we recommend around 100 to 1000 words. More details should be provided in a Readme file.
  • Even in case the dataset is considered as supplementary data for a paper, the dataset abstract shall be independent of the paper abstract: While the paper abstract focuses on the study and the interpretation of the outcome, the dataset description shall focus on the files in the dataset.

Information to include in the Description​

Following some information which should be included in the description. Some of them may not be relevant for all datasets or disciplines, and some of them (in particular if very detailed) may be better included in the Readme file.

  • What: What does the dataset contain? Which parameters (and units)?
    E.g.: "Electron microscopy raw data", "Time series of temperature in °C"
  • Where and when: Coordinates and time of the experiment - relevant in particular in case of field experiments.
  • Lab: In which lab, under which conditions and settings did the experiment take place?
  • Method: Description of the measurement and processing methods.
  • Workflow: Description of the data processing and analysis steps.
  • Who: Who had which contribution, who owns the data or could be contacted for further questions?

Files​

File format​

Criteria​

For the sake of interoperability and reusability, suitable file formats have to be chosen, which can be read by several free software products, and from which one can assume, that also in future, software will be available to read. Criteria for the suitability for long-term archiving are:

  • Simplicity:
    The simpler the format, the easier it is to interprete it, and to write and maintain software which is able to read it.
  • Documentation:
    Formats without an opently accessible documentation should be avoided.
  • Storage consumption:
    In particular for large data volumes, the storage density becomes relevant. For example, a simple tab-separated ASCII table is smaller than an xml file with same tabular information. Compressing a file (e.g. via ZIP) can significantly reduce the file size, in particular in case of text or numerical data.
  • Open:
    Whenever possible, open file formats shall be taken instead or proprietary formats.
  • Widespread usage and software availability:
    The more free software products for reading the file format are available, the higher the change that the data files can be read also in 10 years and beyond. Also a widespread usage of the format is an indicator for its long-term usability.
  • Where appropriate, we recommend ASCII or UTF-8 encoded files. These include unstructured text, tables (e.g. tab- or comma-separated ".csv"), as well as structured or semi-structured text (e.g. Markdown, HTML, XHTML, XML, SVG, JSON). Those files can be read with any text editor and are thus well-suited for longterm archiving.
  • As container format for collecting several or many individual files within a larger file, we recommend ZIP, because ZIP is well-established and open. Furthermore, the current Edmond implementation supports ZIP preview, i.e. the files inside the ZIP file can be listed (and downloaded) without the need to download the full ZIP file beforehand.
  • For pixel graphics, PNG (Portable Network Graphics), TIFF (Tagged Image File Format) and JPEG (file ending typically ".jpg") are widely used.
  • Many further binary formats exist. Some of them can only be read by a limited range of software, or are used by only specific research communities. Such formats should be avoided or only used in case they fulfill the abovementioned criteria and provide a significant advantages compared to text-based formats.

File names and path - structure​

As described here, Edmond supports file paths. Thus, the files of a dataset can be grouped in folders and subfolders. Following some recommendations for file and folder names:

  • Keep folder and file names short.
  • Avoid deep nesting, i.e. paths like folder/subfolder/subfolder/subfolder/subfolder/file.txt.
  • Avoid special characters. We recommended to restrict to latin letters, digits, underscore, minus, and dot, i.e. ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890_-..
  • The dot should only be used once, namely before the file ending, e.g. file.txt.

File size and number of files​

There are no explicit file size or dataset size limits in Edmond. However, when planning to store more than about 1 Terabyte (TB) of data, please contact our Edmond support team beforehand.

Background - note about costs and resources:
Publishing large volumes of data in a Core-Trust-Sealed long-term repository like Edmond causes substantial costs for the MPG. For example, providers like RADAR or DRYAD calculate with some thousand Euro per TB. Reason for such high costs is the requirement of backup strategies with redundant copies, regular replacements of old hardware by new one, and this for many years. Therefore, such large volumes should only be published in case one expects that the data will be downloaded for re-use or verification. In case storing, backup or archiving data on a save place is sufficient, services like those of MPCDF and the GWDG may be preferred.

Furthermore, please note that uploading large amounts of files via the browser can be time-consuming depending on the infrastructure. Based on our experience, we recommend not uploading hundreds of individual files to a dataset. Therefore, we have set an upper limit of 500 files per dataset. In cases where you have many files, we recommend bundling this collection of individual files in a ZIP and then uploading them. Using the ZIP viewer, it is still possible to view the individual files online immediately and download them individually if required.

For uploading many or large files, we recommend to use an upload tool, i.e. the command line client DVUploader or the desktop tool UpVerse.

Please contact our Edmond support team if you have any questions.

Readme file​

In the context of Edmond, a Readme file is considered to be a file describing other files in the dataset, and/or the dataset as a whole.

Description field versus Readme file​

While some information fit in both, the dataset description field and the Readme file, there are some differences to consider:

  • While the dataset description should not be too long, the Readme file can be quite detailed.
  • From a scientific perspective, both contain metadata because they describe data. But from a technical perspective, the description field is a metadata field, while the Readme file is a data file like the other files of the dataset.
    • When changing the description of a published dataset with version 1.0, the new dataset version will be 1.1. But when changing any file, the new dataset version will be 2.0.
    • See here for the difference between metadata fields and files with respect to their persistence.

General recommendations about the Readme file:​

  • Where: For most cases, we recommend to have one top-level Readme file per dataset.
    • That file should be on top-level, i.e. not within a folder, for making it easy to find.
    • It should also not be within a ZIP file, because:
      • The preview function for viewing a text-based files does not work if that file is inside a ZIP file.
      • In case there is a need for updating the Readme file of a published dataset, then a new dataset version has to be published. Because Readme files are typically small compared to the primary data files, storing 2 versions of a Readme file is not a big issue. But if the Readme file would be part of a large ZIP file, then two large ZIP files would need to be stored.
  • Name: The file name is usually "README" plus file ending, e.g. "README.txt" or "README.md".
  • Format: The Readme file should be a UTF-8 file (in best case even ASCII).
    This can be a simple plaintext file (then file ending ".txt" is recommended), a Markdown file (".md"), or a HTML or XHTML file (".html", ".xhtml").