1.1 Motivation

The FAIR principles are formulated in generic terms and without reference to any particular scientific discipline [1]. According to Jacobsen et al. (2020), ”[t]his has likely contributed to [their] broad adoption […], because individual stakeholder communities can implement their own FAIR solutions. However, it has also resulted in inconsistent interpretations that carry the risk of leading to incompatible implementations. Thus, […] for true interoperability we need to support convergence in implementation choices that are widely accessible and (re)-usable.” [2]

Communities who wish to develop their own FAIR conventions may greatly benefit from successful real-world examples that allow researchers and practitioners to identify, evaluate, and select best practices for research data management (RDM). However, the application of RDM in practical applications is still lacking in literature. Many approaches and models exist in theory, but their value to the community remains low if their practice is not systematically examined.

This paper presents lessons learned from the creation and management of FAIR data and metadata in two recent robotics field research projects, RoBivaL and DeeperSense, conducted at the Robotics Innovation Center (RIC) of the German Research Institute for Artificial Intelligence (DFKI) [3] [4]. The paper is an extended and revised version of a presentation given at the NFDI4Ing Conference 2023 [5].

We chose to present RoBivaL and DeeperSense together for two complementary reasons. On the one hand, their commonalities allow us to generalize RDM principles to some degree in certain circumstances: Both projects are field studies, conducted over extended periods of time by diverse research teams from multiple institutions and disciplines. Their differences, on the other hand, allow us to examine the application of RDM principles in different research scenarios: RoBivaL is a terrestrial robotics project studying the performances of different hardware systems at a single location in the context of agriculture. DeeperSense is an underwater robotics project developing an AI for the translation of sonar outputs into camera-like images based on training data collected both at a laboratory and at multiple field locations.

Openly available datasets are crucial to advancing the field of robotics. One reason is that robotics research often requires access to specialized hardware, sensors, and environments. Making high-quality data accessible to more researchers, including those without the means to collect such data on their own, fosters innovation from a wider range of perspectives. Furthermore, open datasets provide common ground for the community to develop and benchmark algorithms collaboratively on standardized data.

Our discussion of RDM, however, is not supposed to be applicable just to robotics. In principle, our findings can be applied to any collaborative (field) research project employing humans and technical systems for data acquisition in multiple steps and iterations. The purpose of our study is to derive requirements and strategies for the creation and management of “rich” metadata in the FAIR sense.

1.2 Outline

Section 2 presents related work on FAIR RDM in general, on open data in robotics in particular, and on formal knowledge representation in robotics.

Section 3 features brief summaries of RoBivaL and DeeperSense. We present their overall project objectives and contrast their base data requirements from a high-level perspective.

The main body of the paper is divided into three parts (Sections 4 - 6). Specific desiderata and additional related work are introduced at the beginning of each part if necessary. The models and concepts presented in these parts were extracted from the experience in RoBivaL and DeeperSense and shall be applied in future projects.

The first main part (Section 4) discusses the content dimension of FAIR RDM. We distinguish between executive metadata necessary for producers to achieve their project goals, and reusable metadata necessary for reusers to satisfy the FAIR principles. We introduce this distinction into the FAIR data debate, because we believe that data producers are more likely to adopt FAIR principles if the specific needs of producers are taken into account by the FAIR community. Both metadata types are illustrated with examples from RoBivaL and DeeperSense. Further, this section introduces base elements for a model of the metadata creation process at the micro level in the context of collaborative metadata management. We relate our model to the ”processing step” class of the Metadata4Ing (M4I) ontology. Though M4I acknowledges the existence of metadata, it does not appear to address the process of metadata creation.

The second main part (Section 5) expands the distinction between different stakeholder groups from the previous section and explores the social dimension of collaborative FAIR RDM more broadly. We argue that a FAIR research data manager acts as a link between three social domains where they perform different primary tasks. We are not aware of a discussion about the social implications of FAIR RDM, but we believe such a discussion to be indispensable for a definition of the FAIR data manager role.

The third main part (Section 6) examines the time dimension of collaborative and iterative FAIR RDM at the macro level. Based on a critical appraisal of prominent data lifecycle models, we suggest a model of a self-improving data lifecycle geared towards collaborative and iterative RDM. We introduce a data provision phase which is necessary for internal collaboration. Further, we introduce an evaluation phase at the end of the lifecycle complementing the planning phase at its beginning, to foster iterative improvement of the data management system. Lastly, we recognize that planning and evaluation are different kinds of activities than data creation, provision, processing, publishing, etc., which gives rise to a lifecycle model with two nested loops. Our model is illustrated with lessons learned from RoBivaL and DeeperSense.

3.1 RoBivaL

The project RoBivaL [3] [27] was conducted between August 2021 and October 2023 by an interdisciplinary and multi-institutional team of roboticists and agriculture researchers in Germany. The project compared different robot locomotion concepts both from space research and agricultural applications on the basis of experiments conducted under agricultural conditions. The goal was to promote knowledge and technology transfer between space and agriculture research. While the experiment designs were inspired by the standards ISO 18646-1 [28] and ISO 18646-2 [29], the environmental properties were adapted to the agricultural context, and the main evaluation focus was on soil interaction. Four robots were used: Two having their origins in space applications, the other two developed for agriculture. The robots were subjected to six experiments addressing different agricultural challenges and requirements. Soil conditions were controlled and varied on the two dimensions moisture (dry, moist, wet) and density (tilled, compacted). Figure 1 gives an impression of selected experiments and robots in the field.

Figure 1: Selected RoBivaL field experiments and robots: (a) Turn around with SherpaTT, (b) Straight travel with Naio Oz, (c) Sill crossing with ARTEMIS. © DFKI, Malte Wirkus. License: CC BY 4.0 International

Field conditions and robot behavior were monitored with various sensors and measuring devices, partly on the robots and partly in the field, in order to document the experiment execution and to determine the robot performance. The data capturing devices, their roles and deployments are summarized in Table 1. (Video camera and Lidar on the system are greyed out, because, although available, they were not used in the project.) The entire dataset including comprehensive metadata is publicly available on the Zenodo platform [30].

3.2 DeeperSense

The project DeeperSense [4] [31] was conducted between January 2021 and December 2023 by an international, interdisciplinary, and multi-institutional team of researchers and domain experts in Germany, Spain, and Israel. This paper focuses on the German use case, which employed roboticists, sensor experts, and technical divers. The objective was to improve the safety of the divers, who work under dangerous conditions and therefore require constant monitoring and assistance. Existing safety systems rely on cameras, which is a problem in turbid water that limits visibility – just when the divers most need outside support. Sonars are more robust to turbidity, but conventional sonar output is difficult to interpret. DeeperSense therefore developed a neural network which translates sonar output into images that appear camera-like, thus combining the best aspects of both modalities. Figure 2 illustrates the sonar-to-image translation.

Figure 2: DeeperSense sonar-to-camera translation. The pair (a) and (b) is a training input sample. The generated image (c) is predicted in production from (b) alone. © DFKI, Bilal Wehbe. License: CC BY 4.0 International

To gather training data, divers performing typical work tasks were recorded underwater with sonar and camera simultaneously. Figure 3 shows the training data collection setup schematically.

Figure 3: DeeperSense training data collection setup. © DFKI, Bilal Wehbe. License: CC BY 4.0 International

For the neural network to be able to handle different types and degrees of turbidity, the training data had to be varied accordingly. Since this is difficult to establish and control efficiently at a single time and location, data was captured during six sessions at four different locations, covering inside and outside conditions, natural and artificial water bodies, and different seasons. Figure 4 gives an impression of the field locations.

Figure 4: DeeperSense field locations. From left to right: Martime Exploration Hall, Bremen; Chalk Lake, Hemmoor; Tank Wash Basin, Neu-Ulm; Starnberg Lake, Percha. © DFKI, Bilal Wehbe / Christian Backe. License: CC BY 4.0 International

Selected parts of the sensor data were published on the Zenodo platform [32] [33]. Due to the size of the sensor data, it is currently impractical to make the entire corpus available online. Instead, the metadata was published as a standalone database [34], allowing researchers to select portions relevant for their use cases, which are made available on demand. This is an effort to comply with the FAIR principle A2 to make metadata accessible independently of the base data.

3.3 Comparison

Table 2 summarizes and constrasts the data-related properties of RoBival and DeeperSense as presented in Sections 3.1 and 3.2. An immediate takeaway is that the scope and form of the base data, its purpose and handling can be quite different between projects even at a single institute. A data management solution should be flexible enough to accommodate such variance.

4.1 Metadata purposes, semantics, and richness

Virtually every general metadata definition starts with the assertion that metadata is ”data about data” [35], [36], [37], [38], [39]. A common purpose-based classification distinguishes at a high level between descriptive, administrative, and structural metadata [40], [41], [42]: Descriptive metadata ”enables discovery, identification, and selection of resources”, administrative metadata ”facilitates the management of resources”, structural metadata ”describes relationships among various parts of a resource”, and is ”generally used in machine processing” [42].

There are many domain-specific approaches to model metadata semantics. For the communication of metadata in engineering disciplines including robotics, the NFDI4Ing community has developed the Metadata4Ing (M4I) ontology [43], [44], [45]. It features a generalized process model, centered around the ”processing step” class. This is an attempt at communicating multi-stage data processing to satisfy the FAIR principle R1.2 of detailed provenance tracking. We compare the M4I processing step class with our own metadata creation process model in Section 4.3.

The FAIR principles [1] (see Table 3) require metadata to include the identifier of the base data (principle F3) and to be independently accessible (A2). Special emphasis is put on “rich” metadata. The term is associated with findability (F2), but defined in the context of reusability (R1). In fact, from the formulation of R1 it appears that rich metadata is the essence of reusability. Its definition is left vague, which is likely intentional to allow the concept to be applied in various domains. Richness implies ”a plurality of accurate and relevant attributes”. The only specific attributes mentioned are a data usage license (R1.1) and provenance (R1.2). Further attributes must ”meet domain-relevant community standards” (R1.3). In our view, this means it is both possible and necessary to develop community-specific interpretations of metadata richness. The focus on reusability implies that richness must be explained from a user perspective.

4.2 Executive metadata and reusable metadata

While the FAIR principles promote the development of metadata for data reusers, data producers already create and manage metadata routinely for their own purposes. This does not imply that they would describe their own practice in these terms or use specific tools and methods. It means that some forms of metadata creation and management are just an innate part of being an effective researcher. Examples will be given below. What is the relationship between the executive metadata necessary for data production and the rich FAIR metadata supporting, enabling, or facilitating data reuse? This question has a content aspect and a form aspect: Which metadata topics are relevant for producers or reusers? And which formal requirements are demanded by either group? Since these questions address two different stakeholders, they foreshadow the discussion of social aspects of FAIR RDM in Section 5.

Table 4 presents the metadata topics of RoBivaL and DeeperSense categorized by project and by relevance for producers or reusers. Intersections are possible on both dimensions. Each topic is labeled with its dominant purpose(s), i.e., descriptive (D), administrative (A), or structural (S).

The assignment of topics to producers or reusers is guided by the assumption that either group has a different primary motive: Producers want a correct execution of their project plan to achieve their primary research goal. Reusers want a sufficient understanding of the base data to assess its utility, and to integrate it into their own work flow. Our assumption about producers is primarily based on our personal experience, i.e., they reflect the motives and requirements prevalent in the two examined projects and within our institutions more generally. We believe that these assumptions are neither surprising nor uncommon. The point here is to observe the contrast between producers and reusers. Our assumption about reusers is based on our interpretation of the FAIR principles. Both assumptions are further substantiated in Section 5.

The different motives also affect the formal requirements. Data producers care less if all metadata is specified and captured explicitely and formally, but tolerate tacit expert knowledge, code logic, informal communication, etc. For the sake of efficiency and expediency, they may limit content and form of metadata to what is essential to their needs. Reusers on the other hand require all metadata to be explicit, since they lack the immediate access to the creation context that producers have. To support efficient machine processing, metadata must be formalized. In order to cover a broad range of possible reuse cases, it must be rich in the FAIR sense.

4.3 Base elements of the metadata creation process

This section analyzes the process of metadata creation and derives some process-related metadata categories. The matter is treated abstractly and at the micro level, i.e., with regard to individual data elements; the big picture of the data lifecycle is discussed in Section 6. The analysis yields elements for the design of metadata production workflows. This is useful in a collaborative setting with a division of labor, where responsibilities must be communicated effectively.

The data flow diagram in Figure 5 illustrates the first order of metadata creation on a single data processing stage. Base data processing is represented vertically from top to bottom, metadata processing horizontally from left to right. The Output represents a piece of base data which is generated by some Procedure. Output and Procedure are the subjects of metadata. For both, metadata creation has two phases: Before the subject exists, it is designed; after it exists, it may be documented. The design is metadata that is injected into the Procedure; the documentation is metadata that is extracted either from the Procedure or from the Output.

Figure 5: Data flow on a single stage of first order metadata creation. Rounded box: Process. Angled box: (Meta)data. Blue: Data. Green: Metadata. © DFKI, Christian Backe. License: CC BY 4.0 International

The entire model can be stacked vertically to represent multi-stage data transformation, i.e., the Procedure may receive output of a previous stage as its input, the Output may serve as input to another procedure on a subsequent stage. This model facilitates division of labor by modularizing metadata both in the content domain (distinguishing metadata subjects Procedure and Output) and across time (distinguishing design and documentation phase).

Table 5 lists examples for each of the four first order metadata categories taken from the DeeperSense project. They are related to the same Procedure (”Capture camera and sonar images of a diver”) and corresponding Output (”Logfiles with raw camera and sonar data”).

The assertion that metadata is data, as mentionend in Section 4.1, implies that metadata creation may be recursive: Higher orders of metadata can treat metadata of lower orders as their base data. Visually, this means we can stack the first order metadata creation model not just vertically, but also horizontally. This is illustrated in Figure 6. It contains a condensed version of Figure 5: The process ”Create MD” represents all four metadata creation processes of the first order, which are applied to the base Procedure and Output. ”First order MD” represents all four first order metadata types. The recursion recognizes that ”Create MD” and ”First order MD” themselves are a procedure-and-output pair, hence they become subjects of meta-metadata creation.

Figure 6: Recursive creation of higher order metadata. © DFKI, Christian Backe. License: CC BY 4.0 International

Table 6 gives two sets of generic examples for higher order metadata on multiple levels. The first example features pieces of literal base data and metadata: A speed measurement is taken at a certain time; the time stamp formatting is expressed in C string format notation; syntax and semantics of this formatting are governed by an ISO standard. The second example has a similar application pattern, but references files, which support structured data and semantic networking.

How does our metadata process model compare to the processing step class of the Metadata4Ing (M4I) ontology depicted in Figure 7? The M4I model acknowledges that each data output is generated by a process, and that data processing may be chained, which corresponds to the vertical direction of our model. But the M4I model does not appear to cover the process of metadata creation itself, i.e., our model’s horizontal direction (injected vs. extracted metadata, higher order metadata). We assume this absence is at least partly a result of the purpose of the M4I model, which is communication of metadata to data consumers after the base data and metadata have been created. As mentioned above, the modularization of metadata in our model serves to design workflows for metadata creation by a data production team during project execution. A further difference between the M4I processing step and our model is that the former specifies a fixed set of attributes, while the latter is agnostic in this regard. Finally, the M4I processing step model provides the opportunity to encapsulate multiple substeps into a single step of larger scale. So far, our model does not feature a similar means of abstraction.

Figure 7: Processing step class of the Metadata4Ing ontology [43]. © Metadata4Ing Workgroup. License: CC BY 4.0 International

5.1 Collaboration with the data production team

The first social domain is the data production team. Here, the primary task of any data manager (irrespective of FAIRness considerations) is collaboration.

Collaborative research in general is challenging, because it involves a multitude of people who must be coordinated and accomodated. If they come from different disciplines and institutions, they may have different motivations, goals, expertise, responsibilities, standards, and practices. These individual attributes may not be equally transparent for everybody, and not be equally present in everyone’s mind, which can complicate intra-group communication.

Collaborative field research is particularly challenging: The pressure to perform is very high, because there are limited opportunities to go into the field; field conditions can be difficult and unpredictable, which often leads to unforeseen problems; equipment and people are put to unusual stress. The main priority is to get all people and systems to work at all at the designated time and place, and to capture the primary data that serves the project goal. This often requires improvisation and adaptation, because prototype systems may break or deviate from specification, and the captured data may not match earlier expectations. Figure 8 illustrates these notions in the context of DeeperSense.

Figure 8: Field team work in DeeperSense. On the last day of data collection, the team is on a boat on a lake, gathering a critical piece of data necessary for the final demonstration event. The underwater system keeps failing. Error messages on the computer screens are difficult to read due to the glaring sun. © DFKI, Christian Backe. License: CC BY 4.0 International

This assessment has two immediate implications for effective RDM in field research: First, RDM must be reliable and unobtrusive. A field research team wants their RDM to ease the effort, not stand in the way or cause extra concerns. Second, RDM must capture unforeseen events, so they can be factored into the preparation of future field missions.

5.2 Mediation between producers and reusers

The previous section dealt with RDM in general. For a FAIR data manager in particular, there is a second social domain, namely the larger research domain. Here, their primary task is mediation between conflicting requirements of their data production team on the one hand, and potential data reusers (as characterized by the FAIR principles) on the other.

We hinted at this conflict in Section 4.2 and can fully express it in light of Section 5.1: Reusers require explicit, formal, rich metadata to thoroughly understand the data that is foreign to them, easily interface with it using machines, and have it serve a broad spectrum of potential use cases. But this demands extra effort from the producers, who not only have the privilege of being more implicit, informal, and brief in their internal communication, but who may actually be forced to cut corners, especially under field conditions, in order to reach their primary research goal.

Table 7 summarizes this proposition and adds two aspects derived from experience in RoBivaL and DeeperSense: In a collaborative setting with division of labor, executive metadata may be distributed over many places convient for different contributors; to become reusable, it must be consolidated. While research is ongoing, the executive metadata design may need to evolve to adapt to changing circumstances; reusers prefer reliable APIs.

The conflicting priorities and requirements of data producers and reusers have two implications for a FAIR research data manager: First, they must motivate their team to apply the extra effort. One possible incentive may be that today’s producers are their own reusers tomorrow, so the investment in more elaborate metadata will pay off directly towards themselves. There is an indirect version of this: By creating metadata they would be happy to receive if they were reusers, producers influence the standards of their community to their own benefit. Another incentive may be increased impact of their research if the underlying data is broadly adopted in the community. A second implication is that FAIR research data managers must design the workflow of their team such that the extra effort necessary to satisfy reuser requirements does not coincide with peak effort towards the primary research goal, because the latter will always have precedence.

5.3 Standardization in the FAIR RDM community

The third social domain for a FAIR research data manager is the FAIR RDM community. Here, their primary task is to participate in the standardization of FAIR practices in a particular research domain and maybe across domains. We believe the outcome of this activity can be conceptualized as higher order metadata.

From a purely theoretical perspective, the metadata recursion could go on to unlimited orders. But in practice, of course, a cut-off is made, from which on the participants (i.e., data producers either among themselves or in relation to reusers) regard all higher metadata orders as common knowledge to be infered from context or prior convention. Still, the communication relies in principle on the assumption that all higher metadata orders could be delivered explicitly. One core role of the FAIR RDM community is to underwrite this assumption, i.e., to work towards a codification of common knowledge (including standards and open vocabularies) to which all participants can refer in their communication.

6.1 Model of a self-improving data lifecycle

There is no consensus which phases constitute a data lifecycle and how the phases shall be ordered. In their survey of 76 data lifecycles, Shah et al. identify at least 14 phases [46]. NFDI4Ing uses a model with six phases, named Planning, Production, Analysis, Storage, Access, and Re-Use [47]. It is similar to other six-phase models prevalent in the FAIR RDM community [48], [49], [50] but there are still differences about the naming and ordering of the phases. These models have two shortcomings regarding their application to collaborative and iterative research.

First, while there is a phase in these models near the end of the cycle for making data available externally to the public (called “Publication”, “Access”, “Sharing”, or “Disclosure”), there is no equivalent phase dedicated to making the data available internally to the research team immediately after creation. In our experience, such a phase is necessary in collaborative research, and it has different requirements than the publication phase. We propose to call it Provision.

Second, almost all phases are actions that apply to data (data is produced, analyzed, …), except for Planning which is the only phase that applies to other actions (production is planned, analysis is planned, …). Another oddity about Planning is that it has no corresponding phase for looking into the past. In iterative research, comparing how things were planned to how they turned out would enable an iterative improvement of the RDM system. Since our research is in fact iterative, such a self-improving data lifecycle would be welcome. Therefore, we propose two additional phases called Execution and Evaluation. Together with Provision, they apply to each data-related action and thus form a separate loop nested with the data-related loop.

In summary, our proposed model has six data-related phases: Creation, Provision, Processing, Publication, Reuse, and Archiving. Each of these is divided into three process-related phases: Planning, Execution, and Evaluation. The model is illustrated in Figure 9.

Figure 9: Self-improving nested data lifecycle. © DFKI, Christian Backe. License: CC BY 4.0 International

The term Creation is chosen over Collection or Acquisition to emphasize the designed and fabricated nature of data and metadata. The broader term Processing is preferable over the narrow term Analysis, because we encounter a range of data processing activities in our practice, both primary (e.g. machine learning model development is synthesis, rather than analysis) and secondary (e.g., data cleaning, fusion, performance tuning, or quality assurance). The term Planning is to include Preparation. A single word is used here for brevity, but one should be aware that it does not signify purely cerebral activity, but also, e.g., handling of hardware.

6.2 Lessons learned from RoBivaL and DeeperSense

This section serves to illustrate the data lifecycle model discussed in Section 6.1 by presenting lessons learned in the different lifecycle phases of RoBivaL and DeeperSense. Many of the lessons are derived from failures, either to perform a task or to anticipate a challenge. Due to space constraints, we focus on consequences and leave specifics of the failures mostly implicit. Methods and strategies from Sections 4 and 5 are addressed where appropriate.