1.1 Statement of need

Researchers in academia and industry need to optimise control of physical systems to realise the potential savings mentioned above. For this purpose, they need a tool that allows them to easily implement a variety of highly customised controllers and easily apply them to models of physical systems. This tool must allow them to run co-simulations of the controllers and the physical systems with a multitude of different configurations and parameters. It should build on established tools and languages for modelling the physical systems without requiring users to focus on the representation of the physical systems. The tool should provide the results of these simulations in an easily processable format that allows the users to analyse and visualise the data in order to evaluate the different control approaches. In order for them to keep track of the different simulation runs and configurations, the tool should provide integrated methods for RDM that allow users to trace and to reproduce individual runs. The tool should provide a framework spanning the workflow described by the requirements defined above while allowing users to tie in existing tools for workflow steps not in the focus of their work. Additionally, this tool should be open source and not depend on proprietary dependencies. It should be possible to find and install the tool easily and incorporate it in common software setups.

Such a tool did not exist prior to the development of SOFIRpy to the best of the authors’ knowledge. Some related tools are discussed in Section 1.2.

Keeping the original use-case in mind but generalising the application, these requirements can be summarised in five workflow steps:

1. modelling physical systems

2. creating custom controllers

3. simulating controllers and physical systems with reciprocal dependencies

4. results analysis

5. data storage and RDM

Steps 2, 3, and 5 are directly addressed by SOFIRpy, while it allows a framework for building on existing tools to address steps 1 and 4 (cf. Section 2.2). Detailed requirements for the different steps are given in the following.

Users need to be able to easily create, configure and change models of the physical systems. They need a tool that can integrate representations of these systems and that provides simple interfaces for input of starting values and system configuration parameters. Users need to be able to run simulations of the physical systems without having to write their own solvers. Thus, they need to be able to build on existing tools for modelling and simulating physical systems.

Users need a tool that allows them as much freedom as possible to implement innovative control methods with potentially highly customised algorithms. They need the tool to allow them to implement the control methods using a widely disseminated and broadly used programming language. Users further need the framework of the tool to place minimal requirements on the implementations.

Users need to run co-simulations of the controllers and the physical systems. This means that there needs to be a defined interface for controllers and physical systems, through which inputs and outputs of each of the entities are exchanged at defined intervals during a simulation run.

Users need to run large parameter studies for studying and evaluating the performance of the control methods. Accordingly, they need to easily set up and configure, as well as parametrise simulation runs.

After running the simulations, users need the tool to allow them to process the results using other established tools compatible with the programming language of the tool itself. Users need the tool to store the results in a file using a non-proprietary, interoperable format. They further need to be able to supplement the stored results with metadata containing information on the setup and start values of the simulation that gave the results. This serves the joint purposes of keeping track of the results for the users of the tool and making the results more transparent and comprehensible for subsequent users of the data.

Finally, users need to be able to reproduce results easily or rerun simulations with parameter variations. For this purpose, they need to be able to load all the data required to recreate the simulation setup from the file containing the results in such a way that the simulation can be rerun.

1.2 State of the art

While there are no software tools that can address all the workflow steps and requirements outlined above according to the best of the authors’ knowledge, several tools exist for individual ones. Some of these tools are among the dependencies of SOFIRpy.

With regard to the first workflow step (modelling physical systems), SOFIRpy expands on OMPython, the OpenModelica Python API for building, compiling and solving Modelica models [5]. SOFIRpy uses OMPython in its dependencies for Modelica FMU exports.

FMPy is a library for simulating FMUs in Python, which also supports co-simulation and model exchange [6]. SOFIRpy uses FMPy to perform part of the third workflow step (simulating physical systems), but expands on it with the option of running co-simulations with the custom controllers written in Python. FMPy is included in the dependencies of SOFIRpy.

The HILO-MPC package offers a broad range of possibilities for modelling, estimation problems, and control based on optimisation, model predictive control (MPC) as well as machine learning [7]. HILO-MPC addresses the first three workflow steps of SOFIRpy. It requires users to explicitly formulate the equations that describe the physical system they are studying, whereas SOFIRpy allows users to use FMUs to represent the physical systems. Functionalities for RDM are not included in HILO-MPC.

There are two frameworks that address several of the workflow steps of SOFIRpy. ETA Utility provides a framework for optimisation, simulation and communication with physical devices [8]. Similarly to SOFIRpy, ETA Utility builds on FMPy to perform simulations. The focus of ETA Utility, however, lies on digital twins of factories and factory operation rather than on fluid and other physical systems. The FMUs in ETA Utility are used to run simulations in parallel to factory operations. For this reason, the framework focuses strongly on communication interfaces commonly used in manufacturing environments for connecting physical devices, which is beyond the scope of SOFIRpy.

The most extensive existing tool with similarities to SOFIRpy is Mosaik [9]. Mosaik is a framework for co-simulating many models by providing an interface to a variety of existing simulators for the purpose of performing smart grid simulations. Similarly to SOFIRpy, the framework organises information exchange between a variety of simulators for the purpose of co-simulation. In contrast to Mosaik, SOFIRpy focuses on FMUs and simulation of entities described in Modelica rather than providing interfaces to many different simulators. While Mosaik has also implemented an adapter to couple FMUs, this functionality is just one of many provided by the package. Both Mosaik and SOFIRpy allow for users to define custom models for controllers in Python. Mosaik offers comprehensive features for data handling with adapters to various databases, including not only InfluxDB and TimescaleDB, but also HDF5. SOFIRpy is limited to HDF5 for storing the results of simulated data. While Mosaik allows users to store metadata and connections related to the simulation results data, the focus of SOFIRpy on RDM and reproducibility remains unique to it. Mosaik is an extensive and feature-rich tool, whereas SOFIRpy is more lightweight. SOFIRpy can be easily adapted to match users’ requirements, suiting researchers to whom flexibility is more important than a large amount of features.

Several software packages perform partial functions of the steps enumerated above, yet SOFIRpy unites the functionalities in one overarching framework. Furthermore, none but one of the software packages mentioned above considers the aspect of RDM and no functionalities are provided for this vital step in scientific practice.

2.1 Design choices

SOFIRpy has been developed as a software framework based on the requirements presented in Section 1.1. In the development phase of SOFIRpy, several options for each of the functionalities were considered. These options, the selection made and the reasoning behind it will be briefly presented.

Coming from the original use-case, both a modelling and a simulation environment was needed for the fluid system as well as the multi-agent system for controlling it. For the fluid system, options for modelling and simulation available to the authors were: (a) using a modelling software, e.g., Dymola, OpenModelica or Matlab/Simulink, widely used in the community, (b) writing a custom model from scratch in Matlab or Python. Option (a) allows not only the possibility to work with a graphic user interface, significantly improving the user experience when modelling, but also the export of FMUs, a standardised format allowing interoperability with other software tools including the components of the SOFIRpy framework. Moreover, it allows creating FMUs of any physical system rather than only fluid systems. Consequently, a solution was chosen that reduces the need to write custom models for fluid systems.

The choice of programming language for the co-simulation of FMU and controller models was made in favour of Python due to its prevalence as a programming language, especially in fields that are interesting for innovative control methods such as machine learning, and because it is non-proprietary. Though the original use-case was for studying multi-agent control of fluid systems, the design of SOFIRpy was generalised to allow studying any type of control method. Accordingly, frameworks specific to agent-based modelling or control were not selected as a fixed choice. Instead, users need to write the custom models for the controllers themselves. This makes it possible to build implemented controllers on a wide range of other Python toolboxes, e.g., for machine learning.

Regarding data storage, there is a great variety of options. The approach chosen for SOFIRpy is to store the simulation setup including source code and serialised class instances of the used models and parameters together with the results of different simulation runs in one HDF5 file. The simulation results are also available as Pandas DataFrames [10]. Storing to HDF5 is performed using a wrapper around H5py, a Python package that offers easy-to-use methods for data storage and retrieval with HDF5 files. One advantage of this approach is that all relevant data and information about an experiment are stored in a single location. HDF5 files are versatile enough to store different data, such as the FMU binaries, the serialisations of the classes of the custom models written in Python and their source code as plain text as well as the simulation setup and the resulting simulation data. Furthermore, HDF5 files allow adding attributes to datasets and (sub-) groups, which is useful for including metadata for describing the simulation setup and runs. Metadata are not only useful for documentation purposes, but also for searching for specific simulation results using a set of parameters. The benefit of considering the RDM requirements in the fundamental design of SOFIRpy is that by providing easy-to-use, lightweight RDM features within the workflow software, users are motivated to apply them, ensuring improved scientific data as well as transparent and reproducible results.

Finally, it was decided to package and publish SOFIRpy as a Python package. This facilitates incorporating it in other software setups. Registering SOFIRpy on the package index PyPI also addresses the issues of findability, accessibility and easy installation.

2.2 Scope

SOFIRpy is aimed at researchers of fluid and other physical systems who want to analyse novel control approaches. The purpose of SOFIRpy is to provide a framework for co-simulating FMUs and custom-written models with integrated research data management.

Of the workflow steps enumerated in Section 1.1, step 1 (modelling physical systems) was not included in the scope of SOFIRpy, restricting the package to interactions with FMU representations of the systems. Step 2 (creating custom models for controllers) is the core of the work of the researchers, for which SOFIRpy offers a framework with as few restrictions as possible. SOFIRpy provides the methods for performing step 3 as a core functionality. For step 4 (results analysis), plenty of tools already exist. Therefore, this step lies in the responsibility of the user and is not entailed in the scope of the presented tool. SOFIRpy provides methods for performing step 5.

SOFIRpy provides three main functionalities:

• (i) Export Modelica models and necessary solvers as an FMU from the code-based workflow

• (ii) Co-simulate FMUs with custom written controllers in Python (workflow step 3)

• (iii) Store data and metadata of the simulation inside an HDF5 file (workflow step 5)

The first of the functionalities allows users to convert models written in the Modelica modelling language to FMUs. Exporting models to FMUs is also possible in the OpenModelica and Dymola GUI as well as many other software tools used in engineering practice. With SOFIRpy, however, this step can be integrated into the code-based workflow. This also allows users to set parameters for the models and exporting them to FMUs from within their scripts. Separate methods are provided for OpenModelica and Dymola. The export method for OpenModelica ensures that SOFIRpy is fully functional without dependence on proprietary software. Since Dymola is a commonly used tool for working with Modelica models, SOFIRpy additionally offers a method for FMU export from Dymola. This ensures SOFIRpy remains independent of commercial software while also providing full functionality for users of Dymola.

For the second functionality, each entity, be it an FMU or a custom written model, is represented as an instance of the same abstract class SimulationEntity. The data class System contains the data and the name of each simulation entity. Further data classes are defined for containing system parameters, logged parameters, connection points of each System and the actual connections between them. These connections are key for co-simulation, as they are used for defining the reciprocal influence of the controllers and FMUs. Multiple FMUs and custom models can be simulated together. The FMU class that inherits from SimulationEntity wraps the FMU2Slave class from the FMI 2.0 standards as implemented in the FMPy package. Finally, the Simulation class allows for a simulation run to be performed of all systems, considering their interdependencies. To handle time, a uniform step size is declared in the simulation setup and each SimulationEntity implements the method do_step() with the step size as input. In the co-simulation, each SimulationEntity progresses one time step, after which the outputs and inputs get passed along the defined connections. The Simulation class also provides a method for storing the simulation results in a Pandas DataFrame.

The third functionality concerns the RDM features. SOFIRpy wraps the package H5py to store the simulation results in HDF5 files. It also provides the option to store units of the time series in the attributes of the corresponding dataset in the HDF5 file. Beyond this basic data storage function, SOFIRpy provides further features for RDM. The HDF5 files used for storing results are structured on the highest level into two or more groups: one for the simulated entities, referred to as models, and one for each simulation run of these models stored in the file. In the models group, the binary files of the FMUs and the classes of custom controllers are stored along with the source code of the classes as plain text. The group for one run contains the resulting time series of a simulation run, as well as datasets with the data for the configuration of the run and software dependencies for the simulation run. By default, SOFIRpy stores a fixed set of important metadata in the attributes of each run group: the date and time, a description, a list of keywords, the operating system the run was performed on, the Python version, and the version number of SOFIRpy itself. The group containing the dataset with the resulting time series is supplemented with metadata for logging step size, simulation step size, and simulation run time. This is illustrated in Figure 1.

Figure 1: Structure of the HDF5 file.

Furthermore, each run group contains subgroups for each simulated model, both FMU and custom controller. These subgroups contain datasets with information on the connections of this model to other models, which parameters of the model are logged and the start values of the model for that particular run. Another dataset in the subgroup references the binary of the model stored in the models group on the highest level of the HDF5 file. In the case of the Python models, there are two datasets: one referencing the binary and the other the source code of the class as plain text. This structure is illustrated in Figure 2.

Figure 2: Structure of the HDF5 file and contents of the sub-groups of the /run group. Each class sub-group contains the same five datasets.

This structure and the stored information allows users to recreate the simulated models along with the run parameters and start values from the data stored in the HDF5 file. In this way, each simulation run can be fully reproduced from the stored data.

2.3 Limitations

The main functionality of SOFIRpy is the co-simulation of FMUs and custom written models. Accordingly, limitations mainly concern the FMU export and the RDM features. These will be discussed in the following.

The first limitation is that SOFIRpy does not implement its own solver for FMUs. This means that it depends on either open source solvers, such as CVode, or proprietary solvers, such as Dassl from Dymola, being available and exported with the FMU for the simulation to work.

There are several limitations regarding the RDM features. Firstly, the metadata stored with the HDF5 files by default are limited to a basic set of functional metadata with only the date and time as administrative metadata for context. This set of minimal default metadata is selected with a focus on recreating simulation runs. Keywords and descriptions are context-specific, and it is left to the discretion of the users whether to use terms from a standardized vocabulary or custom terms. Units can be stored in the attributes of the recorded time series of the run for each logged parameter. It is recommended to use standardised vocabularies such as QUDT Units [11]. Secondly, the metadata are not described using a standardised vocabulary. The metadata recorded in the HDF5 files primarily serve the purpose of recreating simulation runs. Since they are used within the scope of SOFIRpy, they generally do not need to be standardized for general use. However, for ensuring transparency on the provenance of data it is necessary to record further, especially administrative metadata. These would enable automated discovery and subsequent processing of the data, which is facilitated by using standardized vocabularies and ontologies, e.g., metadata4ing. Since the preferred standardized vocabularies are context-specific, it is left to the users which metadata are recorded and with which standardized vocabularies they are described. Instead, SOFIRpy is restricted to recording only the metadata necessary for recreating simulation runs in a manner easily processed by the software itself. Thirdly, the source code of the custom Python models is serialised for storage in the resulting datasets using cloudpickle [12]. Deserialising objects with cloudpickle requires the use of the same operating system used for serialising them. Deserialisation is only guaranteed for the exact same version of Python, though it has been shown to work when using different patch versions of Python. The latter issue can be addressed by using a tool for environment management, e.g., Poetry [13], VirtualEnvironment [14] or Anaconda [15] and providing the specifications of that environment. Furthermore, cloudpickle is not suitable for long-term object storage [12]. Since the source code of the classes is also stored, the code can be executed to recreate the simulation, though binaries of the classes may not be deserialisable in the long term.

3.1 fmu_export

Modelica models created with both OpenModelica and Dymola can be exported to FMUs. For each export, a separate class is implemented that inherits from a base class, FmuExport. Both classes define an export_fmu method that encapsulates the logic required for exporting a Modelica model to an FMU.

The OpenModelica export uses the OMPython package, which provides a method to do the export. In contrast, the Dymola export is more complex. In SOFIRpy, Dymola’s native scripting language is used. A script is dynamically generated to set the required parameters and export the Modelica model. This script is then executed using subprocess, a standard library in Python.

3.2 simulation

The UML diagram in Figure 3 provides an overview of the design for the simulation subpackage. At its core is the Simulator class. The class holds the instance attributes systems, connections and parameters_to_log.

Figure 3: Simulation UML diagram.

The systems attribute is a dictionary that maps the name of the system to an instance of the System class. The System class is designed to represent an entire physical system. An instance of the System class will hold an instance of the FMU class or the CustomPythonModel class, both of which inherit from the abstract base class SimulationEntity to ensure a standardised interface. The SimulationEntity class defines three abstract methods set_parameter, get_parameter_value, and do_step, which govern parameter manipulation, value retrieval, and simulation progression, respectively. The optional methods initialize, get_unit, and conclude_simulation can be implemented. These methods define how to set start values, retrieve units, and perform necessary steps after the simulation has finished.

The connections attribute contains information on how all simulation models are connected to each other and parameters_to_log defines which parameters/values should be logged.

The steps performed during a simulation are shown in Figure 4. Before starting the simulation, the models to be simulated are initialised by calling each model’s initialize method and all connections between the models are initialised. Then the simulation loop starts. In each loop, one simulation step is performed in each model. Subsequently, the inputs in each model are set to the appropriate value. Next, values that should be logged are logged and finally, the simulation time is increased by the step size. If the simulation time is greater than the stop time, the simulation is concluded and stops.

Figure 4: Simulation flowchart.

3.3 rdm

At the core of the rdm subpackage is the Run class. This class contains attributes for all data relevant to a simulation run and provides methods to manipulate the settings of the run. Additionally, the Run class provides the methods simulate, to_hdf5, and from_hdf5. The simulate method allows simulating the defined systems based on the current configuration. Upon completion of the simulation, the results and all associated metadata can be stored in an HDF5 file via the to_hdf5 method. This method stores the data in the format outlined in Section 2.2. Each data entry in the Run class is serialised using custom serialisation methods, ensuring that the data is properly converted into a format suitable for storage in the HDF5 file. The from_hdf5 method, in turn, reconstructs the Run class from a specified HDF5 file and run name, deserialising the stored data and enabling the simulation to be rerun using the previously saved configuration.

4.1 Getting Started

SOFIRpy is hosted on the package index PyPI and can be installed using pip with the following command:

  1        pip install sofirpy

The dependencies are specified in the file pyproject.toml [16] and will be resolved in installation using a dependency management software such as venv or poetry. The SOFIRpy documentation includes an introduction with installation instructions, minimal examples for all functionalities and full API documentation. The documentation is publicly available [17].

4.2 Minimal Example

The code examples below illustrate how to co-simulate an FMU and a custom controller, and how to utilise the RDM features in SOFIRpy.

In the first example, a PID controller is implemented as a custom controller for controlling a direct current (DC) motor of which there is already an FMU. (Cf. the example in the repository for the full implementation of the PID controller [18].) After the PID controller is implemented, the connections between the controller and the DC motor are defined, as well as the start values and the parameters to be logged during the simulation. Finally, the simulation run of the PID controller controlling the DC motor is performed using the defined setup.

  1    from sofirpy import simulate, SimulationEntity

  2

  3    # define fmus to be simulated

  4    fmu_paths = {"DC_Motor": "path/to/fmu"}

  5

  6    # define custom models to be simulated

  7    # here a custom PID controller is implemented

  8    class PID(SimulationEntity):

  9          """Simple implementation of a discrete pid controller"""

10          def do_step(self, time): … # To be implemented

11          def get_parameter_value(self, output_name): … # To be implemented

12          def set_parameter(self, parameter_name, parameter_value): … # To be implemented

13

14    model_classes = {"pid": PID}

15

16    # define how the models are connected

17    connections_config = {

18           "DC_Motor": [

19               {

20                   "parameter_name": "u",

21                   "connect_to_system": "pid",

22                   "connect_to_external_parameter": "u",

23               }

24           ],

25           "pid": [

26               {

27                   "parameter_name": "speed",

28                   "connect_to_system": "DC_Motor",

29                   "connect_to_external_parameter": "y",

30               }

31           ],

32    }

33

34    # define start values for models

35    start_values = {

36           "DC_Motor": {"inertia.J": 2},

37           "pid": {"K_p": 3, "K_i": 20, "K_d": 0.1},

38    }

39

40    # define which variables to log

41    parameters_to_log = {

42           "DC_Motor": ["MotorTorque.tau"],

43           "pid": ["u"],

44    }

45    results, units = simulate(

46           stop_time=10,

47           step_size=1e-3,

48           fmu_paths=fmu_paths,

49           model_classes=model_classes,

50           connections_config=connections_config,

51           start_values=start_values,

52           parameters_to_log=parameters_to_log,

53           logging_step_size=1e-3,

54           get_units=True,

55    )

The following minimal example of the RDM features shows how a simulation can be configured, run and stored in an HDF5 file. In a second step, the simulation run is loaded from the HDF5 file and a parameter is changed. After this, a second simulation is run with the changed parameter.

  1    from sofirpy import Run

  2

  3    run_name = "Run_1"

  4    model_classes = {"pid": PID}

  5    fmu_paths = {"DC_Motor": "path/to/fmu"}

  6    run = Run.from_config(

  7           run_name=run_name,

  8           stop_time=10,

  9           step_size=0.1,

10           fmu_paths=fmu_paths,

11           model_classes=model_classes,

12    )

13    run.simulate() # calls the simulate function with the defined config

14    # accessing the results

15    results = run.time_series

16

17    # storing the run inside the hdf5

18    hdf5_path = "path/to/hdf5"

19    run.to_hdf5(hdf5_path)

20

21    # loading the run from the hdf5

22    run_loaded = Run.from_hdf5(run_name, hdf5_path)

23

24    # manipulating the config of a run, e.g., changing the stop time

25    run_loaded.stop_time = 100

26    # simulating the run with changed config

27    run_loaded.simulate()

Further code examples demonstrating the main functionalities of SOFIRpy are provided in the repository [19]. While the examples for simulation and code features can be run using the files provided, it should be noted that the examples for FMU export depend on third party software to be installed on the system. In the case of the Dymola model export, this is proprietary software.

4.3 Ensuring Code Quality

Aside from being a tool for co-simulation, SOFIRpy is a demonstration example of how to write and document research software in the field of mechanical engineering for good scientific practice in RDM. Accordingly, great emphasis was placed on adhering to best practices in software development and ensuring comprehensive documentation. The used methods will be briefly presented.

4.4 Maintenance and How to Contribute

SOFIRpy is maintained in the publicly accessible GitHub repository. It is possible to submit issues and contribute to the project. Detailed guidelines on how to contribute are provided in the project’s documentation [17].