FINAM -- Introduction

This book has moved to the FINAM documentation. You are viewing an outdated version!

FINAM is an open-source component-based model coupling framework for environmental models. It aims at enabling bi-directional online couplings of models for different compartments like geo-, hydro-, pedo- and biosphere.

The framework is built in Python, with well-defined interfaces for data exchange. This approach allows for coupling of models irrespective of their internal structure, architecture or programming language.

Chapters

  • About this book -- The purpose of this book, and how to read it.
  • FINAM principles -- Explains basic principles of the FINAM framework that are of interest for users as well as developers.
  • Using FINAM -- Guide for users that aim at coupling existing models.
  • Developing for FINAM -- Guide for developers on how to prepare models for FINAM, and how to implement adapters.

External resources

About this book

This book has moved to the FINAM documentation. You are viewing an outdated version!

This book has multiple purposes:

  • FINAM principles: Explain the basic principles of FINAM to the interested audience
  • Using FINAM: Teach end users how to set up and run FINAM compositions
  • Developing for FINAM: Teach developers how to implement FINAM modules, and how to wrap existing models for coupling

All except the chapters on principles require some Python programming skills. We do not teach Python in this book, so different levels of programming knowledge are assumed for certain chapters. Read on for details...

Requirements

The first chapters under FINAM principles are crucial for understanding how FINAM works. They contain only textual and visual descriptions. They contain no code and require no programming knowledge. Read this first!

The chapters under Using FINAM are dedicated to users that want to set up FINAM compositions using existing modules. Compositions are created through Python scripts. These chapters require some basic Python knowledge for writing simple scripts.

The chapters under Developing for FINAM are dedicated to developers that want to build modules for FINAM, or wrap existing models or libraries for the use in FINAM compositions. To write modules following these chapters, intermediate knowledge of Python is required. Particularly, developers need some basic understanding of object-oriented programming in Python.

FINAM principles

This book has moved to the FINAM documentation. You are viewing an outdated version!

This chapter describes:

Components and Adapters

This book has moved to the FINAM documentation. You are viewing an outdated version!

Models are wrapped according to the interfaces into independent components. Communication works over so-called adapters that are plugged between models or other components. These adapters are responsible for transforming data between one model's output and the receiving model's expected input. Examples are spatio-temporal rescaling or coordinate transforms.

FINAM enables setting up a consistent and flexible data stream workflow involving models and drivers, as well as data pre- and post-processing (e.g. visualization).

Components

Components are the primary entities that are coupled in FINAM.

A component can have multiple inputs and outputs to exchange data with other components.

Component

Figure 1: A FINAM component

There are two principle types of components:

Components with a time step are executed when all their inputs are available, according to chapter Coupling and Scheduling. Typically, these components represent simulation models.

Components without time step can work in a push-based or pull-based manner. They are executed each time new data becomes available to an input, or when a pull of their outputs is attempted, respectively. Typical examples are statistical models or file I/O components.

For a list of available components, see chapter Known components and adapters.

See chapter Writing components for how to implement components.

Adapters

Adapters serve for data transformations between coupled components. Thus, components are not required to have their inputs and outputs in the same grid specification, CRS, etc. as other, potentially coupled components.

An adapter has one input and one output.

Adapter

Figure 2: A FINAM adapter

Adapters manage tasks like:

  • Spatial up-/downscaling
  • Geographic re-projections
  • Temporal interpolation and integration
  • ...

For a list of available adapters, see chapter Known components and adapters.

See chapter Writing adapters for how to implement adapters.

Coupling and Scheduling

This book has moved to the FINAM documentation. You are viewing an outdated version!

Coupling

A coupling setup in FINAM is a (potentially cyclic) graph of connected components and adapters. Figure 3 shows an example.

Coupling

Figure 3: A FINAM coupling

See chapter Model coupling scripts for how to set up coupled models.

The figure illustrates a few important aspects of coupling:

  • Cyclic/bi-directional linkage is possible
  • Adapters can be chained
  • Components can be linked directly when no adapter is required

Scheduling

The scheduling of components follows a simple algorithm that allows for arbitrary, and even variable, model/component time steps.

It is simply the component most back in time that is updated next. This way, it is guaranteed that all required input data is available in the outputs of the other components.

The example in Figure 4 illustrates the approach.

Finam scheduling

Figure 4: FINAM scheduling (see text)

Figure 4 shows a snapshot of a simulation featuring three components A, B and C with different time steps. Solid lines and dots denote already simulated model steps, and the right-most solid dot of each component shows it's current simulation time.

According to the above rule, component A is most back in time and needs to be updated. After the update (dashed line and first hollow dot), component A is still on turn. After the second update of A (dotted line and second hollow dot), it has caught up with B. Then, B would be updated.

As illustrated by the curly braces, it is guaranteed that input data for A is available on the first update. Any kind of interpolation between adjacent source component time steps can be applied to derive the input date. This is one responsibility of adapters.

Particularly for components with large time steps, it is also possible to integrate over multiple source component time steps. E.g., component C could use the weighted average of the several steps A would have performed when C updates the next time.

Finally, the illustration shows that:

  • Time steps for a component do not need to be fixed
  • It is not necessary to know the next step size in advance; components only need to be able to report their current simulation time

Initialization

During the initialization process, components populate their inputs and outputs with metadata like expected units and grid specifications. Further, components must push initial values into all outputs.

Due to an iterative initialization approach, components can already exchange data and metadata during that phase. However, after each connection attempt, components need to signal via their status whether they are done with pulling, pulled nothing, or pulled something but are not done yet.

Unresolvable circular dependencies are detected and reported by the scheduler.

See chapter The Connect Phase ™ for details and examples.

Data flow

This book has moved to the FINAM documentation. You are viewing an outdated version!

FINAM is based on a hybrid push-pull information flow. Components and adapters "push" data updates to their outputs. These default outputs (Output) store the data, and notify all connected inputs and adapters (targets) that new data is available, but without passing the data itself. Adapters forward these notifications, until the input of another component is reached. For details, see Figures 1 and 2.

Component

Figure 1: Simple data flow. Solid: data, dashed: notifications/pulls.

  1. Model A updates
    1a: component pushes data to outputs
    1b: outputs forwards notification (w/o effect)
  2. Model B updates
    2a: component pulls data from input
    2b: input pulls from connected output
    2c: data is returned to component

Component

Figure 2: Simple data flow with adapter. Solid: data, dashed: notifications/pulls

  1. Model A updates
    1a: component pushes data to outputs
    1b: outputs forwards notification
    1c: adapter forwards notification (w/o effect)
  2. Model B updates
    2a: component pulls data from input
    2b: input pulls from connected adapter
    2c: adapter pulls from connected output
    2d: adapter transforms data
    2e: transformed data is returned to component

Components with time steps are updated by the driver, while components without time steps can update when notified about new data. They are then free to pull data from their inputs. In the most basic case, this pull propagates backwards through the chain of adapters until a component's output is reached, where the latest available data was stored during the push. Adapters handle the data sequentially, and the pulled input returns the transformed data for usage in the component.

The scheduling of the driver ensures that the requested data is always available.

There is one category of adapters that requires a different strategy: those intended for temporal interpolation, aggregation, etc. Such time-dependent adapters need to transform data from multiple points in time to data for one particular requested point in time. For that sake, these adapters do not simply execute their operations during pull. When notified about new input data that became available, the adapter pulls that data and stores it internally. When data is pulled from downstream of the adapter, it does its calculations (e.g. temporal interpolation for the requested point in time) and returns the result. For details, see Figure 3.

Component

Figure 3: Push-based data flow with adapter. Solid: data, dashed: notifications/pulls.

  1. Model A updates
    1a: component pushes data to outputs
    1b: outputs forwards notification
    1c: adapter pulls and accumulates (or aggregates) data
    1d: adapter forwards notification (w/o effect)
  2. Model B updates
    2a: component pulls data from input
    2b: input pulls from connected adapter
    2c: adapter aggregates and returns data

Time-related adapters should still forward notifications, just as usual adapters do.

Metadata and time

This book has moved to the FINAM documentation. You are viewing an outdated version!

In FINAM, data has certain information associated. This is necessary to ensure valid coupling and is one of the foundations for the ease of use of FINAM.

This chapter gives a brief overview of the most important aspects in this regard. For more details and usage by developers, see chapter Data and metadata.

Time

In FINAM, each data exchange is associated with time information.

All times are represented by datetime objects, namely datetime for time points and timedelta for time spans.

Metadata

In FINAM, all data is associated with metadata.

Inputs and outputs of components specify the metadata describing the data they send or receive. Internally, this is used for consistency checks, and for automated data transformations.

FINAM metadata follows the CF Conventions.

There are two types of mandatory metadata:

Grid specification

Most of the data exchanged through FINAM will be spatio-temporal be their nature. FINAM supports different types of structured grids and unstructured grids/meshes, as well as unstructured point data.

For data that is not on a spatial grid, a placeholder "no-grid" type is provided.

Inputs as well as outputs must specify the grid specification for the data they send and receive, respectively. We provide regridding adapters to transform between different grids or meshes in an automated way.

Coordinate Reference Systems (CRS) conversions are also covered by the regridding adapters.

Units of measurement

All data in FINAM has units of measurement. The units can, however, be "dimensionless" for no actual units.

Unit conversions along links between components is done automatically, based on the metadata provided by the receiving inputs.

FINAM uses the pint library for units handling, and follows the CF Conventions.

More metadata

More than the above metadata items can be present on FINAM data. For more details and usage by developers, see chapter Data and metadata.

Metadata is passed between components during the initialization, which allows for initializing components from external metadata. For details, see chapter The Connect Phase ™.

Using FINAM

This book has moved to the FINAM documentation. You are viewing an outdated version!

The following chapters target users that want to couple models that are already prepared for FINAM.

Chapter Installation covers how to install the FINAM Python package.

Chapter Model coupling scripts covers how to write Python scripts that compose and run model coupling setups.

Chapter Known components and adapters provides a list of known FINAM components and adapters for the use in coupling scripts.

Installation

This book has moved to the FINAM documentation. You are viewing an outdated version!

FINAM can be installed using pip, from a local clone of the Git repository. See the pip website for how to get pip.

  1. Install FINAM from the Git repository:
$ pip install git+https://git.ufz.de/FINAM/finam.git
  1. Test it
$ python

>>> import finam
>>> print(finam.__version__)

Congratulations! You can now start using FINAM.

Model coupling scripts

This book has moved to the FINAM documentation. You are viewing an outdated version!

Coupling setups are created and executed using Python scripts.

Simple example

Here is a simple example coupling two components:

# simple-coupling.py

import random
from datetime import datetime, timedelta

import finam as fm
from finam.modules.generators import CallbackGenerator
from finam.modules.visual.time_series import TimeSeriesView

if __name__ == "__main__":
  # Instantiate components, e.g. models

  # Here, we use a simple component that outputs a random number each step
  generator = CallbackGenerator(
    {"Value": (lambda _t: random.uniform(0, 1), fm.Info(grid=fm.NoGrid()))},
    start=datetime(2000, 1, 1),
    step=timedelta(days=1),
  )

  # A live plotting component
  plot = TimeSeriesView(
    inputs=["Value"],
    start=datetime(2000, 1, 1),
    step=timedelta(days=1),
    intervals=[1],
  )

  # Create a `Composition` containing all components
  composition = fm.Composition([generator, plot])

  # Initialize the `Composition`
  composition.initialize()

  # Couple inputs to outputs
  generator.outputs["Value"] >> plot.inputs["Value"]

  # Run the composition until January 2001
  composition.run(datetime(2001, 1, 1))

In the above example, we couple a simple generator component (CallbackGenerator) with a live plotting component (TimeSeriesView).

Note: with package finam installed, simply run the above scripts with:

$ python simple-coupling.py

The typical steps in a script are:

  1. Instantiate components and adapters (see next example)
  2. Create a Composition and initialize it
  3. Connect outputs to inputs using the overloaded >> operator (__rshift__)
  4. Run the Composition

Inputs and outputs

Inputs and outputs of a component can be retrieved via inputs and outputs properties. Both methods return a Python dict-like, with strings as keys and input or output objects as values, respectively.

An input can be connected to an output using either >> (as in the examples), or the output's method chain(input). Both lines here are equivalent:

generator.outputs["Value"] >> plot.inputs["Value"]
generator.outputs["Value"].chain(plot.inputs["Value"])

Adapters

In the above example, both coupled components match in terms of the exchanged data (numeric value) as well as their time step (1).

This is not necessarily the case for all coupling setups. To mediate between components, FINAM uses adapters. Those can be used to transform data (regridding, geographic projections, ...) or for temporal interpolation or aggregation.

The following examples uses a similar setup like the previous one, but with differing time steps and two chained adapters:

# adapter-coupling.py

import random
from datetime import datetime, timedelta

import finam as fm
from finam.adapters import base, time
from finam.modules import generators, visual

if __name__ == "__main__":
  # Instantiate components, e.g. models

  # Here, we use a simple component that outputs a random number each step
  generator = generators.CallbackGenerator(
    {"Value": (lambda _t: random.uniform(0, 1), fm.Info(grid=fm.NoGrid()))},
    start=datetime(2000, 1, 1),
    step=timedelta(days=10),
  )

  # A live plotting component
  plot = visual.time_series.TimeSeriesView(
    inputs=["Value"],
    start=datetime(2000, 1, 1),
    step=timedelta(days=1),
    intervals=[1],
  )

  # Create two adapters for...
  # temporal interpolation
  time_interpolation_adapter = time.LinearTime()
  # data transformation
  square_adapter = base.Callback(lambda x, _time: x * x)

  # Create a `Composition` containing all components
  composition = fm.Composition([generator, plot])

  # Initialize the `Composition`
  composition.initialize()

  # Couple inputs to outputs, via multiple adapters
  (
          generator.outputs["Value"]
          >> time_interpolation_adapter
          >> square_adapter
          >> plot.inputs["Value"]
  )

  # Run the composition until January 2000
  composition.run(datetime(2001, 1, 1))

Adapter chaining

As can be seen from the example, components and adapters can be chained using the >> operator (or the chain(...) method).

This is achieved by:

  1. An adapter is an input, and at the same time an output
  2. The chained input is returned by >> and chain(...). In case the chained input is an adapter (and thus also an output), it can be immediately reused in a further chaining operation

Logging

FINAM provides a comprehensive logging framework built on Pythons standard logging package.

You can configure the base logger when creating the Composition as shown above:

import logging

comp = Composition(
    modules,
    logger_name="FINAM",
    print_log=True,
    log_file=True,
    log_level=logging.INFO,
)

There you have several options:

  • logger_name: (str) Base name of the logger in the output (FINAM by default)
  • print_log: (bool) Whether logging should be shown in the terminal output
  • log_file: (None, bool, pathlike) Whether a log-file should be created
    • None or False: no log file will be written
    • True: a log file with the name {logger_name}_{time.strftime('%Y-%m-%d_%H-%M-%S')}.log will be created in the current working directory (e.g. FINAM_2022-09-26_12-58-15.log)
    • <pathlike>: log file will be created under the given path
  • log_level: (int) this will control the level of logging (logging.INFO by default)
    • only log messages with a level equal or higher than the given logging level will be shown
    • options are (from most to least verbose): logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL or any positive integer number

A log file could look like this, when setting the logging level to logging.INFO:

2022-08-26 11:31:28,283 - FINAM - INFO - doing fine
2022-08-26 11:31:28,284 - FINAM - WARNING - Boo

or like this, when setting logging level to logging.DEBUG:

2022-08-26 11:31:28,283 - FINAM - INFO - doing fine
2022-08-26 11:31:28,284 - FINAM - WARNING - Boo
2022-08-26 11:31:28,285 - FINAM - DEBUG - Some debugging message

Components and adapters

This book has moved to the FINAM documentation. You are viewing an outdated version!

This chapter lists known components and adapters for use in FINAM compositions.

Included in FINAM core package

Provided by FINAM developers

Known 3rd party

Developing for FINAM

This book has moved to the FINAM documentation. You are viewing an outdated version!

The following chapters target developers that want to prepare their existing models for use in FINAM, or that want to write FINAM-compatible models from scratch.

Besides preparation of models, implementation of adapters that mediate data between models is also covered.

Interfaces

This book has moved to the FINAM documentation. You are viewing an outdated version!

FINAM is primarily a collection of interfaces that allows different models and other components to communicate.

For all interfaces, FINAM also provides abstract or concrete implementations to speed up component development.

Class diagram

The following figure shows a diagram of FINAM's core interfaces and classes. Arrows indicate inheritance. The properties and methods are those typically used or implemented by developers.

Components

Components represent linkable entities like models. There are two interfaces for components: IComponent and ITimeComponent.

IComponent

IComponent serves for pull-based components without an explicit time step. It provides all the basic methods required for component communication and execution.

  • initialize(self) sets up the component
  • connect(self) pushes initial values to output slots
  • validate(self) checks the component for validity
  • update(self) makes a calculation step
  • finalize(self) shuts down the component

These methods are called by the scheduler in the given order (and repeatedly for update()), each for all components, before proceeding to the next method.

For each of these methods, there is a private method with the same name and an underscore prefix, like _initialize(self). Component developers implement these private methods, which are called internally by their public counterpart. For details, see chapter Writing components.

To access a component's input and output slots, there are the properties:

  • inputs returns a dict-like of IInput slots by name
  • outputs returns a dict-like of IOutput slots by name

Finally:

  • status returns the component's current ComponentStatus (CREATED, INITIALIZED, ...)

The abstract class Component provides a basic implementation for IComponent. Classes extending Component must override methods named of the first block, with underscore, like _initialize(). inputs, outputs and status are provided as basic implementations.

ITimeComponent

ITimeComponent extends IComponent and serves for components with explicit time step, like simulation models. In addition to IComponent, it adds one property:

  • time should report the component's current time, as a datetime object

As ITimeComponent extends IComponent, only ITimeComponent needs to be implemented.

The abstract class TimeComponent provides a basic implementation for ITimeComponent. It is basically identical to Component, and in addition provides a basic implementation for time.

Inputs and Outputs

Interfaces IInput and IOutput define coupling slots.

In module sdk, Input and Output are provided as implementations for IInput and IOutput, respectively. They should suffice most use cases.

IInput

IInput represents a data exchange input slot, with the following methods:

  • set_source(self, source) sets an IOutput as source for this input
  • get_source(self) returns the IOutput that is the source for this input
  • source_updated(self, time) informs the input that the connected IOutput has new data available
  • pull_data(self, time) retrieves and returns the connected IOutput's data

Components usually only use pull_data(self, time) in their _update(self) method. All other methods are only used under the hood.

All these methods are implemented in Input, so there is normally no need to write an own implementation for IInput.

Another implementation is provided by CallbackInput, for use in push-based components without a time step. They can connect to source_updated(self, time) by providing a callback function.

Other classes derived from Input can overwrite the private _source_updated(self, time) method, which is called by source_updated(self, time).

IOutput

IOutput represents a data exchange output slot, with the following methods:

  • add_target(self, target) adds an IInput as target for this output
  • get_target(self) returns the list of IInput targets of this output
  • push_data(self, data, time) is used to populate the output with data after an update
  • notify_targets(self, time) informs coupled IInputs that new data is available
  • get_data(self, time) returns the data in this output
  • chain(self, input) connects this output to an IInput (or an adapter)

Components usually only use _push_data(self, data, time) in their update(self) method. During coupling setups, chain(self, input) or it's synonym operator >> are used. All other methods are only used under the hood.

All these methods are implemented in Output, so there is normally no need to write an own implementation for IOutput.

Other classes derived from Output can overwrite the private _get_data(self, time) method, which is called by get_data(self, time).

Adapters

Adapters serve for data transformations between outputs and inputs of different components.

IAdapter

The interface IAdapter serves for implementing adapters. It simply combines IInput and IOutput, so it is both at the same time. IAdapter provides all the methods of IInput and IOutput, but most of them are only used under the hood.

Classes implementing IAdapter can extend Adapter, which provides default implementations for Input and Output methods.

Time-independent/one-shot adapters need to override _get_data(self, time). Inside this method, they get their input via self.pull_data(time), transform it, and return the result.

Time-aware adapters, e.g. for temporal interpolation, usually override _source_updated(self, time) and _get_data(self, time). In _source_updated(self, time), incoming data is collected (and potentially aggregated), while in _get_data(self, time) the result is returned.

For details, see chapter Writing adapters.

NoBranchAdapter

Some time-aware adapters may not allow for branching in the subsequent adapter chain. I.e. they do not support multiple target components. For these cases, NoBranchAdapter is provided as a marker interface without any methods.

Writing components

This book has moved to the FINAM documentation. You are viewing an outdated version!

This chapter provides a step-by-step guide to implement a component with time (e.g. a model) in pure Python. For writing Python bindings for other languages, see Python bindings.

Completing the chapter will result in a simple component called DummyModel. We will build up the component step by step, accompanied by some test code. Finally, it will have two input slots and one output slot, and will calculate the sum of its inputs.

The component will have internal time stepping, like a simulation model would have. For implementing components without internal time, see chapter Components without time step.

It is assumed that you have FINAM installed, as well as pytest.

Set up a Python project

Create the following project structure:

- dummy_model/
   +- src/

We call dummy_model the project directory from here on.

Implement TimeComponent

The class TimeComponent provides an abstract implementation of the interface ITimeComponent to make implementation easier. Start by extending TimeComponent in a class we call DummyModel in src/dummy_model.py.

import finam as fm


class DummyModel(fm.TimeComponent):
    pass

However, we want to test our implementation while building up, so extend the file to the following content:

import finam as fm
import unittest                                                      # <--


class DummyModel(fm.TimeComponent):
    pass


class TestDummy(unittest.TestCase):                                  # <--
    def test_dummy_model(self):                                      # <--
        model = DummyModel()                                         # <--
        self.assertTrue(isinstance(model, DummyModel))               # <--

In your project directory run the following to test it:

$ python -m pytest -s src/dummy_model.py

Constructor

The component needs a constructor which calls the super class constructor.

import finam as fm
import unittest
from datetime import datetime                                        # <--


class DummyModel(fm.TimeComponent):

    def __init__(self, start):                                       # <--
        super().__init__()                                           # <--
        self.time = start


class TestDummy(unittest.TestCase):
    def test_dummy_model(self):
        model = DummyModel(start=datetime(2000, 1, 1))
        self.assertEqual(model.status, fm.ComponentStatus.CREATED)   # <--
        self.assertEqual(model.time, datetime(2000, 1, 1))           # <--

The property status is provided by TimeComponent, as are inputs and outputs, which are initialized with defaults. We will manipulate them later.

TimeComponent's time property must be initialized with a datetime object.

The constructor is also the place to define class variables required by the component. We want our component to have a user-defined time step, so we add it here:

import finam as fm
import unittest
from datetime import datetime, timedelta


class DummyModel(fm.TimeComponent):

    def __init__(self, start, step):                                 # <--
        super().__init__()
        self._step = step                                            # <--
        self.time = start


class TestDummy(unittest.TestCase):
    def test_dummy_model(self):
        model = DummyModel(start=datetime(2000, 1, 1),               # <--
                           step=timedelta(days=7))                   # <--
        self.assertEqual(model.status, fm.ComponentStatus.CREATED)
        self.assertEqual(model.time, datetime(2000, 1, 1))
        self.assertEqual(model._step, timedelta(days=7))             # <--

Run the test again to check everything is working.

Next, we need to implement or override some methods of TimeComponent

Initialize

In _initialize(), we define the component's input and output slots. It is called internally by the initialize() method.

(We will shorten previously completed parts and imports from now on.)

import finam as fm
import unittest
from datetime import datetime, timedelta


class DummyModel(fm.TimeComponent):

    def __init__(self, start, step):
        # ...

    def _initialize(self):                                             # <--
        self.inputs.add(name="A", grid=fm.NoGrid())                    # <--
        self.inputs.add(name="B", grid=fm.NoGrid())                    # <--
        self.outputs.add(name="Sum", grid=fm.NoGrid())                 # <--

        self.create_connector()                                        # <--


class TestDummy(unittest.TestCase):
    def test_dummy_model(self):
        model = DummyModel(start=datetime(2000, 1, 1),
                           step=timedelta(days=7))
        # ...

        model.initialize()
        self.assertEqual(model.status, fm.ComponentStatus.INITIALIZED)  # <--
        self.assertEqual(len(model.inputs), 2)                          # <--
        self.assertEqual(len(model.outputs), 1)                         # <--

Note that inputs and outputs are added with a name and a grid (or grid specification). They can later be accessed by the name, like self.inputs["A"].

The grid specification defines what inputs expect to receive, or what outputs provide. Here, we set it to NoGrid(), as we want to handle scalars only. In most real use cases, however, grid will be a grid specification like rectilinear or unstructured grids. See chapter Data types for more details.

In the last line, we call create_connector(), which sets up an internal helper that manages the initial exchange of data and metadata. For details and possible arguments, see chapter The Connect Phase ™.

Connect and validate

For the coupling to work, it is necessary that every component populates its outputs with initial values. This is done in _connect().

After this connection phase, models can validate their state in _validate(). We do nothing there.

# imports...


class DummyModel(fm.TimeComponent):

    def __init__(self, step):
        # ...

    def _initialize(self):
        # ...

    def _connect(self):                                                      # <--
        self.try_connect(time=self.time, push_data={"Sum": 0})               # <--
        
    def _validate(self):                                                     # <--
        pass                                                                 # <--

In _connect(), we call try_connect with the component's time (it's starting time), and a dictionary of data to push for each input. For more complex use cases like pulling data, see chapter The Connect Phase ™.

For the tests, we need to set up a real coupling from here on, as the component's inputs require connections in this phase.

class TestDummy(unittest.TestCase):
    def test_dummy_model(self):
        # our model
        model = DummyModel(start=datetime(2000, 1, 1),
                           step=timedelta(days=7))
        
        # a component to produce inputs, details not important
        generator = fm.modules.generators.CallbackGenerator(
            callbacks={
                "A": (lambda t: t.day, fm.Info(time=None, grid=fm.NoGrid())),
                "B": (lambda t: t.day * 2, fm.Info(time=None, grid=fm.NoGrid()))
            },
            start=datetime(2000, 1, 1),
            step=timedelta(days=7)
        )
        
        # a component to consume output, details not important
        consumer = fm.modules.debug.DebugConsumer(
            inputs={"Sum": fm.Info(grid=fm.NoGrid())},
            start=datetime(2000, 1, 1),
            step=timedelta(days=7)
        )
        
        # set up a composition
        composition = fm.Composition([model, generator, consumer],
                                     log_level="DEBUG")
        composition.initialize()
        
        # connect components
        generator.outputs["A"] >> model.inputs["A"]
        generator.outputs["B"] >> model.inputs["B"]

        model.outputs["Sum"] >> consumer.inputs["Sum"]
        
        # run the connection/exchange phase
        composition.connect()

        self.assertEqual(consumer.data, {"Sum": 0})

Here, we set up a complete coupling using a CallbackGenerator as source. A DebugConsumer is used as a sink to force the data flow and to allow us to inspect the result.

Update

Method _update() is where the actual work happens. It is called every time the scheduler decides that the component is on turn to make an update.

In _update, we get the component's input data, do a "model step", increment the time, and push results to the output slot.

# imports...


class DummyModel(fm.TimeComponent):

    def __init__(self, step):
        # ...

    def _initialize(self):
        # ...

    def _connect(self):
        # ...

    def _validate(self):
        # ...

    def _update(self):
        a = self.inputs["A"].pull_data(self.time)
        b = self.inputs["B"].pull_data(self.time)
        
        result = a + b
        
        # We need to unwrap the data here, as the push time will not equal the pull time.
        # This would result in conflicting timestamps in the internal checks
        result = fm.data.strip_data(result)

        self._time += self._step

        self.outputs["Sum"].push_data(result, self.time)


class TestDummy(unittest.TestCase):
    def test_dummy_model(self):
        # ...
    
        composition.run(t_max=datetime(2000, 12, 31))

The test should fail, as we still need to implement the _finalize() method.

Finalize

In method _finalize, the component can do any cleanup required at the end of the coupled run, like closing streams or writing final output data to disk.

We do nothing special here.

# imports...


class DummyModel(TimeComponent):

    def __init__(self, step):
        # ...

    def _initialize(self):
        # ...

    def _connect(self):
        # ...

    def _validate(self):
        # ...

    def _update(self):
        # ...

    def _finalize(self):
        pass

Final code

Here is the final code of the completed component.

import unittest
from datetime import datetime, timedelta

import finam as fm


class DummyModel(fm.TimeComponent):
    def __init__(self, start, step):  # <--
        super().__init__()
        self._step = step  # <--
        self.time = start

    def _initialize(self):  # <--
        self.inputs.add(name="A", grid=fm.NoGrid())  # <--
        self.inputs.add(name="B", grid=fm.NoGrid())  # <--
        self.outputs.add(name="Sum", grid=fm.NoGrid())  # <--

        self.create_connector()  # <--

    def _connect(self):  # <--
        self.try_connect(time=self.time, push_data={"Sum": 0})  # <--

    def _validate(self):  # <--
        pass

    def _update(self):
        a = self.inputs["A"].pull_data(self.time)
        b = self.inputs["B"].pull_data(self.time)

        result = a + b

        # We need to unwrap the data here, as the push time will not equal the pull time.
        # This would result in conflicting timestamps in the internal checks
        result = fm.data.strip_data(result)

        self._time += self._step

        self.outputs["Sum"].push_data(result, self.time)

    def _finalize(self):
        pass


class TestDummy(unittest.TestCase):
    def test_dummy_model(self):
        model = DummyModel(start=datetime(2000, 1, 1), step=timedelta(days=7))
        generator = fm.modules.generators.CallbackGenerator(
            callbacks={
                "A": (lambda t: t.day, fm.Info(time=None, grid=fm.NoGrid())),
                "B": (lambda t: t.day * 2, fm.Info(time=None, grid=fm.NoGrid())),
            },
            start=datetime(2000, 1, 1),
            step=timedelta(days=7),
        )
        consumer = fm.modules.debug.DebugConsumer(
            inputs={"Sum": fm.Info(grid=fm.NoGrid())},
            start=datetime(2000, 1, 1),
            step=timedelta(days=7),
        )
        composition = fm.Composition([model, generator, consumer], log_level="DEBUG")
        composition.initialize()

        generator.outputs["A"] >> model.inputs["A"]
        generator.outputs["B"] >> model.inputs["B"]

        model.outputs["Sum"] >> consumer.inputs["Sum"]

        composition.connect()

        self.assertEqual(consumer.data, {"Sum": 0})

        composition.run(t_max=datetime(2000, 12, 31))

Writing adapters

This book has moved to the FINAM documentation. You are viewing an outdated version!

This chapter provides a step-by-step guide to implement adapters in pure Python. For writing Python bindings for other languages, see Python bindings.

Completing the chapter will result in two adapters called Scale and TimeInterpolation. We will build up the adapters step by step, accompanied by some test code.

It is assumed that you have FINAM installed, as well as pytest.

Set up a Python project

Create the following project structure:

- dummy_adapters/
   +- src/

Simple Scale adapter

This is a simple, purely pull-based adapter. When output is requested, it should simply pull from its input, transform and forward it.

We implement Adapter and only need to overwrite its method get_data(self, time), which is called from downstream to request data.

File src/scale.py:

import finam as fm


class Scale(fm.Adapter):
    def __init__(self, scale):
        super().__init__()
        self.scale = scale

    def _get_data(self, time):
        d = self.pull_data(time)
        return d * self.scale

In _get_data(self, time), we:

  1. Pull the input for the requested time
  2. Multiply the input by scale and return the result

Time-dependent TimeInterpolation adapter

The purpose of this adapter is to do temporal interpolation between upstream time steps. As an example, there could be a model with a weekly time step that passes data to another model with a daily time step. Assuming continuous transitions of the modelled data, temporal interpolation between the weekly time steps is required.

  ^                          V
  |                        _.o----
  |                    _.-´
  |                _.-´|
  |            _.-´    |
  |      V _.-´        |
  |  ----o´            |
  +-------------------------------------> t
                       ^

Here, a simple pull-based mechanism is not sufficient. The adapter needs to store each new data entry that becomes available, and calculate the interpolated data when requested.

Due to FINAM's scheduling algorithm, it is guaranteed that the time stamp of any request lies in the interval of the previous two time steps of any other component (see Coubling and Scheduling for details). Thus, it is not required to store data for more than two time stamps.

Accordingly, this is the constructor (file src/time_interpolation.py):

import finam as fm

class TimeInterpolation(fm.Adapter):

    def __init__(self):
        super().__init__()
        self.old_data = None
        self.new_data = None

The adapter needs to react to downstream requests as well as to new data available upstream. This functionality is provided by Adapter's methods _get_data(self, time) and _source_updated(self, time), respectively.

import finam as fm

class TimeInterpolation(fm.Adapter):

    def __init__(self):
        super().__init__()
        self.old_data = None
        self.new_data = None

    def _source_updated(self, time):
        pass

    def _get_data(self, time):
        pass

In _source_updated(...), we need to store incoming data:

import finam as fm

class TimeInterpolation(fm.Adapter):

    def __init__(self):
        super().__init__()
        self.old_data = None
        self.new_data = None

    def _source_updated(self, time):
        self.old_data = self.new_data
        self.new_data = (time, fm.data.strip_data(self.pull_data(time)))

    def _get_data(self, time):
        pass

We "move" the previous new_data to old_data, and replace new_data by the incoming data, as a (time, data) tuple. As the output time will differ from the input time, we need to strip the time off the data by calling strip_data(data).

In _get_data(...), we can now do the interpolation whenever data is requested from upstream.

import finam as fm

class TimeInterpolation(fm.Adapter):

    def __init__(self):
        super().__init__()
        self.old_data = None
        self.new_data = None

    def _source_updated(self, time):
        self.old_data = self.new_data
        self.new_data = (time, fm.data.strip_data(self.pull_data(time)))

    def _get_data(self, time):
        if self.old_data is None:
            return self.new_data[1]

        dt = (time - self.old_data[0]) / (self.new_data[0] - self.old_data[0])

        o = self.old_data[1]
        n = self.new_data[1]

        return o + dt * (n - o)

In _get_data(...), the following happens:

  1. If only one data entry was received so far, we can't interpolate and simply return the available data. Otherwise...
  2. Calculate dt as the relative position of time in the available data interval (in range [0, 1])
  3. Interpolate and return the data

Note that, although we use datetime when calculating dt, we get a scalar output. Due to dt being relative, time units cancel out here.

Data and metadata

This book has moved to the FINAM documentation. You are viewing an outdated version!

This chapter explains data and metadata in FINAM.

Data arrays

Internally, all data is passed as xarray.DataArray. In addition, data is wrapped in pint units, and a time axis with a single entry is added.

Data can be pushed to outputs as any type that can be wrapped in xarray.DataArray. This includes numpy ndarray, lists, and scalar values. Wrapping, adding time axis and units are performed internally, based on the available metadata (see section Metadata).

Inputs receive data in the xarray form, with units and time axis.

Several tool functions are provided in finam.data to convert to and from xarray:

  • to_xarray(data, name, info, time)
    Wraps data, adds time axis and units based on info (see The Info object). Performs a metadata check if data is already an xarray.
  • strip_time(xdata)
    Squeezes away the time axis if there is a single entry only, and raises an error otherwise. Returns an xarray with units.
  • get_data(xdata) Unwraps the data to a numpy array with units (pint.Quantity), and with time axis preserved.
  • strip_data(xdata) Combines strip_time(xdata) and get_data(xdata). Returns a numpy array with units, without the time axis.
  • get_magnitude(xdata) Extracts data without units. Returns a numpy array without units, but with time axis preserved.
  • get_units(xdata) Gets the pint units of the data
  • get_dimensionality(xdata) Gets the pint dimensionality of the data (like length, mass, ...)
  • has_time(xdata) Checks if the data has a time axis
  • get_time(xdata) Gets the time axis values of the data

Metadata

In FINAM, all data is associated with metadata.

Inputs and outputs of components specify the metadata describing the data they send or receive. Internally, this is used for consistency checks, and for automated data transformations.

FINAM metadata follows the CF Conventions.

There are two types of mandatory metadata:

Metadata is passed around as objects of type Info:

The Info object

Objects of type Info represent the metadata associated with an input or output. It has the following properties:

For convenience, entries in meta can be used like normal member variables:

info = Info(grid=NoGrid(), units="m", foo="bar")

print(info.units)
print(info.foo)

When creating inputs or outputs in components, the Info object does not need to be constructed explicitly. In component code, these two lines are equivalent:

self.inputs.add(name="A", grid=NoGrid(), units="m")
self.inputs.add(name="A", info=Info(grid=NoGrid(), units="m"))

Metadata from source or target

Any Info attributes initialized with None will be filled from the metadata on the other end of the coupling link. E.g. if the grid specification of an input is intended to be taken from the connected output, the input can be initialized like this:

self.inputs.add(name="Input_A", grid=None, units="m")

This works in the same way for outputs to get metadata from connected inputs.

For more details on metadata exchange, see chapter The Connect Phase ™.

Grid specification

Most of the data exchanged through FINAM will be spatio-temporal be their nature. FINAM supports different types of structured grids and unstructured grids/meshes, as well as unstructured point data.

For data that is not on a spatial grid, a placeholder "no-grid" type is provided.

Inputs as well as outputs must specify the grid specification for the data they send and receive, respectively. We provide regridding adapters to transform between different grids or meshes in an automated way.

Coordinate Reference Systems (CRS) conversions are also covered by the regridding adapters.

Available grid types are:

Non-spatial grids

NoGrid(dims)

For data that is not on a spacial grid. dims specifies the number of dimensions, like 0 for scalars, 1 for 1D arrays, etc.

Spatial grids

All spatial grids can have up to 3 dimensions.

RectilinearGrid(axes=[axis_x, axis_y, axis_z])

For rectilinear grids, with uneven spacing along some axes.

UniformGrid(dims=(sx, sy, sz), spacing=(dx, dy, dz), origin=(ox, oy, oz))

For uniform rectangular grids, with even spacing along each axis. A sub-class of RectilinearGrid.

EsriGrid(nrows, ncols, cellsize, xllcorner, yllcorner)

For square grids according the ESRI/ASCII grid standard. A sub-class of UniformGrid.

UnstructuredGrid(points, cells, celltypes)

For unstructured grids (or meshes), composed of triangles and/or quads in 2D, and tetrahedrons of hexahedrons in 3D.

UnstructuredPoints(points)

For unstructured point-associated data that does not require cells.

Class diagram grids

The following figure shows a diagram of grid classes inheritance hierarchy.

Common grid properties

CRS: All spatial grid types have a property crs for the Coordinate Reference Systems. The property can take any values understood by pyproj4. In many cases, this will just be an EPSG code, like crs="EPSG:32632"

Order: All structured grids have an order attribute for being in either Fortran ("F") or C ("C") order.

Data location: For all spatial grids except UnstructuredPoints, data can be associated to either cells or points, given by the data_location attribute.

Axis names: Grid axes are names according to the axes_names attribute.

Axis order: Regular grids can have inverted axis order (i.e. zyx instead of xyz), indicated by the axes_reversed attribute.

Axis direction: Axis direction can be inverted, like with descending velues for the y axis. This is indicated by the axes_increase attribute, which is a tuple of boolean values.

Units

All data in FINAM has units of measurement. The units can, however, be "dimensionless" for no actual units.

Unit conversions along links between components is done automatically, based on the metadata provided by the receiving inputs. So if an input was initialized with units="km", and data is passed in meters, the input will internally do the conversion to kilometers.

FINAM uses the pint library for units handling, and follows the CF Conventions.

For direct to pint units, the central units registry is exposed by finam.UNITS.

Metadata flow

For details on how metadata is provided, and how it is passed around during coupling, see chapter The Connect Phase ™.

The Connect Phase ™

This book has moved to the FINAM documentation. You are viewing an outdated version!

The "Connect Phase" between linking components and running the composition is a crucial foundation for the coupling principle of FINAM.

During this phase, coupled components exchange metadata about the data to be passed, and the initial data. FINAM uses an iterative process in order to allow for dependencies between components already before the actual simulation. This enables components to be initialized based on data received from other components, without an explicit order. Even circular!

For convenience, Component provides the methods create_connector() and try_connect() for handling this phase. See section Implementing The Connect Phase ™.

Metadata

Before data can be passed from an output to an inpout, metadata must be exchanged in both directions. This has multiple purposes:

  • It allows for checking compatibility of connection endpoints
  • Adapters can determine the conversions they should perform, e.g. the source and target grid specification and CRS for a regridding adapter
  • Components can use metadata from other components for their own initialization (and later calculations)

Using metadata for initialization works in both directions.

A receiving component can use metadata from a linked sending component. E.g. a component could be initialized from data (and metadata) read by a geodata reader component, and set up a simulation accordingly.

Contrary, a generating component (e.g. for synthetic landscapes or climate) can use metadata from a linked receiving component. It can then generate data matching the expected metadata.

Metadata must be provided for all inputs and all outputs. This can happen in _initialize() when constructing inputs and outputs. If the information is not available there yet (because it depends on linked components), it can happen in _connect() via try_connect() (see section Implementing The Connect Phase ™).

For details on the metadata itself, see chapter Data and metadata.

Data

After metadata was exchanged along a link in both directions, data can be passed. Components must provide initial data for all outputs. Further, components can pull data from inputs during The Connect Phase ™.

Iterative connect

The flexibility described above is achieved by an iterative connection process. The _connect() methods of components are called repeatedly. Components indicate their connect progress via their state:

  • CONNECTED if everything was exchanged successfully, and initialization in complete
  • CONNECTING if not completed yet, but some new data or metadata was exchanged that was not in the previous calls
  • CONNECTING_IDLE if nothing was exchanged that was not already in a previous call

The status is managed internally by the component's try_connect() method. It can, however, be used to check in _connect() if The Connect Phase ™ was completed.

Circular dependencies

The scheduler tries to repeatedly connect components that not in the CONNECTED state yet. With circular dependencies, this would result in an infinite loop.

To avoid this, there are the two different states CONNECTING and CONNECTING_IDLE. If, during an iteration, no component signals any progress (CONNECTING, or newly CONNECTED), initialization has stalled. The scheduler raises an error and informs about components that could not complete the process.

Implementing The Connect Phase ™

The iterative connection process is largely managed by two methods provided by Component:

Method create_connector()

This method must be called at the end of _initialize(), after all inputs and outputs were created.

If the component has no dependencies in this phase, if can be simply called without arguments.

For components with dependencies, they can be specified like this:

self.create_connector(
    required_in_data=["Input_A", "Input_B"],
    required_out_infos=["Output_B", "Output_B"]
)

Where strings are the names of inputs and outputs that data or metadata exchange is required for.

  • required_in_data: inputs that need to be pulled.
  • required_out_infos: outputs that have incomplete metadata that needs to be filled from connected inputs.

For more on filling incomplete metadata, see section Metadata from source or target.

Method try_connect()

This method must be called in _connect(). It tried to exchange metadata and data, and sets the component's status to one of CONNECTED, CONNECTING or CONNECTING_IDLE, depending on the progress.

The method has three optional arguments:

  • exchange_infos: a dictionary of (newly) available metadata for inputs
  • push_infos: a dictionary of (newly) available metadata for outputs
  • push_data: a dictionary of (newly) available data for outputs

Note that exchange_infos and push_infos are not required for inputs and outputs that were created with metadata in _initialize().

As _connect() can be called by the scheduler multiple times, the above metadata and data can be provided stepwise, as it becomes available.

The connector

The above methods internally call the components connector (a ConnectHelper). Besides managing the connection process, it also keeps track of already exchanged metadata and data. There are several properties that let components access retrieved information, and check progress:

  • in_infos: a dictionary of completed/exchanged input metadata infos, may contain None values
  • in_data: a dictionary of successfully pulled input data, may contain None values
  • out_infos: a dictionary of completed/exchanged output metadata, may contain None values
  • infos_pushed: a dictionary of booleans which infos were pushed to outputs
  • data_pushed: a dictionary of booleans which data was pushed to outputs

Simple case - no dependencies

In the most simple case, all metadata is known in _initialze(), and data is pushed in _connect():

class SimpleConnect(TimeComponent):
    
    def _initialize(self):
        self.inputs.add(name="A", grid=NoGrid(), units="m")
        self.inputs.add(name="B", grid=NoGrid(), units="m")
        self.outputs.add(name="Area", grid=NoGrid(), units="m2")
        
        self.create_connector()
        
    def _connect(self):
        push_data = {"Area": 0}
        self.try_connect(time=self.time, push_data=push_data)

In _initialize(), we create inputs and outputs with metadata (here grid and units). Then, we create the connector with self.create_connector(). No arguments required here, as there are no dependencies.

In _connect(), we call self.try_connect() with a dictionary of all data to push as argument push_data.

More complex - info from input to output

In this example, we want to get a grid specification from an input. This grid specification should then be used for the metadata of the output, and the initial data should be generated from it.

class ComplexConnect(TimeComponent):
    
    def _initialize(self):
        self.inputs.add(name="A", grid=None, units="m")
        self.inputs.add(name="B", grid=NoGrid(), units="m")
        self.outputs.add(name="Area")
        
        self.create_connector()
        
    def _connect(self):
        push_infos = {}
        push_data = {}
        
        pushed = self.connector.data_pushed["Area"]
        info = self.connector.in_infos["A"]
        if not pushed and info is not None:
            push_infos["Area"] = info
            push_data["Area"] = _generate_data(info)

        self.try_connect(time=self.time,
                         push_infos=push_infos,
                         push_data=push_data)

In _initialize(), we set the grid of input A to None. It will be filled from the connected output, and becomes available in connector.in_infos after successful exchange.

For output Area, we give no metadata at all, which means that we delay specifying it until we have the grid specification for input A.

In _connect(), we check if the data was already pushed. If not, and if the input info is available, we add the output info to be pushed to push_infos, and the generated data to push_data (only for a single output here). Then, try_connect() is called with this info and data.

It could happen that try_connect() is called with info and data multiple times, in case only the info can be pushed in a first step, but not the data. This will not cause any problems. Developers should, however, be aware of this behaviour. For efficiency, it might be useful to cache the generated output data to avoid re-generating the data multiple times.

Metadata from source or target

In the above example, we have already seen the use of a grid specification retrieved from a connected upstream component. The process works in both directions.

Any metadata field that is initialized with None will be filled with the value from the other end of the connection. This can happen in the initialization of inputs and outputs:

self.inputs.add(name="A", grid=None, units=None)
self.outputs.add(name="Area", grid=NoGrid(), units=None)

Here, grid and units of the input would be filled from a connected output. For the output, units would be filled from a connected input.

The same mechanism can also be applied in _connect():

info = Info(grid=None, units="m")
self.try_connect(time=self.time, in_infos={"A": info})

Summary metadata initialization

To summarize the use of metadata in the initialization of inputs and outputs:

  • Set metadata attributes (like grid or units) to None to get them filled from the other end of the connection.
    This will, of course, only work if the respective attributes are given at the other end.
  • Set no metadata at all (or use info=None) to delay providing it, and do so in try_connect().

Missing data in adapters

For the iterative initialization, adapters must be able to handle the case of being pulled without data available.

For simple adapters that only overwrite _get_data(), developers can rely on the error raised when pulling the adapter's input:

class Scale(Adapter):
    def __init__(self, scale):
        super().__init__()
        self.scale = scale

    def _get_data(self, time):
        # Pull without data available raises FinamNoDataError
        # Simply let it propagate through the adapter chain
        d = self.pull_data(time)
        return d * self.scale

Adapters that use push and pull (e.g. for temporal interpolation) must check in _get_data() if data is available, and raise a FinamNoDataError otherwise:

class PushPullAdapter(Adapter):
    def _source_updated(self, time):
        # Get data here when notified about an upstream update
    
    def _get_data(self, time):
        ...

        if self.data is None:
            raise FinamNoDataError(f"No data available in {self.name}")

        return self.data

Intercepting metadata in adapters

Usually, adapters simply forward metadata during connecting. Some adapters, however, change the metadata through their transformation, e.g. the grid specification in a regridding adapter.

Handling and altering incoming and outgoing metadata can be done by overwriting an adapter's _get_info() method. The method is called by an upstream input with the requested metadata, and should return the metadata that will actually be delivered.

The default implementation looks like this:

def _get_info(self, info):
    in_info = self.exchange_info(info)
    return in_info

The info argument is the metadata requested from downstream. self.exchange_info(info) is called to propagate the metadata further upstream. It returns the metadata received from upstream, and it is simply returned by _get_info().

For a unit conversion adapter, the method could look like this:

def _get_info(self, info):
    in_info = self.exchange_info(info)
    
    self.out_units = info.units
    out_info = in_info.copy_with(units=self.out_units)
    
    return out_info

Note that a unit conversion adapter is actually not required, as units are handled by inputs internally.

The adapter gets it's own target units from the info coming from downstream, i.e. from the request. It overwrites the units in the info received from upstream by in_info.copy_with(), and passes the result downstream by returning it.

Note that the method can be called multiple times, as an output or adapter can be connected to multiple inputs. The adapter is responsible for checking that the metadata of all connected inputs is compatible.

Logging for components

This book has moved to the FINAM documentation. You are viewing an outdated version!

This chapter provides information on how to use logging in components.

It is assumed that you have read the chapter about components and the chapter on how to configure the logger in the composition.

Using the logger in components

The component base classes from the sdk (Component and TimeComponent) are by default loggable, that means, they have a property called logger you can use in every method beside __init__.

That logger provides simple methods to log a certain message:

  • logger.debug(): write a debug message
  • logger.info(): write an info message
  • logger.warning(): write a warning

Here is an example using the dummy model from the previous chapter:

import finam as fm


class DummyModel(fm.TimeComponent):

    def __init__(self, **config):
        # your setup

    def _initialize(self):
        self.logger.debug("trying to initialize the dummy model")

        self.inputs.add("A")
        self.outputs.add("B")
        self.create_connector()

        self.logger.info("dummy model initialized")

Using this dummy model in a composition would return something similar to this (with log_level=logging.DEBUG):

2022-08-26 11:31:28,283 - FINAM.DummyModel - DEBUG - init
2022-08-26 11:31:28,284 - FINAM.DummyModel - DEBUG - trying to initialize the dummy model
2022-08-26 11:31:28,285 - FINAM.DummyModel - INFO - dummy model initialized

When using log_level=logging.INFO, all debug message would be ignored.

This is convenient because most developers put print message in their code during development and debugging and when using the logger instead of plain print statements, you are able to keep these debugging message.

Logging of raised errors

Developers may implement checks in the components and want to raise Errors, if something is wrong. In order to show these errors in the logger, we provide a context manager ErrorLogger:

import finam as fm
from finam.tools import ErrorLogger


class DummyModel(fm.TimeComponent):

    def __init__(self):
        super().__init__()

    def _initialize(self):
        with ErrorLogger(self.logger):
            raise NotImplementedError("this is not implemented yet")

This will log the error and raise it. Without the context manager, the error would be raised but not logged.

Logging of output of external models

Since FINAM is made to use external models, we also provide convenience functions to log model output, that would be printed to the terminal.

In order to do so, we provide context managers to redirect stdout and stderr to the logger. There are two types:

  • LogStdOutStdErr: Context manager to redirect stdout and stderr to a logger.
  • LogCStdOutStdErr: Context manager to redirect low-level C stdout and stderr to a logger.

When using a compiled extension from Fortran or C, you should use LogCStdOutStdErr, because they use a different framework for printing to stdout/stderr.

Here is an example on how to use these:

import finam as fm
from finam.tools import LogCStdOutStdErr
from yourmodel import model


class DummyModel(fm.TimeComponent):

    def __init__(self):
        super().__init__()
        self.model = model()

    def _initialize(self):
        with LogCStdOutStdErr(self.logger):
            self.model.init()

This will redirect all outputs of model.init() to the logger of the component as INFO (stdout) and WARN (stderr) messages.

You can also configure each log-level with:

LogCStdOutStdErr(self.logger, level_stdout=logging.INFO, level_stderr=logging.WARN)

The LogStdOutStdErr context manager works the exact same way but for Pythons stdout and stderr.

Components without time step

This book has moved to the FINAM documentation. You are viewing an outdated version!

So far, we mainly dealt with components that have an internal time step. In some cases, however, components without an internal time step can be useful. Instead of being updated by the scheduler, they can react to push or pull events from other, linked components.

An example for a push-based component is a file writer that writes data for a time step when it receives some new input. Another example is a map visualization that updates when revieving new data.

An example for a pull-based component could be a generator, or a statistical model, that needs to do calculations only when pulled.

Components without a time step must implement IComponent. Developers can derive from the abstract implementation Component. (In contrast to ITimeComponent/TimeComponent)

Components without a time step usually use CallbackInput or CallbackOutput for push-based and pull-based, respectively.

Before starting this chapter, it is highly recommended to complete chapter Writing components first.

Push-based components

Push-based components can use CallbackInput to get informed about incoming data.

import finam as fm

class PushComponent(fm.Component):
    def __init__(self):
        super().__init__()
        self.data = []

    def _initialize(self):
        self.inputs.add(
            fm.CallbackInput(
                callback=self._data_changed, name="In", grid=fm.NoGrid()
            )
        )
        self.create_connector()

    def _data_changed(self, caller, time):
        data = caller.pull_data(time)
        self.data.append((time, data))

    def _connect(self):
        self.try_connect()

    def _validate(self):
        pass

    def _update(self):
        pass

    def _finalize(self):
        write_to_file(self.data)

In _initialize(), a CallbackInput is added that calls _data_changed() when notified about new data.

In _data_changed(), the data from the calling input is pulled, and stored for later writing to file. In _finalize(), the collected data is written to a file.

Be aware that the callback is already called once during The Connect Phase ™.

With multiple inputs, it may be necessary to check that notifications for all of them are synchronized in time, depending on the particular purpose of the component. This might e.g. be the case when inputs are columns in an output table, with a complete row per time step.

Pull-based components

Push-based components can use CallbackOutput to intercept data pulls.

import finam as fm

class PullComponent(fm.Component):
    def __init__(self):
        super().__init__()

    def _initialize(self):
        self.outputs.add(
            fm.CallbackOutput(
                callback=self._get_data, name="Out", grid=fm.NoGrid()
            )
        )
        self.create_connector()

    def _get_data(self, _caller, time):
        return time.day

    def _connect(self):
        self.try_connect()

    def _validate(self):
        pass

    def _update(self):
        pass

    def _finalize(self):
        pass

In _initialize(), a CallbackOutput is added that calls _get_data() when pulled. _get_data() must return the data that would normally be pushed to the output.

Here, simply the day of month of the request data is returned.

Be aware that the callback is already called once during The Connect Phase ™. This can happen multiple times if it returned None to indicate that no data is available yet.

Also note that the outputs of pull-based components can't be connected to time-interpolating adapters, as they rely on being notified by push events.

Python bindings

This book has moved to the FINAM documentation. You are viewing an outdated version!

FINAM requires Python bindings for coupling components or models written in a language other than Python, like C++, Fortran or Rust.

Implementation of bindings is highly dependent on the language, as well as the approach. Therefore, we provide a repository with example projects for different languages and approaches.

Recommendations

[TODO]