Reporting

FlexMeasures feeds upon raw measurement data (e.g. solar generation) and data from third parties (e.g. weather forecasts).

However, there are use cases for enriching these raw data by combining them:

  • Pre-calculations: For example, from a tariff and some tax rules we compute the real financial impact of price data.

  • Post-calculations: To be able to show the customer value, we regularly want to compute things like money or CO₂ saved.

These calculations can be done with code, but there’ll be many repetitions.

We added an infrastructure that allows us to define computation pipelines and CLI commands for developers to list available reporters and trigger their computations regularly:

  • flexmeasures show reporters

  • flexmeasures add report

The reporter classes we are designing are using pandas under the hood and can be sub-classed, allowing us to build new reporters from stable simpler ones, and even pipelines. Remember: re-use is developer power!

We believe this infrastructure will become very powerful and enable FlexMeasures hosts and plugin developers to implement exciting new features.

Below are two quick examples, but you can also dive deeper in Toy example V: Computing reports.

Example: solar feed-in / self-consumption delta

So here is a glimpse into a reporter we made - it is based on the AggregatorReporter (which is for the combination of any two sensors). This simplified example reporter basically calculates pv - consumption at grid connection point. This tells us how much solar power we fed back to the grid (positive values) and/or the amount of grid power within the overall consumption that did not come from local solar panels (negative values).

This is the configuration of how the computation works:

{
    "method" : "sum",
    "weights" : {
        "pv" : 1.0,
        "consumption" : -1.0
    }
}

This parameterizes the computation (from which sensors does data come from, which range & where does it go):

{
    "input": [
        {
            "name" : "pv",
            "sensor": 1,
            "source" : 1,
        },
        {
            "name" : "consumption",
            "sensor": 1,
            "source" : 2,
        }
    ],
    "output": [
        {
            "sensor": 3,
        }
    ],
    "start" : "2023-01-01T00:00:00+00:00",
    "end" : "2023-01-03T00:00:00+00:00",
}

Note

In addition to filtering by specific data source IDs (source / sources), reporter input data can be filtered using:

  • source_types: list of source type names to include (e.g. ["forecaster", "scheduler"])

  • exclude_source_types: list of source type names to exclude

  • account_id: list of account IDs to include only data from sources belonging to those accounts (note: only matches user-type data sources — DataSources created by reporters, schedulers, and forecasters have no account and will not be matched by this filter)

These correspond to the same filters available on Sensor.search_beliefs.

Example: Profits & losses

A report that should cover a use case right off the shelf for almost everyone using FlexMeasures is the ProfitOrLossReporter ― a reporter to compute how profitable your operation has been. Showing the results of your optimization is a crucial feature, and now easier than ever.

First, reporters can be stored as data sources, so they are easy to be used repeatedly and the data they generate can reference them. Our data source has ProfitOrLossReporter as model attribute and these configuration information stored on its attribute defines the reporter further (the least a ProfitOrLossReporter needs to know is a price):

{
  "data_generator": {
    "config": {
      "consumption_price_sensor": 1
    }
  }
}

And here are more excerpts from the tutorial mentioned above. Here we configure the input and output:

$ echo "
  {
      'input' : [{'sensor' : 4}],
      'output' : [{'sensor' : 9}]
  }" > profitorloss-parameters.json

The input sensor stores the power/energy flow, and the output sensor will store the report. Recall that we already provided the price sensor to use in the reporter’s data source.

$ flexmeasures add report\
  --source 6 \
  --parameters profitorloss-parameters.json \
  --start-offset DB,1D --end-offset DB,2D

Here, the ProfitOrLossReporter used as source (with Id 6) is the one we configured above. With the offsets, we control the timing ― we indicate that we want the new report to encompass the day of tomorrow (see Pandas offset strings).

The report sensor will now store all costs which we know will be made tomorrow by the schedule.

Automating reports

Reports can be queued as background jobs (add --as-job to flexmeasures add report, and let a worker process the reporting queue, see Redis Queues), and computed on a recurring basis by an automation defined on the asset (see Automations for the full concept, including how to manage and run automations).

The reporter and its configuration are stored on a data source (steady across runs, so all report results attribute to the same source), while the report parameters are stored on the automation itself and their timing is resolved freshly on each run:

  • Use start-offset and/or end-offset fields (comma-separated Pandas offsets, like the CLI options above) for a rolling window relative to the run time, in the timezone of the first output sensor. For instance, "start-offset": "-1D,DB" with "end-offset": "DB" reports on the whole previous day.

  • Omit timing fields entirely to report on the period since the automation’s actual last run (falling back to the last cron period — from the previous cron fire time until the run time — when no last run is known, e.g. on the first run).

  • Absolute start/end fields are also accepted, but draw a warning, as each run would then compute the same period.

For example, this automation computes a report over each past day, every morning at 1 AM:

flexmeasures add automation --asset 3 --name "Daily aggregation report" --cron "0 1 * * *" --type reports \
  --reporter PandasReporter --config reporter-config.yml --parameters report-parameters.yml

Report templates

FlexMeasures ships with prepared report templates: ready-made report definitions (a reporter class, a complete reporter config and a parameters skeleton), so you don’t have to author a report definition from scratch. List them with:

$ flexmeasures show report-templates

Name              Reporter              Description
----------------  --------------------  ---------------------------------------------------------------------------------------------------------------
energy-costs      ProfitOrLossReporter  Energy costs over the reporting window, from a power/energy sensor and a consumption price sensor (costs are positive).
self-consumption  PandasReporter        Share of produced energy consumed on-site, from a production and a consumption sensor.

Print a template in full (e.g. to pipe it to a file and edit it):

$ flexmeasures show report-templates --name self-consumption > self-consumption.yml

In a template’s parameters skeleton, you fill in your own sensors by replacing the FILL_IN placeholders (a clear validation error points out any placeholders you left unfilled). The templates also recommend a rolling reporting window (start-offset/end-offset fields, reporting on the previous day), for recurring use.

You can pass a template directly to flexmeasures add report or flexmeasures add automation --type reports with the --template option. The template then acts as defaults: an explicitly given --reporter and any top-level keys in your --config/--parameters files override it, and if you provide any timing fields yourself (start/end/offsets, in the parameters or as CLI options), the template’s recommended timing fields are dropped. For example, this sets up a daily self-consumption report in which only the sensors needed to be filled in:

$ echo "
  input:
    - name: production
      sensor: 1
    - name: consumption
      sensor: 2
  output:
    - name: self-consumption
      sensor: 3
  " > parameters.yml
$ flexmeasures add automation --asset 3 --name "Daily self-consumption report" \
    --cron "0 1 * * *" --type reports --template self-consumption --parameters parameters.yml