.. _reporting: Reporting ============ FlexMeasures feeds upon raw measurement data (e.g. solar generation) and data from third parties (e.g. weather forecasts). However, there are use cases for enriching these raw data by combining them: - Pre-calculations: For example, from a tariff and some tax rules we compute the real financial impact of price data. - Post-calculations: To be able to show the customer value, we regularly want to compute things like money or CO₂ saved. These calculations can be done with code, but there'll be many repetitions. We added an infrastructure that allows us to define computation pipelines and CLI commands for developers to list available reporters and trigger their computations regularly: - ``flexmeasures show reporters`` - ``flexmeasures add report`` The reporter classes we are designing are using pandas under the hood and can be sub-classed, allowing us to build new reporters from stable simpler ones, and even pipelines. Remember: re-use is developer power! We believe this infrastructure will become very powerful and enable FlexMeasures hosts and plugin developers to implement exciting new features. Below are two quick examples, but you can also dive deeper in :ref:`tut_toy_schedule_reporter`. Example: solar feed-in / self-consumption delta ------------------------------------------------ So here is a glimpse into a reporter we made - it is based on the ``AggregatorReporter`` (which is for the combination of any two sensors). This simplified example reporter basically calculates ``pv - consumption`` at grid connection point. This tells us how much solar power we fed back to the grid (positive values) and/or the amount of grid power within the overall consumption that did not come from local solar panels (negative values). This is the configuration of how the computation works: .. code-block:: json { "method" : "sum", "weights" : { "pv" : 1.0, "consumption" : -1.0 } } This parameterizes the computation (from which sensors does data come from, which range & where does it go): .. code-block:: json { "input": [ { "name" : "pv", "sensor": 1, "source" : 1, }, { "name" : "consumption", "sensor": 1, "source" : 2, } ], "output": [ { "sensor": 3, } ], "start" : "2023-01-01T00:00:00+00:00", "end" : "2023-01-03T00:00:00+00:00", } .. note:: In addition to filtering by specific data source IDs (``source`` / ``sources``), reporter input data can be filtered using: - ``source_types``: list of source type names to include (e.g. ``["forecaster", "scheduler"]``) - ``exclude_source_types``: list of source type names to exclude - ``account_id``: list of account IDs to include only data from sources belonging to those accounts (note: only matches user-type data sources — DataSources created by reporters, schedulers, and forecasters have no account and will not be matched by this filter) These correspond to the same filters available on ``Sensor.search_beliefs``. Example: Profits & losses --------------------------- A report that should cover a use case right off the shelf for almost everyone using FlexMeasures is the ``ProfitOrLossReporter`` ― a reporter to compute how profitable your operation has been. Showing the results of your optimization is a crucial feature, and now easier than ever. First, reporters can be stored as data sources, so they are easy to be used repeatedly and the data they generate can reference them. Our data source has ``ProfitOrLossReporter`` as model attribute and these configuration information stored on its ``attribute`` defines the reporter further (the least a ``ProfitOrLossReporter`` needs to know is a price): .. code-block:: json { "data_generator": { "config": { "consumption_price_sensor": 1 } } } And here are more excerpts from the tutorial mentioned above. Here we configure the input and output: .. code-block:: bash $ echo " { 'input' : [{'sensor' : 4}], 'output' : [{'sensor' : 9}] }" > profitorloss-parameters.json The input sensor stores the power/energy flow, and the output sensor will store the report. Recall that we already provided the price sensor to use in the reporter's data source. .. code-block:: bash $ flexmeasures add report\ --source 6 \ --parameters profitorloss-parameters.json \ --start-offset DB,1D --end-offset DB,2D Here, the ``ProfitOrLossReporter`` used as source (with Id 6) is the one we configured above. With the offsets, we control the timing ― we indicate that we want the new report to encompass the day of tomorrow (see Pandas offset strings). The report sensor will now store all costs which we know will be made tomorrow by the schedule. .. _automating_reports: Automating reports -------------------- Reports can be queued as background jobs (add ``--as-job`` to ``flexmeasures add report``, and let a worker process the ``reporting`` queue, see :ref:`redis-queue`), and computed on a recurring basis by an *automation* defined on the asset (see :ref:`automations` for the full concept, including how to manage and run automations). The reporter and its configuration are stored on a data source (steady across runs, so all report results attribute to the same source), while the report parameters are stored on the automation itself and their timing is resolved freshly on each run: - Use ``start-offset`` and/or ``end-offset`` fields (comma-separated Pandas offsets, like the CLI options above) for a rolling window relative to the run time, in the timezone of the first output sensor. For instance, ``"start-offset": "-1D,DB"`` with ``"end-offset": "DB"`` reports on the whole previous day. - Omit timing fields entirely to report on the period since the automation's actual last run (falling back to the last cron period — from the previous cron fire time until the run time — when no last run is known, e.g. on the first run). - Absolute ``start``/``end`` fields are also accepted, but draw a warning, as each run would then compute the same period. For example, this automation computes a report over each past day, every morning at 1 AM: .. code-block:: bash flexmeasures add automation --asset 3 --name "Daily aggregation report" --cron "0 1 * * *" --type reports \ --reporter PandasReporter --config reporter-config.yml --parameters report-parameters.yml .. _report_templates: Report templates -------------------- FlexMeasures ships with prepared report templates: ready-made report definitions (a reporter class, a complete reporter config and a parameters skeleton), so you don't have to author a report definition from scratch. List them with: .. code-block:: bash $ flexmeasures show report-templates Name Reporter Description ---------------- -------------------- --------------------------------------------------------------------------------------------------------------- energy-costs ProfitOrLossReporter Energy costs over the reporting window, from a power/energy sensor and a consumption price sensor (costs are positive). self-consumption PandasReporter Share of produced energy consumed on-site, from a production and a consumption sensor. Print a template in full (e.g. to pipe it to a file and edit it): .. code-block:: bash $ flexmeasures show report-templates --name self-consumption > self-consumption.yml In a template's parameters skeleton, you fill in your own sensors by replacing the ``FILL_IN`` placeholders (a clear validation error points out any placeholders you left unfilled). The templates also recommend a rolling reporting window (``start-offset``/``end-offset`` fields, reporting on the previous day), for recurring use. You can pass a template directly to ``flexmeasures add report`` or ``flexmeasures add automation --type reports`` with the ``--template`` option. The template then acts as defaults: an explicitly given ``--reporter`` and any top-level keys in your ``--config``/``--parameters`` files override it, and if you provide any timing fields yourself (``start``/``end``/offsets, in the parameters or as CLI options), the template's recommended timing fields are dropped. For example, this sets up a daily self-consumption report in which only the sensors needed to be filled in: .. code-block:: bash $ echo " input: - name: production sensor: 1 - name: consumption sensor: 2 output: - name: self-consumption sensor: 3 " > parameters.yml $ flexmeasures add automation --asset 3 --name "Daily self-consumption report" \ --cron "0 1 * * *" --type reports --template self-consumption --parameters parameters.yml