Robustness against imperfect data

[nhoening]

During the last TSC, we found that one stepping stone for FlexMeasures to be considered mature (-> v1.0) could be more robustness against imperfect input data.

The best point to deal with this might be when we accept data from the outside (via the API usually), here & here, which both use data/utils/save_to_db().

In particular:

• handle gaps/NaN Q: more precisely, where does this go bad?
• handle non-exact timings, i.e. round event_start times like 23:59:59 to 00:00:00. In Pandas and/or with Python

Happy to hear more details about each of these problems, or if I missed one. And of course suggestions how to treat them.

P.S. The more we do when accepting data, the better it would be to move accepting data into a job - the API endpoints need to return swiftly. api_utils.save_and_enqueue() could be doing that (create a data-entry job) instead of calling save_to_db() right away. Also, note that no call to this function currently passes forecasting jobs.

[nhoening]

Noting that one of the options is to move to TimescaleDB. It probably can help here. But in the maintainer team of FlexMeasures, we have no experience with it. We have an issue for that topic here, but have not researched issues there like querying inexact data, or aggregation suuport.

[nhoening]

Hi Ragnar, just to be sure - you are running the new forecasting infrastructure in main, right?

[Ragnar-the-mighty]

No running the old forecasting

[nhoening]

We will make a new RC image of the upcoming FlexMeasures v0.30 today, where forecasting is updated. Maybe you'd like to try using that?

I'm not sure if the load_series() function is still part of the new infrastructure at all. Maybe @BelhsanHmida knows?

[BelhsanHmida]

Hello guys, load_series() is not utilized in the new forecasting infrastructure. also the new is more robust against data issues (duplicates and missing data). so @Ragnar-the-mighty looking forward to hear your feedback on it.

[Flix6x]

The RC image is released here.

[beralt85]

Not sure about the best and most streamlined solution, but I can describe the context of my problem:
For a HEMS setup, I will be relying on data sources from home-assistant.
When posting data to flexmeasures, I already try to align the timestamp, so that I do not have the problem with misaligned timestamps.
However, I still have the problem with data gaps:
For instance, in the night my inverter shuts down the connection to the PV modules and thus, the corresponding sensor does not get any data in the night. I then have to manually fill with zeros to get a complete time-series. This is not very robust.

In particular, for time-series with gaps, scheduling still seems to work, while forecasting (at least with the current production code and the old forecasting module) fails:

Internally, pandas tries to infer frequency and fails:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/pandas/core/arrays/datetimelike.py", line 2122, in _validate_frequency
    raise ValueError
ValueError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/rq/worker.py", line 1659, in perform_job
    return_value = job.perform()
                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/rq/job.py", line 1318, in perform
    self._result = self._execute()
                   ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/rq/job.py", line 1376, in _execute
    result = self.func(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flexmeasures/data/services/forecasting.py", line 234, in make_rolling_viewpoint_forecasts
    forecasts, model_state = make_rolling_forecasts(
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/timetomodel/forecasting.py", line 86, in make_rolling_forecasts
    feature_frame: pd.DataFrame = construct_features(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/timetomodel/featuring.py", line 49, in construct_features
    outcome_var_df = specs.outcome_var.load_series(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/timetomodel/speccing.py", line 190, in load_series
    data = self.resample_data(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/timetomodel/speccing.py", line 302, in resample_data
    data.index.freq = expected_frequency
    ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pandas/core/indexes/datetimelike.py", line 106, in freq
    self._data.freq = value  # type: ignore[misc]
    ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pandas/core/arrays/datetimelike.py", line 2052, in freq
    self._validate_frequency(self, value)
  File "/usr/local/lib/python3.12/dist-packages/pandas/core/arrays/datetimelike.py", line 2133, in _validate_frequency
    raise ValueError(
ValueError: Inferred frequency None from passed values does not conform to passed frequency 5min

[nhoening]

The new forecasting in the dev image we shared is supposed to deal with gaps in data well. It would be great to hear if you can test the same data with that.