Flooring event_start times

[nhoening]

Sometimes we might get sent data that is not on times which represent clean sensor resolution. That blocks FlexMeasures currently in things like scheduling. Later, we could make FM more robust, but flooring data we are sent in the POST endpoint.

We might add an option, so that users could choose between flooring/rounding/ceiling. But will that add a lot of value?

Important decision: Should we move to correcting event_start as the new default, and remove the option to send data as-is?

[Flix6x]

Related: #10

[nhoening]

Question: Should we combine flooring of slightly inexact event_starts with rejecting very inexact event_starts?

• 10:00:40-> 10:00:00
• 10:01:40-> ERROR?

In this example, we floor if we find under a minute of deviation, but more than that is an error.

It could also be determined by the resolution, e.g. an error only if delta > event_resolution / 5.

The above example would be an error if event_resolution is PT5M, but we would floor to 10:00:00 if resolution is PT15M.

This proposal probably goes together with both flooring and ceiling.

[Flix6x]

I am leaning towards not doing the error thing, at least not as a default.

One thing that I think complicates this issue is that multiple events might get floored to the same time. For example, incoming data may look like:

• 10:00:40, 1
• 10:02:40, 2
• 10:04:40, 3
• 10:06:40, 4
• 10:08:40, 5
• 10:10:40, 6

Flooring might lead to a duplicate SQLAlchemy error, if these records share a belief time.

Instead, I suggest that we resample to the sensor resolution, with some interpolation policy. For instance, if that is 5 minutes, we could get:

• 10:00:00, mean(1, 2, 3)
• 10:05:00, mean(4, 5)
• 10:10:00, 6

We could later have different interpolation policies if needed (time-weighted, step-wise vs linear). And for a single value, this policy would be equivalent to flooring.

The interpolation policy, and also a possible error policy, feel like they would better be modelled as sensor configurations.

[nhoening]

Alright, so we would implement some aggregation, but only if all data for a time slot comes in one go. So not a full support.

[Flix6x]

Exactly.

[nhoening]

Important to differentiate the two ways we get data:

• a POST endpoint with a start time, duration and just a list of values. Here, we only need to floor the start time and be done with it.
• a POST endpoint with an uploaded file. Here, the whole datetime column needs flooring. The resolution of this data will still be checked against the sensor resolution, and resampling would possibly commence.

@Flix6x In your example above, how will this dataset with 2-minute resolution for a 5-minute sensor be sent to us? Am I overlooking something?

[Flix6x]

start: 10:00:40
duration: PT12M
values: [1, 2, 3, 4, 5, 6]

Or by file.

The first option only supports sending a time series with fixed resolution, whereas the file could contain irregular steps.

[nhoening]

We'd floor the start time in the first example, treating it the same as before otherwise.

For upload, I guess we could demand a dataset where a resolution can be determined, and only for those accept gaps.

[Flix6x]

I started a tech spike in #1828, where we can extend the number of test cases.