add sdpi p history service option

[d-gregorczyk]

☑ Mandatory Tasks

The following aspects have been respected by the pull request assignee and at least one reviewer:

• Changelog update (necessity checked and entry added or not added respectively)
  • Pull Request Assignee
  • Reviewer

[PaulMartinsen]

Suggest changing R0512 to something like:

The provider shall transmit hm:HistoricMdib before transmitting each contiguous MDIB Sequence of Changes in the range requested by a consumer.
Note: this allows a provider to skip over gaps in its historical record while providing a MDIB Configuration for the consumer to apply changes to.

Possible need to tweak R0514 to something like:

For an MDIB Sequence of Changes that is requested from a SOMDS Provider, the SOMDS Provider shall provide exactly one MDIB Configuration in hm:ChangeSequence/hm:HistoricMdib for each contiguous MDIB sequence of changes.

Reasoning:

I was thinking more about the case where a provider starts providing history information but some of the history becomes unavailable before it is delivered. This could occur when:

1. the consumer requests all (or most) of the history available, and
2. the provider has limited memory and must discard old history records to make room for new changes to the mdib.

Some available options :

1. End subscription:

• The provider ends the history subscription, possibly with a (new) fault message.
• The consumer starts a new history subscription and gets the next lot of data.
• Repeat above steps each time items are lost from this history.

2. Send with gaps:

• The provider skips over the missing data and continues with what it has.
• This is (probably?) useless to the consumer who doesn't have a mdib starting point to apply changes to.

3. Send with gaps and starting mdib

• The provider sends a full mdib snapshot (if this was requested by the original subscription)
• Provider continues with change reports.

I quite like option 3 because:

• it avoids multiple subscriptions (with potential race conditions) and doesn't require any special handling of fault messages on the consumer end.
• the provider does its best to provide the information requested and while still letting the consumer create coherent mdib state.
• its more likely to avoid lost history than if the consumer has to re-subscribe. The consumer can unsubscribe if it doesn't want any more data following a gap.

If the consumer didn't request a baseline mdib, we could sensibly fallback to option 1: end the subscription.

[PaulMartinsen]

I would quite like to be able to omit waveform streams from the history because:

1. they use up a lot of memory,
2. an hour old waveform stream may often not be relevant anymore, and
3. a deeper history of states other than waveform streams may be more relevant than a shallow history with waveform streams.

But the general case, where providers may omit any information, seems quite complicated and could be problematic. That is, important context may be lost.

Could the provider be allowed to omit waveform streams either:

1. without disclosing this to the consumer (except perhaps in the documentation), or
2. by sending RealTimeSamleArrayMetricState with @Off.

Or maybe there is no value in waveform streams in the historic record and they could just be omitted completely? That is, remove msg:WaveformStream from the HistoricReportType in HistoryModel.xsd

The main problem seems to be missing state versions (R1005), although I think we need to get rid of that one anyway.

[PaulMartinsen]

I wonder if the <hm:LanguageFilter/> here could be more generic because histories are potentially large and some one to manage that may be helpful.

So, for example, <hm:Filter Dialect="urn:oid:language-oid"/>. With 0 or more allowed. And that providers must also support http://docs.oasis-open.org/ws-dd/ns/dpws/2009/01/Action and should support sdc.filter://Handle. (must and should for consistency with DWPS and 20701 R0039 ).

It would be nice if absence of the action or handle filter implied including everything. Is it a problem that's inconsistent with the language filter behaviour?

[d-gregorczyk]

What do you mean by "more generic"? Currently, a consumer may provide a list of languages to filter against, or, omits the language filter if it requests all languages.

[d-gregorczyk]

There is no support for action and handle based filters here - the subscribe is rather the request input to a stream of information that is send back by the provider. As HTTP 1.1 does not support multiple responses to a single request, we introduced this subscription workaround.

[PaulMartinsen]

I was thinking we ought to require support of action filters and optional support for handle filters (must and should for consistency with other profiles/standards). This would allow a consumer to select only the information they care about. For example, an alarm panel might have no interest in waveform streams (even if they were available). So more generic i was just thinking why is language filtering supported but not other filters?

[d-gregorczyk]

Initially, the history service was not supposed to work that way. You always request a full disclosure of reports as a provider needs to store them anyway as otherwise it cannot respond to queries that may select full disclosure of data.

If the provider is not capable of storing everything, it just responds with incomplete data no matter the input filtering. This btw. allows for the provider to efficiently store historic reports (e.g. pre-compressed).

So more generic i was just thinking why is language filtering supported but not other filters

There is also a filter to skip initial MDIBs. It is basically condensed to the essentials:

• Backfilling: no localized texts and and initial MDIBs required
• Full disclosure: initial MDIB and localized texts (perhaps filtered by a certain set of languages) included

We may add further filters, but those will increase complexity, and complexity increases variance which in turn reduces the degree of achievable interoperability. If there is pressing need by the stakeholders to support "action filters (akap report type filters)" or "handle filters", we may add them as additional filters (which can also be done by introducing further HistoryQuery flavors).

[PaulMartinsen]

In designing libraries I think it is worth anticipating use cases that haven't been codified. Perhaps, for example, there is a consumer that wishes to retrieve a day's history from a provider that's been emitting updates every 10 seconds. With maximum message size of 2MB, that would be transfer of about 17GB. At which point we'd have to start wondering if soap was the cleanest approach (hehe), but if we're stuck with soap for a while then filtering on actions and/or handles might be useful.

But I don't think there's a right answer to this one. Perfection is the barrier to progress so if we have to pick I choose progress. There seems an opportunity to incorporate some potentially useful filtering at this point, but I don't feel strongly about it.

[PaulMartinsen]

Do we need to deal with time epochs here? I can see three options:

1. drop support for requesting history by time, or use relative time like subscriptions.
2. leave the provider free to decide which time-reference frame it should use when comparing mdib timestamps and history query time-stamps.
3. add attributes to select epoch.

1 seems best to me, but I'm not sure what the use cases are for this one. 2 seems okay as well because time is supposed to be UTC time, which is independent of non-slewing time-adjustments. If 2, perhaps a note along the lines : "The provider may not reliably select the correct change reports for time ranges that span an abrupt time adjustment".

[PaulMartinsen]

Is there any value for consumers in allowing hm:Range and/or either range limit to be omitted to ask for:

• everything available (hm:Range omitted),
• everything up to a specific mdib version/time @StartVersion or @StartTime omitted,
• everything from a specific mdib version/time to the time the subscription was created (@EndVersion or @EndTime omitted)

[d-gregorczyk]

Please let us not over-complicate the handling here. Selecting epochs would instantly require each SDPi implementation to support epochs, which is not going to happen. We either remove selection of time frames or introduce some sort of error handling to signify that a time-based request cannot be fulfilled due to past time synchronization issues.

[PaulMartinsen]

Time is quite useful for testing; saves figuring out all the mdib versions. Otherwise I don't see the use cases. It could be okay to just have a warning that if you use time selection then it might not be perfect.

Does omitting parts of the range over complicate? I wonder if it could simplify things from the consumer side by not having to figure out what range to provide.

[PaulMartinsen]

The sequence diagrams weren't showing up for me. It looks like the action isn't copying them into the supplement from .ci\asciidoc-converter\images when it makes the zip file I downloaded from https://github.com/IHE/DEV.SDPi/actions/runs/15761385158

[PaulMartinsen]

Is it worth pointing out that the consumer can also control the data rate it accepts the information using the HTTP protocol (i.e., delaying the response code)?

[d-gregorczyk]

Not sure if it's worth. The main issue we formerly had were the providers being overwhelmed by archive requests being too big to transmit in one chunk. If you drop me some lines, I can add it to the text.

[PaulMartinsen]

Sorry for delay; some days it feels like there's more going on than a Saudi Arabian sword juggler.

How about, after …avoid exceeding data size notification limitations:

Similarly, the SOMDS Consumer may slow the data rate that it receives information from the SOMDS Provider by delaying acknowledgement of notifications (e.g., the HTTP response message to the provider's notification for the MDPWS protocol). Excessive delays, either by the SOMDS Provider or SOMDS Consumer, may lead to timeouts.

[PaulMartinsen]

I suggest that (for consistency with other profiles/standards) providers must support action based filtering and should support handle based filtering for the history service. So the protocol should support this somehow (see suggestion for one approach in another comment).

Because the history service is a foundational feature that may be used by large and small participants in the future, a little flexibility seems worthwhile even if the business cases aren't 100% clear at the beginning. I know low-resource providers would prefer that consumers only request what they need but providers have no control over that and best they can do is support the mechanisms.

Implementation doesn't seem hard to me if there is a mechanism in the protocol but I don't feel very strongly about this. Mostly because it requires consumers to use it for providers to benefit, which may not happen in practice.

[d-gregorczyk]

I suggest that (for consistency with other profiles/standards) providers must support action based filtering and should support handle based filtering for the history service.

Action based filtering does not have any meaningful effect on this port type as there is only one notification in there: To enable subscribing to one or more Notifications exposed by a HOSTED SERVICE, this profile defines a Filter Dialect designated "http://docs.oasis-open.org/ws-dd/ns/dpws/2009/01/Action". You would either be able to select everything or nothing - to select nothing, just do not send s subscribe :-)

Handle based filtering may be applicable to the history service, however, it is an optional feature in SDC Glue. I do not see embedded devices performing handle-based filtering on huge amounts of historic data. The limited filtering comes at a cost: less flexibility. But it provides embedded devices with the ability to efficiently store their historic data without performing expensive filtering operations when requested. If we receive proof that such an operation is efficiently processable and needed by the market today, I am totally happy to add it as a mandatory feature to the history service. The things for which there is no demand by the market, I would like to keep it out the spec for as long as possible. It's all big enough already...

a little flexibility seems worthwhile even if the business cases aren't 100% clear at the beginning.

Flexibility is always given - the current filter can be replaced by another one that supports nifty filtering options if we realize, the limited filtering does not suffice. Let's be evolutionary over revolutionary. ;-)

[PaulMartinsen]

For action based filtering, I'm thinking of the StateEventService/EpisoidicMetricReport, StateEventService/EpisoidicAlertReport, ContextService/EpisoidicContextReport, etc. That is a consumer might decide they only want alert history from the day, then retrieve metric history for particular epochs around an alert.

We currently do handle based filtering on embedded devices. Mostly because 20701:R0039 said we should. Its only useful if consumers follow 20701:R0037 though and so far I haven't found any to validate interoperability of our prototype implementation unfortunately.

We don't have additional overhead to apply the filter (that is, everything is filtered anyway) and it would be fine in our prototype history service if there was a way to do communicate selection. Mdibs in embedded devices may be smaller making handling filtering easier, though that's also the use-case that could benefit from the most from reduced traffic. Maybe handle filtering is only useful for embedded devices? Assuming consumer's take advantage of it (and there's no way to tell if it is available; makes me think should requirements are a little pointless for feature support).

Anyway, your right; new features can always be added and things can be changed. That can take a while, though, and this is just one opportunity to do that but I don't feel strongly about it either way.

[PaulMartinsen]

I haven't looked at localization really, but reading this I wondered R0509 requires providers to send localized text even if they only have static localized text. Can't the consumer just get the localized text from the localized text service in this case? I think this is worth thinking through because:

1. many (most?) providers may only have static localized text
2. there might be a lot of localized text so it seems quite inefficient to send it to every history subscription if it never changes. Unless we trying to encourage everyone to support gzip :).

[d-gregorczyk]

BICEPS cannot express if localized texts are static or dynamic, hence you need the option to request localized texts from the past, though you are right and almost all devices I know never change their texts (at least not between major updates).

[PaulMartinsen]

I think most (all?) localized text changes require description modification events. So if that service isn't present a consumer could know the text is static :-).

[d-gregorczyk]

I would quite like to be able to omit waveform streams from the history because:

1. they use up a lot of memory,
2. an hour old waveform stream may often not be relevant anymore, and
3. a deeper history of states other than waveform streams may be more relevant than a shallow history with waveform streams.

But the general case, where providers may omit any information, seems quite complicated and could be problematic. That is, important context may be lost.

Could the provider be allowed to omit waveform streams either:

1. without disclosing this to the consumer (except perhaps in the documentation), or
2. by sending RealTimeSamleArrayMetricState with @Off.

Or maybe there is no value in waveform streams in the historic record and they could just be omitted completely? That is, remove msg:WaveformStream from the HistoricReportType in HistoryModel.xsd

The main problem seems to be missing state versions (R1005), although I think we need to get rid of that one anyway.

I think waveforms are very important to request as consumers may use the data to calculate trends. It is also important to recreate a coherent flow of data for backfilling purposes.

We may add a flag somewhere. What we should not do is to alter descriptors and send them out as MDIB snapshots should not be modified from it's original state. If leaving the information out the data stream is not conflicting with any other requirement, we can also just silently skip any data from the report stream a provider is not capable to serve.

[d-gregorczyk]

Suggest changing R0512 to something like:

The provider shall transmit hm:HistoricMdib before transmitting each contiguous MDIB Sequence of Changes in the range requested by a consumer.
Note: this allows a provider to skip over gaps in its historical record while providing a MDIB Configuration for the consumer to apply changes to.

Possible need to tweak R0514 to something like:

For an MDIB Sequence of Changes that is requested from a SOMDS Provider, the SOMDS Provider shall provide exactly one MDIB Configuration in hm:ChangeSequence/hm:HistoricMdib for each contiguous MDIB sequence of changes.

Reasoning:

I was thinking more about the case where a provider starts providing history information but some of the history becomes unavailable before it is delivered. This could occur when:

1. the consumer requests all (or most) of the history available, and
2. the provider has limited memory and must discard old history records to make room for new changes to the mdib.

Some available options :

1. End subscription:

• The provider ends the history subscription, possibly with a (new) fault message.
• The consumer starts a new history subscription and gets the next lot of data.
• Repeat above steps each time items are lost from this history.

2. Send with gaps:

• The provider skips over the missing data and continues with what it has.
• This is (probably?) useless to the consumer who doesn't have a mdib starting point to apply changes to.

3. Send with gaps and starting mdib

• The provider sends a full mdib snapshot (if this was requested by the original subscription)
• Provider continues with change reports.

I quite like option 3 because:

• it avoids multiple subscriptions (with potential race conditions) and doesn't require any special handling of fault messages on the consumer end.
• the provider does its best to provide the information requested and while still letting the consumer create coherent mdib state.
• its more likely to avoid lost history than if the consumer has to re-subscribe. The consumer can unsubscribe if it doesn't want any more data following a gap.

If the consumer didn't request a baseline mdib, we could sensibly fallback to option 1: end the subscription.

Rather than trying to mitigate this critical error, I would recommend to answer with an error. The more we talk about it, the more I am in favor of creating some kind of error feedback. I dislike using the subscription/SOAP faults as they are very specific to the transport layer, but the errors we see here are specific to the application layer.

[PaulMartinsen]

I think waveforms are very important to request as consumers may use the data to calculate trends. It is also important to recreate a coherent flow of data for backfilling purposes.

Embedded devices may not be able to provide a useful history service (i.e. more than a few minutes) if waveforms can't be left out due to memory constraints. I agree that the waveforms are important, but if it comes down to all or nothing we might have to go with nothing. I haven't explored compression yet though.

We may add a flag somewhere. What we should not do is to alter descriptors and send them out as MDIB snapshots should not be modified from it's original state. If leaving the information out the data stream is not conflicting with any other requirement, we can also just silently skip any data from the report stream a provider is not capable to serve.

That's good; I'm not keen on description modifications either. They are so complicated. This might be one best left to see how prototype implementations cope. Silently dropping other things (e.g., alerts) could be quite bad though. Maybe a providers shall consider the risks of leaving out anything from the data stream?

[PaulMartinsen]

If the consumer didn't request a baseline mdib, we could sensibly fallback to option 1: end the subscription.

Rather than trying to mitigate this critical error, I would recommend to answer with an error. The more we talk about it, the more I am in favor of creating some kind of error feedback. I dislike using the subscription/SOAP faults as they are very specific to the transport layer, but the errors we see here are specific to the application layer.

I'm more fault agnostic. I dislike them all :-).

I think we are allowed to extend SOAP faults, defining our own in the WSDL. And, fundamentally, faults are just QName values, which are slotted into the [Code], or possibly [SubCode], along with optional descriptions and other details. Each protocol may have its own way to transmit the faults. Are QNames universal? Hopefully we don't need Types from 10101.

[PaulMartinsen]

Suggest changing R0512 to something like:

The provider shall transmit hm:HistoricMdib before transmitting each contiguous MDIB Sequence of Changes in the range requested by a consumer.
Note: this allows a provider to skip over gaps in its historical record while providing a MDIB Configuration for the consumer to apply changes to.

Rather than trying to mitigate this critical error, I would recommend to answer with an error. The more we talk about it, the more I am in favor of creating some kind of error feedback. I dislike using the subscription/SOAP faults as they are very specific to the transport layer, but the errors we see here are specific to the application layer.

Hmm. Although this was about possible gaps in the history sequence. Here I was thinking this is a clever solution.

That is, if there is a gap in the providers history, the consumer possibly just wants to do a new subscription starting from the gap MDIB version (which it doesn't really have a way to find yet). So the provider can just skip the gap, but it needs to send the MDIB as a starting point for the contiguous section so the consumer can apply changes from the the report sequence.

If the consumer dislikes gaps it can unsubscribe when it encounters one. Otherwise, this fits the philosophy of the provider delivering as much as it can from the history. And, gaps ought to be rare, but perhaps not as rare as non-slewing time adjustments.

Well, I thought it was clever :).

[d-gregorczyk]

If the consumer didn't request a baseline mdib, we could sensibly fallback to option 1: end the subscription.

Rather than trying to mitigate this critical error, I would recommend to answer with an error. The more we talk about it, the more I am in favor of creating some kind of error feedback. I dislike using the subscription/SOAP faults as they are very specific to the transport layer, but the errors we see here are specific to the application layer.

I'm more fault agnostic. I dislike them all :-).

I think we are allowed to extend SOAP faults, defining our own in the WSDL. And, fundamentally, faults are just QName values, which are slotted into the [Code], or possibly [SubCode], along with optional descriptions and other details. Each protocol may have its own way to transmit the faults. Are QNames universal? Hopefully we don't need Types from 10101.

QNames ist a broken concept outside XML Schema boundaries. It ties a model to XML, which I find very leaky. Please check the error element I added to the History Model Schema. Idea: strong typed set of "errors" (or rather issues on the response to a request), plus a generic "other" type. Do you have further issues to be included in the error model (see hm:ResponseStatusType)?

[d-gregorczyk]

Well, I thought it was clever :).

Hehe, I bet it is! To me more often it is the question of complexity vs. benefit. Keep it simple and stupid is my highest design goal, complexity only if unavoidable, and complication must be avoided at all costs. Btw. looking at SDC, yikes, some of these goals were not fully met...

[PaulMartinsen]

Well, I thought it was clever :).

Hehe, I bet it is! To me more often it is the question of complexity vs. benefit. Keep it simple and stupid is my highest design goal, complexity only if unavoidable, and complication must be avoided at all costs. Btw. looking at SDC, yikes, some of these goals were not fully met...

Actually, here I was inspired by your idea not to generate faults if there was nothing to do with the information. So, yes it is simple too.

[ToddCooper]

@d-gregorczyk - Today @JavierEspina and I merged the "Vienna PR" into master. This is the one that updates the Required Actor Grouping tables + added some additional profile options (gateway directionality, real-time waveforms, etc.).

Could you sync the History Service branch with master to bring in those changes? With that, I'll update the TF-1 in this branch to support adding the History Service option to the SDPi-P profile, per what was just added.

[ToddCooper]

@d-gregorczyk - Earth to David ...

[d-gregorczyk]

David to earth! :-)

[d-gregorczyk]

Done.

@d-gregorczyk - Today @JavierEspina and I merged the "Vienna PR" into master. This is the one that updates the Required Actor Grouping tables + added some additional profile options (gateway directionality, real-time waveforms, etc.).

Could you sync the History Service branch with master to bring in those changes? With that, I'll update the TF-1 in this branch to support adding the History Service option to the SDPi-P profile, per what was just added.