From 3a42a15d405f1533d8ae628c2bcda250ed651b0f Mon Sep 17 00:00:00 2001 From: James Sandford Date: Tue, 11 Nov 2025 14:00:18 +0000 Subject: [PATCH 1/9] Add ADR regarding signalling of retention time --- docs/README.md | 1 + docs/adr/0043-signalling-retention-time.md | 246 +++++++++++++++++++++ 2 files changed, 247 insertions(+) create mode 100644 docs/adr/0043-signalling-retention-time.md diff --git a/docs/README.md b/docs/README.md index 1717741e..f7356036 100644 --- a/docs/README.md +++ b/docs/README.md @@ -78,5 +78,6 @@ For more information on how we use ADRs, see [here](./adr/README.md). | [0040](./adr/0040-tag-usability-enhancements.md) | Tag Usability Enhancements | | [0041](./adr/0041-require-explicit-framerate.md) | Requiring explicit frame rates | | [0042](./adr/0042-uncontrolled-object-instance-labels.md) | Make `label` Mandatory for Uncontrolled Object Instances | +| [0043](./adr/0043-signalling-retention-time.md) | Signalling retention time | \* Note: ADR 0004a was the unintended result of a number clash in the early development of TAMS which wasn't caught before publication diff --git a/docs/adr/0043-signalling-retention-time.md b/docs/adr/0043-signalling-retention-time.md new file mode 100644 index 00000000..c9dae543 --- /dev/null +++ b/docs/adr/0043-signalling-retention-time.md @@ -0,0 +1,246 @@ +--- +status: "proposed" +--- +# Signalling Retention Time + +## Context and Problem Statement + +It is common in media workflows to have a continuous ingest where media is only retained for a fixed period of time. +Such systems are commonly known as "Loop Recordings" or "Circular Buffers". +These systems are often used where storage is limited, or where the value of the media diminishes over time. +They may also be used as part of a workflow where valuable content is explicitly selected for longer term storage, with anything else being automatically deleted. +Or where media is transcoded or otherwise processed within a set window following capture/ingest. + +Cloud-based object storage does afford greater flexibility in terms of the scale and duration for which media may be stored. +But such workflows are still useful to avoid the ongoing cost and resource use of storing media indefinitely. + +TAMS' strong timing, support for short segments, and event mechanisms make implementing such features trivial. +Such functionality can even be implemented efficiently as a client of TAMS, rather than as a deeply integrated feature of the service. +An [open-source implementation](https://github.com/aws-samples/time-addressable-media-store-tools?tab=readme-ov-file#deploylooprecorder) is available in the AWS TAMS Tools repository. +This implementation uses a Flow tag with the name `loop_recorder_duration` which takes a duration in seconds. + +Given the prevalence of these workflows, this ADR discusses if and how TAMS might explicitly support such functionality. + +Firstly, some generally applicable discussion of how this feature may fit with the wider TAMS design. + +The TAMS specification has always been tightly scoped as a media store API. +Its purpose is to facilitate the storage, retrieval, and advertisement of media. +It is often stated that "TAMS is not a MAM", referring to the complex features of a Media Asset Management system. +The assumption being that, for more complex use cases, a MAM layer may sit on top of a TAMS service. +Or a MAM may provide a TAMS API as an interoperable means to interact with the MAM at a basic level. +Loop recording could be consider a concern for another system, such as a MAM. + +To judge this position, we might consider what other types of media management TAMS is currently responsible. + +TAMS implementations are currently responsible for the reference counting of Media Objects and deleting the Objects from storage once it is no longer references by any Flow Segments. +It is vital for the management of Media Objects given that they have no conceptual "owner" within the TAMS system, and thus no other system responsible for their deletion. +This operation is performed by reference counting Object IDs as opposed to explicit signalling by the client. +It requires no configuration +This is also a case where an instance may be considered to be performing its own maintenance routines. +Loop recording, on the other hand, may be configured via data in the TAMS Service (tags or a field in the core spec) but actually enacted by a secondary system. + +Webhooks are an example where TAMS may be configured in a way that influences an external system. +But only in that they set where a stream of event data is to be sent, and what data will be included in that stream. +In that sense, it may be considered a form of the advertisement of data in a TAMS system rather than the explicit configuration of a system that modifies that data. + +Webhooks are also an example of an optional feature in TAMS. +The availability of this feature on a given store implementation is advertised via the `event_stream_mechanisms` parameter on the `/service` endpoint. + +The proposed approach to Fine-Grained Auth in TAMS is an example where configuration in TAMS (a list of groups with permissions on a TAMS resource) will be used to influence the behaviour of a non-core feature. +Fine-grained auth may be tightly integrated into a Service implementation or be implemented in a secondary service (i.e. an auth proxy). +This signalling will initially be implemented in tags. +Similar considerations about how to manage an optional feature apply to Fine-Grained Auth. + +With this in mind, loop recording configuration would present a step beyond the types of functionality currently provided by the TAMS specification. + +TAMS clients have shown that there is a desire to co-ordinate loop recording via metadata in TAMS. +But we should consider the implications of the options for doing so on implementations. +Should loop recording be an optional feature? +If so, how should the availability of that feature be advertised, if indeed it is advertised? +Should this feature be deeply integrated into TAMS, or should we expect that this feature continues to be provided by a secondary system? + +In addition to loop recording, another desirable type of automated clean-up is the deletion of Flows at or after a period of time. +This ADR will also explore some options for such a feature. + +## Considered Options + +* Option 1: Leave loop recorder signalling up to the implementer +* Option 2: Explicitly make loop recording the responsibility of the client which would otherwise configure such a feature +* Option 3: Define a well known tag in the tags listing +* Option 4a: Define an optional parameter in the core specification, with support signalled +* Option 4b: Define an optional parameter in the core specification, with support not signalled +* Option 4c: Define an optional parameter in the core specification, with mandatory support +* Option 5a: Describe a method of deleting Segments after a period of time from their registration +* Option 5b: Describe a method of deleting Segments at a time offset from their TimeRange end +* Option 5c: Describe a method of deleting Segments to maintain a maximum duration for a Flow +* Option 6a: Describe a method of deleting an entire Flow at a specific time +* Option 6b: Describe a method of deleting an entire Flow at a time offset from its creation time +* Option 6c: Describe a method of deleting an entire Flow at a time offset from its Segments updated time +* Option 6d: Describe a method of deleting an entire Flow at a time offset from the most recent of Segments updated or Metadata updated time +* Option 7: Describe potential implementation architectures in an Application Note + +## Decision Outcome + +Chosen options: + +* Option 3: Define a well known tag in the tags listing +* Option 5b: Describe a method of deleting Segments at a time offset from their TimeRange end +* Option 6d: Describe a method of deleting an entire Flow at a time offset from the most recent of Segments updated or Metadata updated time +* Option 7: Describe potential implementation architectures in an Application Note + +This combination of options shall result in a clearly defined approach to implementing loop recording functionality. +It avoids adding any features to the core specification which cannot be relied upon at all times. +It avoids blurring the line between the core TAMS Service and wider ecosystem by communicating features potentially provided by that wider ecosystem via the core specification. +It does, however, require those deploying and using TAMS systems to know out-of-band what additional features (i.e. those beyond the core spec) are and aren't available for their use. + +### Implementation + +{Once the proposal has been implemented, add a link to the relevant PRs here} + +## Pros and Cons of the Options + +### Option 1: Leave loop recorder signalling up to the implementer + +This option would see loop recorder signalling and functionality to be defined by the implementers. +TAMS would not define any specific signalling mechanism. + +* Good, because it avoids a spec change +* Good, because it avoids adding a feature to the spec that would likely need to have optional support +* Bad, because it may result in fragmented compatibility around a common function + +### Option 2: Explicitly make loop recording the responsibility of the writing client of loop recording Flows + +This option would see loop recorder signalling and functionality to be defined by the implementers. +TAMS would not define any specific signalling mechanism. +TAMS would explicitly state that such functionality is the responsibility of the client writing loop recording Flows. + +* Good, because it avoids a spec change +* Good, because it avoids adding a feature to the spec that would likely need to have optional support +* Good, because it provides clear guidance on responsibility for this feature +* Good, because it avoids fragmented compatibility around a common function +* Bad, because it couples writing and deleting responsibility in this case +* Bad, because it may require some organisations to use different ingest clients for different purposes + +### Option 3: Define a well known tag in the tags listing + +This option would see the signalling/configuration of the feature be via a well known Flow tag. + +* Good, because it avoids a spec change +* Good, because it avoids adding a feature to the spec that would likely need to have optional support +* Good, because it avoids fragmented compatibility around a common function +* Good, because it does not make an assumption about where the feature would be implemented (i.e. in the service, or elsewhere) +* Neutral, because it avoids a proliferation of features in the core spec +* Neutral, because it results in a common feature being signalled via tags +* Bad, because it requires support for the feature to be verified on deployment of clients which use it + +### Option 4a: Define an optional parameter in the core specification, with support signalled + +This option would see the configuration of the feature be via a new parameter in the core spec. +This option would see support of this feature be optional. +This would see support for the feature signalled at the `/service` endpoint, similar to the signalling of support for event stream mechanisms. + +* Good, because it avoids fragmented compatibility around a common function +* Neutral, because it requires a spec change +* Neutral, because it may tend towards a proliferation of features in the core spec +* Neutral, because it avoids a common feature being signalled via tags +* Bad, as the requirement to signal support for the feature would likely couple this feature to the Service Implementation + +### Option 4b: Define an optional parameter in the core specification, with support not signalled + +This option would see the configuration of the feature be via a new parameter in the core spec. +This option would see support of this feature be optional. +This would see no signalling of support for the feature via the API. +It would be up to integration engineers to ensure that the feature is provided by the Service Implementation, or some other mechanism, when they deploy a client that needs to use it. + +* Good, because it avoids fragmented compatibility around a common function +* Neutral, because it requires a spec change +* Neutral, because it may tend towards a proliferation of features in the core spec +* Neutral, because it avoids a common feature being signalled via tags +* Bad, because it requires support for the feature to be verified on deployment of clients which use it + +### Option 4c: Define an optional parameter in the core specification, with mandatory support + +This option would see the configuration of the feature be via a new parameter in the core spec. +This option would see support for this feature be mandatory. + +* Good, because it avoids fragmented compatibility around a common function +* Neutral, because it requires a spec change +* Neutral, because it may tend towards a proliferation of features in the core spec +* Neutral, because it avoids a common feature being signalled via tags +* Bad, because it requires Service Implementations to support a feature that will only be used by a relatively small subset of deployments + +### Option 5a: Describe a method of deleting Segments after a period of time from their registration + +This option would require additional Segment metadata. +We do not currently store the creation time for Segments. +Adding this metadata to every Segment would represent a significant increase in the amount of data stored in TAMS. +It may also result in unexpected behaviour where Segments are written out of order. +This type of behaviour could be more reliably implemented by Option 5c. + +* Neutral, because it requires a spec change +* Bad, because it requires significantly more data to be stored by a TAMS service +* Bad, because it may have unexpected behaviour + +### Option 5b: Describe a method of deleting Segments at a time offset from their TimeRange end + +This option would see an offset specified that will be added to the TimeRange end of a segment to determine when it shall be deleted. +Alternatively, the offset could be subtracted from the current time to be provided as `_)` in the `timerange` query parameter on DELETE requests to the Flow's segment delete endpoint. +This can be run at regular intervals with a single API request and no need to query the segments endpoint. + +* Good, because it doesn't require a spec change +* Good, because it enables loop recorder implementations with minimal requests +* Neutral, because a Flow that is no longer being written to will tend towards all Segments being deleted + +### Option 5c: Describe a method of deleting Segments to maintain a maximum duration for a Flow + +This option would see a target duration specified. +The loop recorder system would get the TimeRange end of the latest segment via a GET request to the `/segments` endpoint with `limit` set to `1` and `reverse_order` set to `true`. +The loop recorder would then subtract the specified duration from end Timestamp of the returned Segment to calculate the end timestamp for the delete request. +This would then be provided as `_)` in the `timerange` query parameter on DELETE requests to the Flow's segment delete endpoint. +The potentially unexpected behaviour of this option is that Flows with gaps may end up with much less content present than the duration specified would imply. + +* Good, because it doesn't require a spec change +* Neutral, because it requires multiple requests per delete cycle +* Neutral, because a Flow that is no longer being written to will maintain a stock of Segments +* Bad, because behaviour may be unexpected on Flows with gaps + +### Option 6a: Describe a method of deleting an entire Flow at a specific time + +This option would see the time for a Flow's automated deletion to be signalled with an absolute time. + +* Good, because it is immediately obvious when a Flow will be deleted +* Neutral, because enacting this requires no further calculations +* Bad, because this mode of operation cannot be statically templated + +### Option 6b: Describe a method of deleting an entire Flow at a time offset from its creation time + +This option would see the time for a Flow's automated deletion to be signalled with an offset to be applied to the Flow's Creation time. + +* Good, because it supports static templating (e.g. via Flow Profiles) +* Neutral, because enacting this requires a simple calculation to identify the actual time for deletion +* Neutral, because it allows Flows to be assigned an explicit life time +* Bad, because it doesn't take into account whether the Flow is still being written to + +### Option 6c: Describe a method of deleting an entire Flow at a time offset from its Segments updated time + +This option would see the time for a Flow's automated deletion to be signalled with an offset to be applied to the Flow's Segments Updated time. + +* Good, because it supports static templating (e.g. via Flow Profiles) +* Good, because it allows Flows to be assigned an explicit life time beyond when Segments are written to it +* Neutral, because enacting this requires a simple calculation to identify the actual time for deletion + +### Option 6d: Describe a method of deleting an entire Flow at a time offset from the most recent of Segments updated or Metadata updated time + +This option would see the time for a Flow's automated deletion to be signalled with an offset to be applied to the Flow's the later of Segments Updated or Metadata Updated time. + +* Good, because it supports static templating (e.g. via Flow Profiles) +* Good, because it captures any type of Flow modification +* Neutral, because enacting this requires a simple calculation to identify the actual time for deletion +* Neutral, because it allows Flows to be assigned an explicit life time beyond when the Flow metadata was last modified + +### Option 7: Describe potential implementation architectures in an Application Note + +This option would be combined with any of the above signalling options. +It would see an Application Note produced describing possible approaches to implementing loop record functionality. + +* Good, because it reduces the design work required on the part of implementers of such functionality From d2366f8c0b4e5bc529a7500b0acaf8aec51f26ab Mon Sep 17 00:00:00 2001 From: James Sandford Date: Tue, 11 Nov 2025 14:23:38 +0000 Subject: [PATCH 2/9] Add the implementation specific `loop_recorder_duration` to the tags listing --- docs/appnotes/0003-tag-names.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/appnotes/0003-tag-names.md b/docs/appnotes/0003-tag-names.md index 7213738b..8a770f99 100644 --- a/docs/appnotes/0003-tag-names.md +++ b/docs/appnotes/0003-tag-names.md @@ -218,6 +218,14 @@ Status: **Experimental** Suggested as a way to build lightweight Attribute-based Access Control in [AppNote0016: Authorisation in TAMS workflows](./0016-authorisation-in-tams-workflows.md). A comma separated list of auth classes used to derive permissions on the Flow. +### loop_recorder_duration + +Status: **Implementation specific** + +Used in the AWS TAMS Tools loop recorder. +The target duration of the Flow in seconds. +Oldest Segments that take the duration over the specified duration will be deleted periodically. + ## Known Source Tags ### hls_exclude From fd1066ef45afaaac7bd178457baa553859d9e395 Mon Sep 17 00:00:00 2001 From: James Sandford Date: Wed, 12 Nov 2025 15:31:45 +0000 Subject: [PATCH 3/9] Add ADR0043 tags to listing --- docs/appnotes/0003-tag-names.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/appnotes/0003-tag-names.md b/docs/appnotes/0003-tag-names.md index 8a770f99..ca9aa5a5 100644 --- a/docs/appnotes/0003-tag-names.md +++ b/docs/appnotes/0003-tag-names.md @@ -226,6 +226,28 @@ Used in the AWS TAMS Tools loop recorder. The target duration of the Flow in seconds. Oldest Segments that take the duration over the specified duration will be deleted periodically. +### flow_retention_offset + +Status: **Proposed** + +Proposed in [ADR0043 - Signalling Retention Time](../adr/0043-signalling-retention-time.md). + +Used to signal the retention time for a Flow. +Specified as a TAMS-compatible [Timestamp](../appnotes/0008-timestamps-in-TAMS.md#timestamp-representation). +The earliest time in [TAI](https://en.wikipedia.org/wiki/International_Atomic_Time) at which the Flow should be automatically deleted is calculated by adding this offset to the most recent of `metadata_updated` or `segments_updated`. +Example: For a Flow with `metadata_updated` of `2023-09-14T09:45:26Z`, `segments_updated` of `2023-09-15T10:23:28Z`, and `flow_retention_offset` of `86400:0` (aka 24 hours) - the `segments_updated` shall be used and the Flow should be deleted a day later at `2023-09-16T10:23:28Z`. + +### segment_retention_offset + +Status: **Proposed** + +Proposed in [ADR0043 - Signalling Retention Time](../adr/0043-signalling-retention-time.md). + +Used to signal the retention time for Flow Segments. +Specified as a TAMS-compatible [Timestamp](../appnotes/0008-timestamps-in-TAMS.md#timestamp-representation). +The earliest time in [TAI](https://en.wikipedia.org/wiki/International_Atomic_Time) at which a given Segment should be automatically deleted is calculated by adding this offset to the end Timestamp component of the Segments TimeRange. +Example: For a Flow using TAI for its timeline, a `segment_retention_offset` of `600:0` will result in Segments being deleted after 10 minutes (specifically, after the end of the Segment). + ## Known Source Tags ### hls_exclude From 9f056c9263e0beee3dcf5c920f1f43f5e6958397 Mon Sep 17 00:00:00 2001 From: James Sandford Date: Fri, 14 Nov 2025 16:36:04 +0000 Subject: [PATCH 4/9] Add AppNote 0019 describing how to implement retention management --- docs/README.md | 1 + ...0019-implementing-rentention-management.md | 83 +++++++++++++++++++ 2 files changed, 84 insertions(+) create mode 100644 docs/appnotes/0019-implementing-rentention-management.md diff --git a/docs/README.md b/docs/README.md index f7356036..f82c4081 100644 --- a/docs/README.md +++ b/docs/README.md @@ -25,6 +25,7 @@ For more information on how we use application notes, see [here](./appnotes/READ | [0016](./appnotes/0016-authorisation-in-tams-workflows.md) | Authorisation in TAMS workflows | | [0017](./appnotes/0017-reuse-of-ids.md) | When to re-use IDs in TAMS and compatible systems | | [0018](./appnotes/0018-managing-multiple-object-instances.md) | Managing Multiple Object Instances | +| [0019](./appnotes/0019-implementing-retention-management.md) | Methods of implementing retention management | ## ADRs diff --git a/docs/appnotes/0019-implementing-rentention-management.md b/docs/appnotes/0019-implementing-rentention-management.md new file mode 100644 index 00000000..65d814cc --- /dev/null +++ b/docs/appnotes/0019-implementing-rentention-management.md @@ -0,0 +1,83 @@ +# 0019: Methods of implementing retention management + +This describes how to implement the automatic deletion of Flows and Segments. + +## Segment Retention + +This approach may be used to control the automatic deletion of Flow Segments. +Segment retention time may be signalled via the `segment_retention_offset` tag. +This tag provides an offset, formatted as a TAMS-compatible Timestamp, that should be applied to the end component of a Segment's TimeRange to calculate the time at which it should be deleted. + +Note that a Flow's timeline might not use time of day. +But this mechanism can still be used to progressively delete Segments given an appropriate `segment_retention_offset`. + +The execution of the deletion of Flow Segments may be carried out by the TAMS service instance itself, or a worker client. +In either case, the following should be considered: + +* The `segment_retention_offset` may be adjusted up or down after having being set initially +* The system enacting the delete will need Delete permissions on the Flow +* The Flow Segments may be manually deleted by a client before the time configured for automated deletion +* There may be gaps between segments on the Flow's timeline + +### Worker Client + +Worker clients will initially need to populate their list of Flows which have Segment Retention configured. +This may be achieved by performing a GET to `/flows` with the query parameter `tag_exists.segment_retention_offset=true`. +This will retrieve a list of Flows and their metadata including tags required for this function. + +Worker clients will also need to keep their internal list up to date. +This may be achieved with by regularly polling the API as above or by subscribing to the `flows/created`, `flows/updated`, and `flows/deleted` events via webhooks. +Where polling is used, a suitable interval should be chosen that balances the latency of the deletes and the number of requests made to the TAMS API. +Latency, in this case, would occur where the configured delete time is earlier than the first poll that would identify it. + +In some cases, worker clients may wish to check for updates to relevant Flow metadata (i.e. via a GET to `/flows/`) before enacting a delete, to detect any changes to the delete time. +But due to the high number of delete requests associated with this process may make this impractical. + +This simplest way to enact the Segment deletes is via regular DELETE requests to `/flows//segments` with the `timerange` query parameter set to `_)` where `` is the TAI Timestamp for the current time minus the value of `segment_retention_offset`. +This method requires fewer API requests than deleting each Segment individually at it's scheduled delete time. +But it does result in a delay of up to the amount of time between delete requests. + +Where minimal latency for deletes is required, a GET to `/flows/{flowId}/segments` shall return Segments and their timeranges which may be used to calculate their delete times. +Delete requests to `/flows//segments` may then be performed at the appropriate time with `timerange` query parameter set to `_)` where `` is the end time of the Segment to be deleted. + +## Flow Retention + +This approach may be used to control the automatic deletion of Flows. +Flow retention time may be signalled via the `flow_retention_offset` tag. +This tag provides an offset, formatted as a TAMS-compatible Timestamp, that should be applied to the greater of the `metadata_updated` or `segments_updated` times of a Flow. +The resultant time is when the Flow should be deleted. + +The execution of the deletion of the Flow may be carried out by the TAMS service instance itself, or a worker client. +In either case, the following should be considered: + +* The `flow_retention_offset` may be adjusted up or down after having being set initially +* `metadata_updated` and `segments_updated` will be updated with a later datetime whenever the metadata or segments are updated respectively +* The system enacting the delete will need Delete permissions on the Flow +* The Flow may be manually deleted by a client before the time configured for automated deletion + +### Worker Client + +Worker clients will initially need to populate their list of Flows which have Flow Retention configured. +This may be achieved by performing a GET to `/flows` with the query parameter `tag_exists.flow_retention_offset=true`. +This will retrieve a list of Flows and their metadata including tags, `metadata_updated`, and `segments_updated` required for this function. + +Worker clients will also need to keep their internal list up to date. +This may be achieved with by regularly polling the API as above or by subscribing to the `flows/created`, `flows/updated`, and `flows/deleted` events via webhooks. +Where polling is used, a suitable interval should be chosen that balances the latency of the deletes and the number of requests made to the TAMS API. +Latency, in this case, would occur where the configured delete time is earlier than the first poll that would identify it. + +Worker clients should also check for updates to relevant Flow metadata before enacting a delete, to detect any changes to the delete time. +This may be achieved via a GET to `/flows/`. +The delete of a flow may be achieved via a DELETE to `/flows/`. +Deletes may be enacted at the specific time configured (e.g. by triggering via a cron job), or may be processed in batches within a reasonable period after the configured time (e.g. in the same process which polls the API for updates). + +## Empty Flow Clean-up + +This approach may be used to automatically remove empty Flows. +The execution of the deletion of empty Flows may be carried out by the TAMS service instance itself, or a worker client. +Either way, it should be configured directly with an offset much like the one used in regular Flow Deletion. + +### Worker Client + +Empty Flows may be located via a GET request to `/flows` with the `timerange` query parameter set to `()`, aka the "empty" Timerange. +Flows in the return data with both `metadata_updated` and `segments_updated` earlier than the current time minus the pre-configured offset should be deleted via a DELETE to `/flows/`. From efb83dd69699256ebc77a4c077f73fc9dfde843e Mon Sep 17 00:00:00 2001 From: James Sandford Date: Fri, 14 Nov 2025 16:39:03 +0000 Subject: [PATCH 5/9] Sort tag listing alphabetically --- docs/appnotes/0003-tag-names.md | 190 ++++++++++++++++---------------- 1 file changed, 95 insertions(+), 95 deletions(-) diff --git a/docs/appnotes/0003-tag-names.md b/docs/appnotes/0003-tag-names.md index ca9aa5a5..cca160bd 100644 --- a/docs/appnotes/0003-tag-names.md +++ b/docs/appnotes/0003-tag-names.md @@ -49,6 +49,29 @@ The tags below are marked with the following statuses: ## Known Flow Tags +### auth_classes + +Status: **Experimental** + +Suggested as a way to build lightweight Attribute-based Access Control in [AppNote0016: Authorisation in TAMS workflows](./0016-authorisation-in-tams-workflows.md). +A comma seperated list of auth classes used to derive permissions on the Flow. + +### c2pa-provenance + +Status: **Proposed** + +Proposed in [APPNOTE0011 - C2PA provenance across related Sources and Flows](../appnotes/0011-c2pa.md). + +Signals the presence of C2PA provenance data (a "manifest store") in a Flow or child Flow. + +Known values: + +* No Tag: A C2PA manifest has not been identified in this Flow +* `none`: A C2PA manifest is not present inside this Flow +* `embedded`: A C2PA manifest is present inside this multi-essence Flow +* `detached`: A C2PA manifest has been copied from this multi-essence Flow into a collected mono-essence data Flow. + The Flow will have the role `c2pa` in the multi-essence Flow's `flow_collection` array. + ### created_by Status: **Deprecated** @@ -63,6 +86,17 @@ Status: **Deprecated** Replaced by the `created` field in Flow metadata. Contains the ISO formatted date-time when a Flow was created. +### flow_retention_offset + +Status: **Proposed** + +Proposed in [ADR0043 - Signalling Retention Time](../adr/0043-signalling-retention-time.md). + +Used to signal the retention time for a Flow. +Specified as a TAMS-compatible [Timestamp](../appnotes/0008-timestamps-in-TAMS.md#timestamp-representation). +The earliest time in [TAI](https://en.wikipedia.org/wiki/International_Atomic_Time) at which the Flow should be automatically deleted is calculated by adding this offset to the most recent of `metadata_updated` or `segments_updated`. +Example: For a Flow with `metadata_updated` of `2023-09-14T09:45:26Z`, `segments_updated` of `2023-09-15T10:23:28Z`, and `flow_retention_offset` of `86400:0` (aka 24 hours) - the `segments_updated` shall be used and the Flow should be deleted a day later at `2023-09-16T10:23:28Z`. + ### flow_status Status: **Proposed** @@ -82,6 +116,46 @@ Known values: * `replication_in_progress` - Content is currently being ingested to this Flow from another store via a replication process * `closed_complete` - Flow is complete and will not receive any more content +### hls_exclude + +Status: **Experimental** + +Used in the TAMS demonstration at NAB 2025. + +The type is a `boolean`. +It is used to indictate the Flow should be excluded from HLS manifest generation. +Defaults to `false` if the tag is not set. + +### hls_segments + +Status: **Experimental** + +Used in the TAMS demonstration at IBC 2024. + +The type is a `number`. +It is used to limit the number of Segments presented in the HLS manifest. +Defaults to `150` if the tag is not set. +Use the value `inf` to list all Segments. +However, listing all Segments may result in the generation of the HLS manifest timing out. + +### hls_segment_length + +Status: **Deprecated** + +Replaced by the `segment_duration` field in Flow metadata. + +### input_quality + +Status: **In use** + +A human readable string identifying the quality of the media. + +Known values: + +* `intermediate` - i-frame only +* `contribution` - long-GOP +* `web` - long-GOP, low bit-rate + ### language_code Status: **Experimental** @@ -98,6 +172,14 @@ Examples: * [`deu`](https://iso639-3.sil.org/code/deu) for German captions * `["eng", "fra"]` for audio where both English and French are spoken +### loop_recorder_duration + +Status: **Implementation specific** + +Used in the AWS TAMS Tools loop recorder. +The target duration of the Flow in seconds. +Oldest Segments that take the duration over the specified duration will be deleted periodically. + ### originating_id Status: **Proposed** @@ -124,18 +206,6 @@ Used where this Flow is a proxy of another Flow. Contains the ID of the Flow this is a proxy of. Deprecated in favour of discovery via Flows of the same Source with appropriate metadata (e.g. lower generation, appropriate resolution, etc). -### input_quality - -Status: **In use** - -A human readable string identifying the quality of the media. - -Known values: - -* `intermediate` - i-frame only -* `contribution` - long-GOP -* `web` - long-GOP, low bit-rate - ### salmon_created_by_job Status: **Implementation specific** @@ -143,6 +213,17 @@ Status: **Implementation specific** Used by BBC R&D's experimental internal stream ingest service named "Salmon". Records the Salmon job ID which created the Flow. +### segment_retention_offset + +Status: **Proposed** + +Proposed in [ADR0043 - Signalling Retention Time](../adr/0043-signalling-retention-time.md). + +Used to signal the retention time for Flow Segments. +Specified as a TAMS-compatible [Timestamp](../appnotes/0008-timestamps-in-TAMS.md#timestamp-representation). +The earliest time in [TAI](https://en.wikipedia.org/wiki/International_Atomic_Time) at which a given Segment should be automatically deleted is calculated by adding this offset to the end Timestamp component of the Segments TimeRange. +Example: For a Flow using TAI for its timeline, a `segment_retention_offset` of `600:0` will result in Segments being deleted after 10 minutes (specifically, after the end of the Segment). + ### writing_flow_timing_temi_timestamps Status: **Implementation specific** @@ -167,88 +248,14 @@ A string representation of a Fraction, used to set the Segment rate (Segments pe This is an average rate. Actual Segment rates/durations may vary. -### c2pa-provenance - -Status: **Proposed** - -Proposed in [APPNOTE0011 - C2PA provenance across related Sources and Flows](../appnotes/0011-c2pa.md). - -Signals the presence of C2PA provenance data (a "manifest store") in a Flow or child Flow. - -Known values: - -* No Tag: A C2PA manifest has not been identified in this Flow -* `none`: A C2PA manifest is not present inside this Flow -* `embedded`: A C2PA manifest is present inside this multi-essence Flow -* `detached`: A C2PA manifest has been copied from this multi-essence Flow into a collected mono-essence data Flow. - The Flow will have the role `c2pa` in the multi-essence Flow's `flow_collection` array. - -### hls_segments - -Status: **Experimental** - -Used in the TAMS demonstration at IBC 2024. - -The type is a `number`. -It is used to limit the number of Segments presented in the HLS manifest. -Defaults to `150` if the tag is not set. -Use the value `inf` to list all Segments. -However, listing all Segments may result in the generation of the HLS manifest timing out. - -### hls_segment_length - -Status: **Deprecated** - -Replaced by the `segment_duration` field in Flow metadata. - -### hls_exclude - -Status: **Experimental** - -Used in the TAMS demonstration at NAB 2025. - -The type is a `boolean`. -It is used to indicate the Flow should be excluded from HLS manifest generation. -Defaults to `false` if the tag is not set. +## Known Source Tags ### auth_classes Status: **Experimental** Suggested as a way to build lightweight Attribute-based Access Control in [AppNote0016: Authorisation in TAMS workflows](./0016-authorisation-in-tams-workflows.md). -A comma separated list of auth classes used to derive permissions on the Flow. - -### loop_recorder_duration - -Status: **Implementation specific** - -Used in the AWS TAMS Tools loop recorder. -The target duration of the Flow in seconds. -Oldest Segments that take the duration over the specified duration will be deleted periodically. - -### flow_retention_offset - -Status: **Proposed** - -Proposed in [ADR0043 - Signalling Retention Time](../adr/0043-signalling-retention-time.md). - -Used to signal the retention time for a Flow. -Specified as a TAMS-compatible [Timestamp](../appnotes/0008-timestamps-in-TAMS.md#timestamp-representation). -The earliest time in [TAI](https://en.wikipedia.org/wiki/International_Atomic_Time) at which the Flow should be automatically deleted is calculated by adding this offset to the most recent of `metadata_updated` or `segments_updated`. -Example: For a Flow with `metadata_updated` of `2023-09-14T09:45:26Z`, `segments_updated` of `2023-09-15T10:23:28Z`, and `flow_retention_offset` of `86400:0` (aka 24 hours) - the `segments_updated` shall be used and the Flow should be deleted a day later at `2023-09-16T10:23:28Z`. - -### segment_retention_offset - -Status: **Proposed** - -Proposed in [ADR0043 - Signalling Retention Time](../adr/0043-signalling-retention-time.md). - -Used to signal the retention time for Flow Segments. -Specified as a TAMS-compatible [Timestamp](../appnotes/0008-timestamps-in-TAMS.md#timestamp-representation). -The earliest time in [TAI](https://en.wikipedia.org/wiki/International_Atomic_Time) at which a given Segment should be automatically deleted is calculated by adding this offset to the end Timestamp component of the Segments TimeRange. -Example: For a Flow using TAI for its timeline, a `segment_retention_offset` of `600:0` will result in Segments being deleted after 10 minutes (specifically, after the end of the Segment). - -## Known Source Tags +A comma separated list of auth classes used to derive permissions on the Source. ### hls_exclude @@ -260,13 +267,6 @@ The type is a `boolean`. It is used to indicate the Source should be excluded from HLS manifest generation. Defaults to `false` if the tag is not set. -### auth_classes - -Status: **Experimental** - -Suggested as a way to build lightweight Attribute-based Access Control in [AppNote0016: Authorisation in TAMS workflows](./0016-authorisation-in-tams-workflows.md). -A comma separated list of auth classes used to derive permissions on the Source. - ### language_code Status: **Experimental** From 64186c9e194f84da053781186ef9e89526f199f6 Mon Sep 17 00:00:00 2001 From: James Sandford Date: Wed, 10 Dec 2025 14:01:05 +0000 Subject: [PATCH 6/9] Fix missing full stop --- docs/adr/0043-signalling-retention-time.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/0043-signalling-retention-time.md b/docs/adr/0043-signalling-retention-time.md index c9dae543..a4d81a92 100644 --- a/docs/adr/0043-signalling-retention-time.md +++ b/docs/adr/0043-signalling-retention-time.md @@ -35,7 +35,7 @@ To judge this position, we might consider what other types of media management T TAMS implementations are currently responsible for the reference counting of Media Objects and deleting the Objects from storage once it is no longer references by any Flow Segments. It is vital for the management of Media Objects given that they have no conceptual "owner" within the TAMS system, and thus no other system responsible for their deletion. This operation is performed by reference counting Object IDs as opposed to explicit signalling by the client. -It requires no configuration +It requires no configuration. This is also a case where an instance may be considered to be performing its own maintenance routines. Loop recording, on the other hand, may be configured via data in the TAMS Service (tags or a field in the core spec) but actually enacted by a secondary system. From bf41c57f044af055918677f766a72d2c101bd639 Mon Sep 17 00:00:00 2001 From: James Sandford Date: Thu, 22 Jan 2026 16:02:04 +0000 Subject: [PATCH 7/9] Formatting based on feedback --- docs/adr/0043-signalling-retention-time.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/adr/0043-signalling-retention-time.md b/docs/adr/0043-signalling-retention-time.md index a4d81a92..9954f811 100644 --- a/docs/adr/0043-signalling-retention-time.md +++ b/docs/adr/0043-signalling-retention-time.md @@ -212,26 +212,26 @@ This option would see the time for a Flow's automated deletion to be signalled w * Neutral, because enacting this requires no further calculations * Bad, because this mode of operation cannot be statically templated -### Option 6b: Describe a method of deleting an entire Flow at a time offset from its creation time +### Option 6b: Describe a method of deleting an entire Flow at a time offset from its `created` time -This option would see the time for a Flow's automated deletion to be signalled with an offset to be applied to the Flow's Creation time. +This option would see the time for a Flow's automated deletion to be signalled with an offset to be applied to the Flow's `created` time. * Good, because it supports static templating (e.g. via Flow Profiles) * Neutral, because enacting this requires a simple calculation to identify the actual time for deletion * Neutral, because it allows Flows to be assigned an explicit life time * Bad, because it doesn't take into account whether the Flow is still being written to -### Option 6c: Describe a method of deleting an entire Flow at a time offset from its Segments updated time +### Option 6c: Describe a method of deleting an entire Flow at a time offset from its `segments_updated` time -This option would see the time for a Flow's automated deletion to be signalled with an offset to be applied to the Flow's Segments Updated time. +This option would see the time for a Flow's automated deletion to be signalled with an offset to be applied to the Flow's `segments_updated` time. * Good, because it supports static templating (e.g. via Flow Profiles) * Good, because it allows Flows to be assigned an explicit life time beyond when Segments are written to it * Neutral, because enacting this requires a simple calculation to identify the actual time for deletion -### Option 6d: Describe a method of deleting an entire Flow at a time offset from the most recent of Segments updated or Metadata updated time +### Option 6d: Describe a method of deleting an entire Flow at a time offset from the most recent of `segments_updated` or `metadata_updated` time -This option would see the time for a Flow's automated deletion to be signalled with an offset to be applied to the Flow's the later of Segments Updated or Metadata Updated time. +This option would see the time for a Flow's automated deletion to be signalled with an offset to be applied to the Flow's the later of `segments_updated` or `metadata_updated` time. * Good, because it supports static templating (e.g. via Flow Profiles) * Good, because it captures any type of Flow modification From 6b6e8ae525c4a167b7ac2a72652a834be8ead2cd Mon Sep 17 00:00:00 2001 From: James Sandford Date: Thu, 22 Jan 2026 16:04:49 +0000 Subject: [PATCH 8/9] Mark ADR0043 as accepted --- docs/adr/0043-signalling-retention-time.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/0043-signalling-retention-time.md b/docs/adr/0043-signalling-retention-time.md index 9954f811..bf28cb48 100644 --- a/docs/adr/0043-signalling-retention-time.md +++ b/docs/adr/0043-signalling-retention-time.md @@ -1,5 +1,5 @@ --- -status: "proposed" +status: "accepted" --- # Signalling Retention Time From d8fc2065da7c5078f956f26de2f3a3ac669151d4 Mon Sep 17 00:00:00 2001 From: James Sandford Date: Thu, 22 Jan 2026 16:07:10 +0000 Subject: [PATCH 9/9] Fix typo in AppNote0019 filename --- ...on-management.md => 0019-implementing-retention-management.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/appnotes/{0019-implementing-rentention-management.md => 0019-implementing-retention-management.md} (100%) diff --git a/docs/appnotes/0019-implementing-rentention-management.md b/docs/appnotes/0019-implementing-retention-management.md similarity index 100% rename from docs/appnotes/0019-implementing-rentention-management.md rename to docs/appnotes/0019-implementing-retention-management.md