connect response doesn't indicate support for Enhanced RTMP

[zenomt]

the client can indicate support for Enhanced RTMP by including a fourCcList member of the connect command's argument/command object (though see #10 about the name of that member).

however, clients can't tell if servers support Enhanced RTMP. while an unaware server can simply forward Enhanced RTMP messages as they come in, this won't have the desired effect for clients subscribing to a stream after its publish has started. in particular, servers unaware of Enhanced RTMP won't have special treatment of PacketSequenceStart, PacketTypeMetadata, and PacketTypeMPEG2TSSequenceStart messages, remembering them and sending them to new subscribers before the coded frames.

a publishing client should be able to tell that the server will or won't perform the special sequence/metadata processing for subscribers, and subscribing clients should be able to tell that they may not receive the sequence/metadata messages for enhanced messages. this could be accomplished by echoing the fourCcList (and/or others as appropriate) back in the connect transaction's _result Info Object.

[zenomt]

this should probably be a separate Issue, but on the subject of the fourCcList property: the spec says "supported by the client". given the meaning of the videoCodecs and audioCodecs bitmap properties, one would assume the fourCcList indicates the codecs that the client is capable of receiving, vs ones that it can or might send. if that's the case, that distinction should be made clear.

[veovera]

this should probably be a separate Issue, but on the subject of the fourCcList property: the spec says "supported by the client". given the meaning of the videoCodecs and audioCodecs bitmap properties, one would assume the fourCcList indicates the codecs that the client is capable of receiving, vs ones that it can or might send. if that's the case, that distinction should be made clear.

• Really good feedback, thank you!
• I don't believe indicating which codecs the client can receive vs send is in the original spec?
• fourCcList adopts current behavior while indicating what codecs are supported in the list
• That said that does not mean we should not add more info in the connect command in the next version of the enhancement to indicate clients capabilities to send vs receive any particular fourCc item.
• And yes, agreed this should be a separate issue. Perhaps if you get a chance open an additional issue and provide more color to the behavior?

[veovera]

the client can indicate support for Enhanced RTMP by including a fourCcList member of the connect command's argument/command object (though see #10 about the name of that member).

however, clients can't tell if servers support Enhanced RTMP. while an unaware server can simply forward Enhanced RTMP messages as they come in, this won't have the desired effect for clients subscribing to a stream after its publish has started. in particular, servers unaware of Enhanced RTMP won't have special treatment of PacketSequenceStart, PacketTypeMetadata, and PacketTypeMPEG2TSSequenceStart messages, remembering them and sending them to new subscribers before the coded frames.

a publishing client should be able to tell that the server will or won't perform the special sequence/metadata processing for subscribers, and subscribing clients should be able to tell that they may not receive the sequence/metadata messages for enhanced messages. this could be accomplished by echoing the fourCcList (and/or others as appropriate) back in the connect transaction's _result Info Object.

Nice catch especially since this is so nuanced. To echo the 'fourCcList' is of course straight forward and probably would be a good addition to the spec. BTW: Shouldn't this already potentially be an issue? AVCPacketType has same challenges, although by now most servers support AVC codec id and the AVC sequence header... this is still not guaranteed since today the RTMP spec does not require a server to echo it's set of support codecs?

[zenomt]

in retrospect i don't think echoing the fourCcList is the right answer, since that might imply additional unintended semantics (like which codecs the server is capable of handling). but some indication that the server understands Enhanced RTMP and will do something appropriate especially with the new enhanced start/init/metadata messages i think would be useful for feature & codec negotiation.

[zenomt]

I don't believe indicating which codecs the client can receive vs send is in the original spec?

the original spec is missing a lot of things (i wish i had had the time way back then for a full rewrite instead of just editorial cleanup). :)

the behavior of Adobe Media Server (at least) is to use the videoCodecs and audioCodecs bitmaps as selectors/filters for what can be sent to the client. AMS won't send video or audio messages to a client unless the codec being used has its bit set in the bitmap. AMS doesn't care which codecs a client can send, since AMS can record & forward all (traditional) codecs. also, many clients have asymmetric capabilities (for example, Flash Player can receive & play back AAC, but i don't think it can encode & send AAC) and this asymmetry isn't signaled in the connect/_result handshake.

[zenomt]

And yes, agreed this should be a separate issue. Perhaps if you get a chance open an additional issue and provide more color to the behavior?

i opened #14

[zenomt]

the "Enhancing NetConnection connect Command" section of the latest spec appears to address this Issue, and technically this issue could be closed. however, it's a bit hard to tell with the current wording of the last paragraph of that section:

As you can see, the client declares to the server what enhancements it supports. The server responds with a command, either _result or _error, to indicate whether the response is a result or an error. During the response, the server provides some properties within an Object as one of the parameters. This is where the server needs to state its support for E-RTMP. The server SHOULD state its support via attributes such as videoFourCcInfoMap, capsEx, and similar properties.

i think it'd be valuable for a future revision to reword this section, and perhaps have a separate paragraph for the server's response that explicitly lists that an Enhanced RTMP server should be sending one or more of the enhanced connect properties back to the client in the NetConnection.Connect.Success _result, and how a client should be interpreting such an enhanced response from the server.

also, specifically for capsEx, one could reasonably assume that a missing capsEx means 0, but it'd be good to call that out explicitly.

[zenomt]

meanwhile, i've modified my tcserver to answer thus:

connection onStatus: {
    "audioFourCcInfoMap": {
        "*": 4
    },
    "capsEx": 0,
    "code": "NetConnection.Connect.Success",
    "connectionID": "5c052b8e63ffcf1a12f89b6657ab3bb1b55c08a1321489c139fe4b066394cceb",
    "description": "you connected!",
    "fourCcList": [
        "*"
    ],
    "level": "status",
    "objectEncoding": 0,
    "serverFingerprint": "264886da312d618a82dd3ef57e5d3900126a95eade7c713d551e084e3132d75a",
    "serverInfo": "welcome to REDACTED",
    "videoFourCcInfoMap": {
        "*": 4
    }
}

to say that it can forward any video or audio codecs, but does not support Multitrack, nor will it ever send a NetConnection.Connect.ReconnectRequest to the client.

[veovera]

Thank you for the thoughtful feedback! I'll take a closer look at the wording and make the necessary adjustments. For now, I'll keep this issue open until we've had a chance to polish the wording.

[Thulinma]

Adding a couple more cents to this discussion: the wording definitely needs updating to be more clear.

"Some properties in some form" is quite fuzzy since the connect reply contains both a "Properties" and an "Information" object: which should be extended with the new properties? It looks like @zenomt added them to the "Information" object (which contains the level/code/description fields as well), but wouldn't it make more sense to consider them "connection properties" instead..?

I interpret fourCcList to indicate supported codecs for any operation: encode/decode/passthrough/send/receive.

I interpret audioFourCcInfoMap and videoFourCcInfoMap to imply that CanDecode also means it can receive, and to imply that CanEncode also means it can send (anything else would be... weird, after all). CanForward I take to mean it can both send and receive, but potentially not act on the contents (by decoding, encoding, or any other non-trivial operation that may need knowledge of the payload format to be successful).

I would suggest changing the paragraph from:

As you can see, the client declares to the server what enhancements it supports. The server responds with a command, either _result or _error, to indicate whether the response is a result or an error. During the response, the server provides some properties within an Object as one of the parameters. This is where the server needs to state its support for E-RTMP. The server SHOULD state its support via attributes such as videoFourCcInfoMap, capsEx, and similar properties.

to:

As you can see, the client declares to the server what enhancements it supports. The server responds with a command, either _result or _error, to indicate whether the response is a result or an error.
In a _result response, the server responds to each of the new properties (such as videoFourCcInfoMap, capsEx, etc) by including a matching property in the "Information" object of the _result response. The response property should indicate server support, rather than merely confirming the client's support.
The actually supported capabilities of the client-server connection are thus expressed by the intersection of both the command and the response.
If a property is absent in the server reply, the client should assume no support is available for the intended capability, but may decide to continue regardless if there is no possible fallback behavior (e.g. there is no alternative codec it could feasibly use).
If the server responds with a property that the client did not send, the client may opt (but is not required) to use the relevant capabilities as well.

That should clear up any confusion, I hope. 🤔

[veovera]

Hi @Thulinma ,

Thanks for raising this and for the thoughtful feedback — it helped us revisit the section for better clarity. We made a small adjustment to simplify the language while keeping the intent intact.

Proposed wording:
As you can see, the client declares to the server what enhancements it supports.
The server responds with a command, either _result or _error, to indicate whether the response is a result or an error.
In its _result response, the server returns an Information object containing properties such as videoFourCcInfoMap, capsEx, and others that describe its supported capabilities.

The server SHOULD explicitly state its support using these properties to ensure successful E-RTMP negotiation.

Why we are proposing this route:

• Clarifies that the properties are in the Information object
• Avoids vague phrases like “some properties within an Object”
• Keeps the section concise and defers detailed capability negotiation rules to other parts of the spec

Appreciate you surfacing this — your input made the wording more precise and easier to follow.