Add per-channel HLS server failover between primary and secondary endpoints

[Copilot]

SiriusXM provides LIVE_PRIMARY_HLS and LIVE_SECONDARY_HLS endpoints for redundancy. Implemented automatic per-channel failover when the active server experiences persistent failures.

Changes

Per-channel server tracking

• Track active server (primary/secondary) and consecutive failure counts per channel
• Each channel independently manages its server state
• Constant: SERVER_FAILURE_THRESHOLD=3

Failover logic

• record_channel_failure(): Increments failure count per channel, triggers switch at threshold
• record_channel_success(): Resets failure count for channel
• switch_channel_server(): Handles per-channel server transitions and clears cached playlist
• reset_channel_server(): Resets channel to primary server when starting new playback

Playlist selection

• get_playlist_url(): Selects correct playlist entry from API response based on channel's current server
• API returns separate playlist entries for primary and secondary servers with name field
• Correctly matches both size (MEDIUM) and server name (primary/secondary) to select appropriate playlist

Integration points

• get_playlist_variant_url(): Track playlist request outcomes per channel
• get_segment(): Track segment request outcomes per channel, use channel's current server

Behavior

Switches to backup after 3 consecutive HTTP errors (403, 500, etc.). Stays on current server for duration of stream session. Automatically resets to primary when playback stops and resumes. If secondary fails, switches back to primary.

# Before: hardcoded primary
my $url = LIVE_PRIMARY_HLS . "/$base_path/$path";

# After: per-channel server selection with correct playlist handling
my $server_name = $self->get_channel_server($channel_id);
my $server_url = $server_name eq 'primary' ? LIVE_PRIMARY_HLS : LIVE_SECONDARY_HLS;
my $url = $server_url . "/$base_path/$path";
$self->record_channel_success($channel_id);  # or record_channel_failure($channel_id, $error_msg)

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[Copilot]

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• esm.ubuntu.com
  • Triggering command: /usr/lib/apt/methods/https /usr/lib/apt/methods/https (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[paul-1]

@copilot this seems overly complicated

If we get get multiple errors from the primary, switch to the secondary. Just stay on the secondary while the current stream is playing.

If playpack is stopped, and then resumed, then try the primary again.

If we get get multiple errors from the secondary, switch back to the primary. Just stay on the secondary while the current stream is playing.

Also, the current implementation assumes the playlist urls are the same, but they are not necessarily the same. The data from get_playlist_url looks like this

[
{
'size' => 'SMALL',
'name' => 'primary',
'url' => '%Live_Primary_HLS%/AAC_Data/9450/9450_variant_small_v3.m3u8'
},
{
'size' => 'MEDIUM',
'name' => 'primary',
'url' => '%Live_Primary_HLS%/AAC_Data/9450/9450_variant_medium_v3.m3u8'
},
{
'name' => 'primary',
'url' => '%Live_Primary_HLS%/AAC_Data/9450/9450_variant_large_v3.m3u8',
'size' => 'LARGE'
},
{
'size' => 'SMALL',
'url' => '%Live_Secondary_HLS%/AAC_Data/9450/9450_variant_small_v3.m3u8',
'name' => 'secondary'
},
{
'size' => 'MEDIUM',
'url' => '%Live_Secondary_HLS%/AAC_Data/9450/9450_variant_medium_v3.m3u8',
'name' => 'secondary'
},
{
'size' => 'LARGE',
'url' => '%Live_Secondary_HLS%/AAC_Data/9450/9450_variant_large_v3.m3u8',
'name' => 'secondary'
}
]

[Copilot]

@copilot this seems overly complicated

If we get get multiple errors from the primary, switch to the secondary. Just stay on the secondary while the current stream is playing....

Simplified the implementation as requested:

Per-channel tracking: Each channel independently tracks its server (primary/secondary) and failure count

Playlist selection fixed: Now correctly selects from separate primary/secondary playlist entries in the API response, not URL substitution

Failover behavior:

• Switches after 3 consecutive failures
• Stays on current server during stream
• Resets to primary when playback restarts

Commit: 3af44b0