Media Server Disconnected - US-EAST-1 

Date/Time Impacted

Start Time: Jan 26, 2024 15:01:00 UTC | End Time: Jan 26, 2024 15:20:00 UTC

Start Time: Feb 13, 2024 15:06:00 UTC | End Time: Feb 13,2024 15:07:00 UTC

Services Impacted

Video - Session Connection, Video - Archiving (Enterprise)

Video - Broadcast, Video - Media Connection (Enterprise).

Summary

A media server pod failed during rotation, causing the disconnection of existing sessions. New sessions were routed to a functioning media server pod.

Root Cause

During the rotation, one of the media server nodes was overloaded due to a bug on the Load Balancer end, which caused it not to scale the connections.

Restoration Actions

The issue was corrected on Jan 26, 2024, 15:20:00 UTC and Feb 13, 2024, 15:07:00 UTC by removing the media server pod.

Next Steps

It’s a known issue and the fix is ready to be deployed in production this week to protect us across all regions.

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/DyYcm1otcZrt6k7s5