Broadcast Server in US East 1 Data center terminated

‌

Date/Time Impacted

2023-11-09 15:03 - 15:04 UTC

‌

Services Impacted

Video API - Live Streaming Broadcast

‌

Summary

On November 9th, 2023, at 15:03 UTC, active broadcasts hosted in a US-EAST-1 datacenter server terminated inadvertently. New broadcast requests were immediately routed to other servers.

‌

Root Cause

On November 9th, 2023, at 15:03 UTC, our platform operations team force-drained a server instance in the US-EAST-1 datacenter in response to an alert regarding the number of processes executed on the instance. This action accidentally terminated a broadcast server, disrupting the active broadcasts held on that server.  Our system alerted us about broadcast disconnection at 15:07 UTC, and customers were informed via the status page.

‌

Restoration Actions

Customer impact was reduced with all new broadcast requests automatically routed to other servers in the data center starting November 9th, 2023, at 15:04 UTC.

‌

Next Steps

To more quickly detect and resolve potential issues in the future,

• Vonage will review and enhance current internal procedures for draining a server.
• Vonage will tune the alert mechanism setup to detect processes executed on broadcast servers.