Video API Service Issue in Platform

Incident Report for Vonage API Status

Postmortem

Incident Statement -

Date/Time Impacted

September 06, 2024 from 18:32 UTC to 22:07 UTC

‌

Services Impacted

Video API OpenTok Environment
- Session Connection
- Archive Inspector
- Insights & Advanced Insights
- Session Inspector
- Opentok Playground
- Pre-Call Test
- Video Account Portal

Voice
- Voice Inspector

‌

Summary

Between 18:32 UTC and 22:07 UTC, Video API customers using the OpenTok Environment encountered errors when connecting to sessions or accessing the OpenTok Dashboard and Tools. The impact on core video services affected new connection attempts, with errors progressively increasing from 18:32 until 19:04. Our monitoring shows that new connections began to recover at 19:05, and the majority of new requests were successful after this time. Engineering teams continued restoring other services until 22:07, which is when the OpenTok Dashboard and other auxiliary tools were fully restored for all customers. Video API customers using the Unified Environment (Vonage Dashboard) would not have been able to access some tools that are shared across environments, but were otherwise not impacted.

‌

Root Cause

This incident was caused by an infrastructure change that resulted in the unavailability of critical components.

Restoration Actions

Engineering teams relaunched the impacted components, fully resolving the incident.

‌

Next Steps

We are conducting an internal review of the protections in place for our critical resources and will make procedural enhancements to further protect these resources.
During triage of this incident, some minor configuration issues were discovered which added to the time to discover and recover from the incident. Our teams will address these issues which will reduce the time to recover in future incidents.
We will implement a service downtime failure page that informs end users of an outage of the web portal.
A minor bug in the UI of Session Inspector is currently preventing users from viewing a list of previously saved sessions. This will be restored by the end of the week.

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/BRDnAmmotJVzfp4e6

Posted Sep 10, 2024 - 17:34 UTC

Resolved

This incident has been resolved, and no further issues should be experienced. All services have been restored.

Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.

Posted Sep 07, 2024 - 09:02 UTC

Monitoring

We have implemented a fix to restore the affected services.

The list of services that were impacted have been confirmed as:
Session Connection
Session Inspector
Archive Inspector
Account Portal
Insights
Playground
Pre-Call test
Voice Inspector

Our monitoring indicates that Video API platform communications failures began to recover at 19:05 UTC, and should be fully recovered now.
Developer Tools should now be also recovered in all regions.
When inspecting sessions in Inspector, saved sessions will be temporarily unavailable, but new sessions are unaffected.
Customers are advised to login again to Developer Tools for a successful experience.

We will continue to monitor the services during the next few hours and post updates should anything change.

Posted Sep 06, 2024 - 23:45 UTC

Update

Our monitoring systems show that core functionality is restored, however engineering are still working to resolve issues with auxiliary services.

For any additional questions, please contact support at support@api.vonage.com

Posted Sep 06, 2024 - 23:08 UTC

Update

Engineering is actively working to restore service. Our monitoring systems show that sessions are recovering.

For any additional questions, please contact support at support@api.vonage.com

Posted Sep 06, 2024 - 22:11 UTC

Update

Engineering continues actively working to restore service and recover from the failure. Engineering is metering that systems have reduced significantly the rate of failures.

For any additional questions, please contact support at support@api.vonage.com

Posted Sep 06, 2024 - 21:34 UTC

Update

Engineering continues actively working to restore service and recover from the failure.
We have started restoring the service and will continue updating this status as we have more information to share.
For any additional questions, please contact support at support@api.vonage.com

Posted Sep 06, 2024 - 20:54 UTC

Identified

Starting at 18:32 UTC, new Video API requests, sessions and services could be subject to failures.
Engineering is working actively to restore service and recover from the failure.
For any additional questions, please contact support at support@api.vonage.com.

Posted Sep 06, 2024 - 20:11 UTC

Update

Customers are currently experiencing problems impacting video session connection, video playground, session inspector, and the video account portal.

You may see errors when attempting to connect to sessions or access the account portal.

Our teams continue working on the issue.

We will update this status as soon as we have more information on this issue.

Posted Sep 06, 2024 - 19:34 UTC

Investigating

Our monitoring system has alerted us to a potential service issue with our platform.

We are alerting customers immediately while our engineering teams investigate further.

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://api.support.vonage.com/hc/en-us/articles/6749696044828-Vonage-API-Support-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted Sep 06, 2024 - 19:05 UTC

This incident affected: Video (Session Connection, Archiving (Standard), Archiving (Enterprise), Broadcast (HLS/RTMP), Monitoring Callbacks, SIP Interconnect, Advanced Firewall Control (TURN) (Standard), Advanced Firewall Control (TURN) (Enterprise), China Relay, Media Connection (Standard), Media Connection (Enterprise), Archive Inspector, Insights & Advanced Insights, Inspector, Playground, Pre-Call Test, Account Portal, Experience Composer, RMZ (US, EU,Germany, Australia, Japan, SKorea, Canada), Email Notifications, SDK CDN, Captions, Signalling) and Voice (Voice Inspector).