Service Issue in Platform

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

2023/06/13 19:01:00 UTC - 2023/06/13 21:23:00 UTC

‌

Services Impacted

Messages - Messages API Platform
Video - Video Archive Inspector
Video - Inspector
Video - Archiving (Standard)
Video - Archiving (Enterprise)
Voice - Programmable SIP
Voice - Transcription
Experiences - AI Studio
Experiences - Automatic Speech to Text

‌

Summary

Our cloud hosting provider experienced a regional outage in the U.S., resulting in degraded performance for multiple Vonage services.

‌

Root Cause

Our cloud hosting provider experienced a regional outage. This outage resulted in key services becoming unavailable, causing dependant Vonage services to fail.

‌

Restoration Actions

We failed over impacted services to a stable region
Our cloud hosting provider resolved the regional outage

‌

Next Steps

Explore multi-region redundancy options to reduce impact and failover time
Improve our monitoring and alerting to minimize time to detect the issue.

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey:https://forms.gle/huA689KhLaTKBCjS9

Posted Jun 16, 2023 - 10:00 UTC

Resolved

The wide-spread AWS issues affecting us-east-1 (Virginia) region have been resolved.

The following services were impacted June 13th at 19:01 UTC until June 13th at 21:23 UTC:
SMS, 10DLC
Messages API
Voice transcription
AI Studio - studio builder
AI Studio Insights API, live telephony & messages sessions
Voice Programmable SIP Domain Updates
Video Inspector

All services have been restored.
Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.

Posted Jun 13, 2023 - 22:42 UTC

Monitoring

The wide-reaching AWS issues affecting us-east-1 (Virginia) have recovered.

The list of services that were impacted from June 13th at 19:01 UTC until June 13th at 21:23 UTC have been confirmed as:
SMS, 10DLC
Messages API
Voice transcription
AI Studio - studio builder
AI Studio Insights API, live telephony & messages sessions
Voice Programmable SIP Domain Updates
Video Inspector

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Jun 13, 2023 - 21:32 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jun 13, 2023 - 20:53 UTC

Identified

The issue has been identified. There are wide-reaching AWS issues affecting us-east-1 (Virginia) region since June 13th at 19:01 UTC.

Affected products are:
SMS, 10DLC
Messages API
Voice transcription
AI Studio - studio builder
AI Studio Insights API, live telephony & messages sessions
Voice Programmable SIP Domain Updates

We will continue to provide updates here.

Posted Jun 13, 2023 - 20:38 UTC

Investigating

Our monitoring has alerted us to a potential service issue with our platform. Although It is not yet certain whether there is any customer impact, we are alerting customers immediately while our engineering teams investigate.

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://api.support.vonage.com/hc/en-us/articles/6749696044828-Vonage-API-Support-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted Jun 13, 2023 - 19:48 UTC

This incident affected: Voice (Programmable SIP & SIP Interconnect, Voice API), SMS & Messages (SMS, 10 DLC Platform, Messages API), AI Studio (AI Studio Builder UX/UI, Studio Telephony Sessions), and Video (Inspector).