Service Incident - Intermittent Connectivity Issues to US API Endpoints

Date/Time Impacted

October 04, 2023 00:42:00 UTC - October 04, 2023 02:41:00 UTC

‌

Services Impacted

SMS - SMS Platform, MMS Platform 

Video - Session Connection 

Voice - Programmable SIP,  Voice API, Voice SDK 

Auth & Identity - Number Insight, Number Insight V2, Verify, Verify V2

‌

Summary

On October 4th at 00:42:00 UTC, our monitoring alerted us to intermittent failures on API endpoints in the US East region. Our engineering team investigated further to identify the scope of the impact and reported the issue to our hosting provider.

‌

Root Cause

One of our hosting providers experienced an issue resulting in disruption to our customer traffic. This disruption led to intermittent connectivity issues to API Endpoints in the US East region. Customers in the Eastern US region hitting the api-us.vonage.com,  api.vonage.com, or api-us-3.vonage.com endpoints would have been impacted.

‌

Restoration Actions

At 2023-10-04 01:19:00 UTC our engineering team made a configuration change to divert as much traffic as possible away from the impacted endpoints, mitigating most of the impact to our customers. At Oct 04, 2023 02:41:00 UTC our hosting provider resolved the issue and any continued impact was halted at that time.

‌

Next Steps

In order to more quickly detect and resolve potential issues in the future, Vonage will review and enhance current internal processes around implementing failovers during service-impacting issues and handling 3rd-party provider escalations. We will also evaluate our current monitoring configurations and add additional smoke tests where necessary in order to be able to detect service-impacting issues more quickly and accurately.

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/UcAqeSK3ijBqQVkV6