Service Issue in Platform

Incident Report for Vonage API Status

Postmortem

Service issue in platform - Vonage Dashboard, Numbers API and Voice API

Date/Time Impacted

Start Date/ Time: 2022-10-04 12:45 UTC

End Date/ Time: 2022-10-04 13:00 UTC

Services Impacted

Vonage Dashboard

Numbers API

Voice API

‌

Summary

On 2022-10-04 12:45 UTC, our engineering team was alerted to a problem impacting multiple services including Vonage dashboard, Numbers API and Voice API. Further investigation revealed that the impact was limited to sign ups and log in issues for dashboard services, search and buy numbers for numbers API and voice call rejections for voice API.

We immediately attributed the cause to a recent problematic DNS update. We reverted the DNS update and no further impact was seen after this time.

Root Cause

Due to a problematic DNS update, some requests started looping back into the load balancers causing increased CPU and memory usage. This started affecting traffic on other services on the load balancer.

‌

Restoration Actions

We reverted the problematic DNS update to remedy the issue.

‌

Next Steps

Implement managed DNS update service that can help reduce chances of errors.
Expand our monitoring scope for some specific errors.
Dedicated load balancer for each service for better resilience.

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey:

https://forms.gle/dsqiNac2GyRTm2WZ8

Posted Oct 06, 2022 - 14:15 UTC

Resolved

The issue impacting multiple services in Vonage platform has been resolved. Customers may have faced issues with Sign ups and login. Also, with buying numbers and searching numbers in Vonage dashboard.

The following services were impacted from 4th October, 2022 12:45 UTC until 4th October, 2022 13:00 UTC.
Vonage dashboard
Numbers API
Voice API

All services have been restored.
Please let support@api.vonage.com know if you continue to experience problems with this.

Posted Oct 04, 2022 - 15:38 UTC

Monitoring

We have implemented a fix for the issue impacting multiple services in Vonage platform. Customers may have faced issues with Sign ups and login. Also, with buying numbers and searching numbers in Vonage dashboard.

The list of services that were impacted from 4th October, 2022 12:45 UTC until 4th October, 2022 13:00 UTC are as follows:
Vonage dashboard
Numbers API
Voice API

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Oct 04, 2022 - 13:25 UTC

Investigating

Our monitoring has alerted us to a potential service issue with our platform. Although It is not yet certain whether there is any customer impact, we are alerting customers immediately while our engineering teams investigate.

If there is a customer impact, it is likely to be affecting the following products:
Customers may see issues with updates to their account via Customer Dashboard and DevAPI

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://api.support.vonage.com/hc/en-us/articles/360015693092-Nexmo-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted Oct 04, 2022 - 12:53 UTC

This incident affected: Voice (Voice API), Vonage Dashboard (Account Configuration & Feature Access), and Vonage Platform Management (Numbers API).