Voice Service Issue in Platform

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

On August 23rd, 2022 at 15:34 to 15:40 UTC.

Services Impacted

Voice API
SIP Trunking

Summary

Voice Service Issue caused call failures during the occurrence of the incident. We have received alerts on excessive traffic. This network traffic spike led to application timeouts causing the inability to establish calls.

Root Cause

Multiple large recordings uploading to the Media Service, in parallel, overloaded the shared load balancers in our Data Center. This prevented Voice API and SIP trunking services inside our Data Center from communicating with other dependent services.

Restoration Actions

We remedied the source of the traffic spike and failed over to other servers.

Next Steps

We are currently investigating upload rate limiting on large recordings uploading to the Media Service.
We will add alerting when multiple files greater than a given size are being generated.
Server bandwidths are being monitored to ensure scalability during an increase in load.

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/vTX3KrZTiJoFVvCB8

Posted Aug 25, 2022 - 17:22 UTC

Resolved

The Voice Service Issue has been resolved. On August 23rd 2022 at 15:34 to 15:40 UTC, we experienced Voice API requests failures.

All services have been restored.
Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.

Posted Aug 23, 2022 - 18:42 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Aug 23, 2022 - 17:41 UTC

Investigating

We are currently investigating the issue. We will provide updates when more information and details become available.

Posted Aug 23, 2022 - 16:34 UTC

Monitoring

The Voice API service was impacted from 15:34 to 15:40 UTC. We no longer see the issue occurring.

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Aug 23, 2022 - 16:18 UTC

Investigating

Our monitoring has alerted us to a potential service issue with our platform. Although It is not yet certain whether there is any customer impact, we are alerting customers immediately while our engineering teams investigate.

If there is a customer impact, it is likely to be affecting the following products:
Voice API

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://api.support.vonage.com/hc/en-us/articles/360015693092-Nexmo-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted Aug 23, 2022 - 15:40 UTC

This incident affected: Voice (Voice API).