APAC Voice, Messages, Dispatch API Issue

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

Oct 12, 2022 at 20:52 UTC

Services Impacted

Oct 12, 2022 at 23:29 UTC

Summary

On October 12th at 20:52 UTC, we experienced issues with Voice and Conversation API, Messaging API, and Dispatch in the APAC region. Customers received HTTP 401s during the incident and could not carry out their Voice and Conversation API, Messages API, or Dispatch requests.

Root Cause

An unexpected server maintenance in APAC included a database server update. After the update, our services could not reconnect to the database and this caused major disruption to customers requesting authentication in APAC.

The connectivity to the updated database failed because the existing TLS version was no longer supported.

Restoration Actions

The issue was resolved by updating the TLS options in the database client to resolve the compatibility issues.

Next Steps

Identify and update service connection strings to use newer TLS version
Improve alerting on server maintenance

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/Eizg5CKM42ZVPk5v9

Posted Oct 14, 2022 - 15:17 UTC

Resolved

The issue impacting multiple services has been resolved.

The following services were impacted from 12-October-2022 20:52 UTC until 12-October-2022 23:29 UTC.
Voice API
Conversation API
Messages API
Dispatch API

All services have been restored.
Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.

Posted Oct 13, 2022 - 14:03 UTC

Monitoring

As of 23:29 UTC, a fix has been implemented and we are monitoring the results.

Posted Oct 12, 2022 - 23:47 UTC

Identified

On October 12th at 20:52 UTC, we experienced issues with APAC Voice and Conversation API, Messaging API, and Dispatch. The affected customers will encounter 401 errors on these impacted services. Our engineering team continues to work on the issue.

Posted Oct 12, 2022 - 22:58 UTC

Investigating

Posted Oct 12, 2022 - 22:57 UTC

Identified

We have identified the root cause of the issue. We are currently working on getting the issue fixed and will post further updates here.

Posted Oct 12, 2022 - 22:08 UTC

Investigating

Our monitoring has alerted us to a potential service issue with our platform. Although It is not yet certain whether there is any customer impact, we are alerting customers immediately while our engineering teams investigate.

If there is a customer impact, it is likely to be affecting the following products:
Voice API
Messages API
Dispatch API

We aim to provide an update on our investigation within 1 hour.

Posted Oct 12, 2022 - 21:11 UTC

This incident affected: Voice (Voice API) and SMS & Messages (Messages API, Dispatch).