Service Issue in Platform - SMS

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

Incident Start Time: 2023-05-02 13:14 UTC

Incident End Time: 2023-05-02 14:30 UTC

‌

Services Impacted

SMS API

SMPP

Verify API

Messages API

‌

Summary

‌

From 2023-05-02 13:14 UTC to 2023-05-02 14:05 UTC, customers faced delivery latency across all regions due to database connectivity issues. We rectified the situation by failing over to healthy regions (regions that had less delays). Following that, our smoke tests passed at 2023-05-02 14:05 where we observed normal traffic.

The services started to have database connectivity issues again around 2023-05-02 14:20 UTC that impacted MOs and DLRs only. The impact recovered when database connectivity improved at 2023-05-02 14:30 UTC. Outbound messages were being delivered without delay during that time.

‌

Timelines:

Between 2023-05-02 13:15 UTC and 2023-05-02 13:45 UTC there were intermittent delays of up to 2.5 minutes for outbound messages in the US, AP and EU regions.
Between 2023-05-02 13:45 UTC and 2023-05-02 14:00 UTC, there were delays for Outbound messages of up to 20 mins in the EU region.
Between 2023-05-02 14:20 and 14:30 UTC, there were delays on inbound messages and Delivery receipts. But the Outbound messages were being delivered during that time.

‌

Root Cause

A DNS issue affected our databases in different regions, preventing our services from connecting to them. This caused our internal services to restart after losing connectivity. The issue resulted in high latency for SMS across all regions.

‌

Restoration Actions

We failed over the traffic to healthy regions to mitigate the impact. This reduced the impact on outbound traffic. For inbound traffic recovered when the connectivity with the database improved.

‌

Next Steps

‌

Measures to improve alerts in order to improve recovery time.

Extend testing scenarios to reduce detection and recovery time of the incident.

Improve database outage notification service updates.

Posted May 03, 2023 - 23:37 UTC

Resolved

This issue has been resolved.

The following services were impacted from [02-05-2023 13:14] UTC until [02-05-2023 14:30] UTC.

To summarise customers submitting SMS via the following products may have experienced increased delivery latency in the EU, US and AP regions between [02-05-2023 13:15] UTC and [02-05-2023 13:45] UTC additionally, we had further delays in the EU region between[02-05-2023 13:45] UTC and [02-05-2023 14:00] UTC:

SMS API
SMPP
Verify API
Messages API

Also, customers may have experienced delays receiving inbound SMS or delivery receipts - irrespective of the region - via the following services between [02-05-2023 14:20] UTC and [02-05-2023 14:30] UTC:

SMS API
SMPP
Messages API

All services have been restored.
Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.

Posted May 02, 2023 - 16:57 UTC

Monitoring

We have implemented a fix for this issue.

Customers submitting SMS via the following products may have experienced increased delivery latency in the EU, US or AP regions:

SMS API
SMPP
Verify API
Messages API

Additionally, customers may have experienced delays receiving inbound SMS or delivery receipts - irrespective of region - via the following services:

SMS API
SMPP
Messages API

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted May 02, 2023 - 15:08 UTC

Update

Customers submitting SMS via the following products may have experienced increased delivery latency in the EU, US or AP regions:

SMS API
SMPP
Verify API
Messages API

Additionally, customers may have experienced delays receiving inbound SMS or delivery receipts - irrespective of region - via the following services:

SMS API
SMPP
Messages API

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted May 02, 2023 - 15:05 UTC

Identified

We have identified the root cause of the issue.

Customers submitting SMS via the following services may experience increased delivery latency in the EU region:
SMS
SMPP
Messages API
Verify API

Additionally, customers receiving inbound SMS or delivery receipts - irrespective of region - may experience increased delivery latency.

We are currently working on getting the issue fixed and will post further updates here.

Posted May 02, 2023 - 13:51 UTC

Update

We are continuing to investigate this issue.

Posted May 02, 2023 - 13:46 UTC

Investigating

Our monitoring has alerted us to a potential service issue with our platform. Although It is not yet certain whether there is any customer impact, we are alerting customers immediately while our engineering teams investigate.

If there is a customer impact, it is likely to be affecting outbound SMS deliverability.

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://api.support.vonage.com/hc/en-us/articles/6749696044828-Vonage-API-Support-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted May 02, 2023 - 13:29 UTC

This incident affected: SMS & Messages (SMS, SMPP, Messages API) and Auth & Identity (Verify).