USA SMS - Degradation affecting 10DLC numbers towards AT&T and Verizon network
Incident Report for Vonage API Status
Postmortem

Date/Time Impacted

Between October 24, 2024 17:59:00 UTC and October 25, 2024 00:25:00 UTC

Services Impacted

SMS delivery degradation and latency affecting 10DLC numbers towards AT&T and Verizon network.

Summary

Vonage received a notification from our 10DLC supplier in the US about a downstream operator network connectivity issue affecting a few A2P 10DLC Customers traffic towards +1 US destinations. SMS delivery degradation and latency started from 17:59 UTC on October 24th. Restoration of delivery degradation was observed at 00:25 UTC on October 25th.

The incident was raised to address delivery degradation and latency. Degraded performance statistics were observed across AT&T and Verizon via our 10DLC supplier in the US.

Root Cause

Outage was observed on the supplier's side. Supplier suspected a network hardware at partner's facility may have been overwhelmed due to ingesting additional traffic.

Restoration Actions

According to our supplier, the downstream Operator's partner worked on rolling back some of the volume to reduce the overload on the network hardware. Also, the downstream Operator along with their vendor remediated the incident and all messages that were held in queue were successfully delivered, albeit with delay. Our supplier confirmed that no messages were lost during the incident.

Next Steps

  • Downstream Operator’s partner will work on throttling traffic to mitigate network hardware overload issues.
  • If you have any questions about what numbering options are available to you that would have avoided this issue, please reach out to your account manager to explore them.

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/TkfdpW6W4iuNECzQ6

Posted Oct 31, 2024 - 18:32 UTC

Resolved
This incident has been resolved.
Posted Oct 25, 2024 - 00:26 UTC
Monitoring
A fix has been implemented and we are monitoring the results. All messages that were held in queue were successfully delivered, with delay. No messages were lost during the incident.
Posted Oct 25, 2024 - 00:02 UTC
Update
We are observing improved performance to Verizon.
Posted Oct 24, 2024 - 23:13 UTC
Identified
We have been notified by our supplier that the incident has been identified. Also we are observing improved performed to AT&T
Posted Oct 24, 2024 - 22:02 UTC
Investigating
We have been notified by our supplier about delivery issues for SMS from 10DLC numbers towards AT&T and Verizon networks in USA.

The carrier is aware of this issue and investigating the root cause. We will issue an update as soon as more information becomes available.

Please contact support@api.vonage.com if you have any questions.
Posted Oct 24, 2024 - 17:58 UTC
This incident affected: Global Connectivity (SMS North America).