Service Issue in Platform

Incident Report for Vonage API Status

Postmortem

Incident Statement - SMS Rejections

Date/Time Impacted

2024/02/14 13:09:00 UTC - 2024/02/14 13:23:00 UTC

‌

Services Impacted

Messages - Messages API Platform
SMS - SMS API Platform
SMS - SMPP
Auth & Identity - Verify
Auth & Identity - Verify V2

‌

Summary

Customers sending SMS through impacted services experienced an unexpected increase in rejected messages. The error code associated with the erroneous rejections was 11 - Unrouteable.

‌

Root Cause

A routine software deployment caused an unexpected bug to manifest in production. This bug impacted our internal routing engine, which caused a portion of messages to become unrouteable. The inability to route messages upstream resulted in customers to experience rejections.

‌

Restoration Actions

We rolled back the problematic deployment.

‌

Next Steps

We have already fixed the bug in the software deployment
Review and improve our deployment processes by adding enhanced pre-production tests and deployment sequencing.
Enhance our routing engine monitoring and alerting to reduce detection times

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/huA689KhLaTKBCjS9

Posted Feb 16, 2024 - 10:27 UTC

Resolved

This incident has been resolved.
Please contact support@api.vonage.com if you have any questions.

Posted Feb 14, 2024 - 17:44 UTC

Monitoring

Our monitoring has alerted us to a service issue where customers may have experienced SMS rejections.

SMS sent via following products have been impacted between 14 Feb 2024 13:09 and 14 Feb 2024 13:23 UTC:
SMS API including SMPP customers
Messages API
Verify API

The issue has already been resolved. We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Feb 14, 2024 - 14:01 UTC

This incident affected: SMS & Messages (SMS, SMPP, Messages API), Multiple Services Affected (Multiple Services Affected), and Auth & Identity (Verify).