SMS Service Issue in Platform

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

2023-01-31 10:07 UTC - 2023-01-31 10:39 UTC

‌

Services Impacted

SMS API via HTTP and SMPP

‌

Summary

On 2023-01-31 10:07 UTC, customers sending outbound SMS messages via SMS API (HTTP and a small amount of SMPP) began to experience delays in the delivery of the messages. After the messages were accepted by the Vonage API platform, they were not routed in a timely manner due to an issue with our global routing systems.

We implemented a fix and active queues were drained at 10:39 UTC. Further back up queues of messages sent between 10:07 UTC and 10:39 UTC continued to clear and be delivered until 12:41 UTC.

Normal service was resumed for new messages at 10:39 UTC.

‌

Root Cause

The Vonage API platform includes a routing engine that helps route messages globally, with the help of routing entries from another service using HTTP API.

As the number of routing entries increased over time, they reached a limit where the routing engine did not load them correctly because API response size was too large to parse for the client service. As a result of this the routing engine ceased to function.

The issue was fixed by configuring the service to use a backup routing engine.

‌

Restoration Actions

A configuration change was made to use a backup routing engine for all MTs globally.

‌

Next Steps

Increase the maximum HTTP response size in the client service to prevent such issues in future.
Set a limit for maximum entries in the routing database.
Add monitoring alerts on response sizes of dependent services.

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/CaiZE4ScG118ZmkS6

Posted Feb 01, 2023 - 17:30 UTC

Resolved

The issue impacting SMS service has been resolved. Customers may have experienced increased latency on outbound SMS via HTTP and SMPP between 2023-01-31 10:07 UTC and 2023-01-31 10:39 UTC.

The list of services that were impacted have been confirmed as:
HTTP SMS
SMPP SMS

All services have been restored.
Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.

Posted Jan 31, 2023 - 13:15 UTC

Update

While we continue to monitor the incident, we have an update regarding service impact.
Customers may have experienced increased latency on outbound SMS via HTTP and SMPP between 2023-01-31 10:07 UTC and 2023-01-31 10:39 UTC.

The list of services that were impacted have been confirmed as:
HTTP SMS
SMPP SMS

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Jan 31, 2023 - 11:59 UTC

Monitoring

We have implemented a fix for the issue impacting outbound SMS via HTTP. Customers may have experienced increased latency on outbound SMS between 2023-01-31 10:07 UTC and 2023-01-31 10:39 UTC.

The list of services that were impacted have been confirmed as:
HTTP SMS

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Jan 31, 2023 - 10:48 UTC

Identified

Our monitoring has alerted us to a service issue with our platform. Customers may face latency with outbound HTTP SMS.

The issue is affecting the following products:
HTTP SMS

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://api.support.vonage.com/hc/en-us/articles/6749696044828-Vonage-API-Support-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted Jan 31, 2023 - 10:21 UTC

This incident affected: SMS & Messages (SMS, SMPP).