SMS Delay in US Data Center

Date/Time Impacted

2023-10-10 22:15:00 - 22:39:00 UTC

‌

Services Impacted

SMS - HTTP, SMPP

Message API

Verify API

Dispatch API

‌

Summary

On October 10th 2023 at 22:15:00 UTC, our monitoring system alerted us about increasing SMS volume in the US datacenter. Our engineering team investigated the alert and observed SMS delay issues. The issue was resolved by automatic failover. The root cause of the delay was identified later and customer impact was evaluated. 

‌

Root Cause

Kafka MSK maintenance in one of our US datacenters disrupted communication between some internal SMS services, which resulted in delays in SMS traffic.. 

‌

Restoration Actions

On October 10th 2023 at 22:39:00 UTC, impacted traffic automatically failed over to another datacenter, no more SMS delays. 

Some messages were stuck. These messages were processed on October 11th 2023 at 10:04:00 UTC.

‌

Next Steps

In order to more quickly detect and resolve potential issues in the future, Vonage will enhance current internal processes to reduce the recovery time by including the MSK status checks as part of the incident response process and team training. To reduce resolution time the stuck messages alert severity will be increased. We also plan to improve the underlying code to minimize the delays caused by the Kafka MSK maintenance to reduce the breadth of the impact.