Date/Time Impacted

2024-06-24 15:05 UTC until 2024-06-24 16:17 UTC

‌

Services Impacted

SMS (HTTP and SMPP)

Messages API

Verify API 

‌

Summary

Our backend services responsible for managing service discovery across our APIs experienced a configuration problem that led to service disruptions in SMS  and Voice platforms.

At 15:05 UTC our monitoring alerted us to a problem impacting the SMS platform. Some customers experienced rejected SMS MT requests using the SMS API, Messages API, and Verify API. HTTP customers may have received error code 5 and SMPP customers may have received error codes 8 and 69 during the incident.

Once the issue was identified, we started implementing mitigation actions at 15:16 UTC, and a full recovery was observed at 16:17 UTC, marking the end of customer impact.

‌

Root Cause

This incident originated from an internal change that was deployed on June 24th at 14:57 UTC in our backend service that manages service discovery across all APIs. This change has created service instability in multiple APIs, including the SMS platform. As a result, some customers experienced rejected SMS MT requests.

‌

Restoration Actions

The restoration actions started at 15:16 UTC by reverting the change across services for the next hour.

Next Steps

• To implement more robust pre-deployment testing in these services
• To implement a rollout process in these services with per-region soaking

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/oHG3KinmGarhpvU78