Date/Time Impacted

2024-10-10 14:24 UTC - 2024-10-10 15:28 UTC

‌

Services Impacted

Outbound SMS, through the following services:

SMS API

SMPP

Messages API

Verify

‌

Summary

Following a routine software deployment, our SMS platform experienced unexpected service impact, resulting in delays delivering outbound SMS traffic. 

During the impact period, SMS API, SMPP and Messages API customers may have experienced delays of up to 66 minutes sending outbound SMS. Verify customers using SMS workflows may have experienced an increase in expired verifications or velocity blocking. 

There was no impact on Delivery Receipts, Inbound SMS or accepting customer SMS submissions.

‌

Root Cause

On 2024-10-10 at 13:22 UTC, we commenced a routine software deployment, with all activities completed by 14:22 UTC.

Due to this taking place at the same time as routine cycling of authentication tokens, the deployment unknowingly included expired authentication credentials for internal services responsible for processing SMS traffic. The incorrect configuration caused an authentication error, resulting in a loss of contact with upstream services; therefore, messages could not be progressed for delivery. Messages were then queued until contact with the relevant upstream services reestablished. 

‌

Restoration Actions

• We redeployed with the new authentication credentials

‌

Next Steps

• We have already fixed the configuration issue included in the deployment.
• Review our change management processes and implement improvements to prevent similar causational impacts.
• Review our monitoring and alerting processes and implement improvements to reduce our time to detect similar issues.

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/9KxEBEsJKCGoN8J69