Date/Time Impacted

First occurrence of issue: 2023-05-04 10:05 UTC

Incident fully resolved: 2023-05-04 17:39 UTC

‌

Services Impacted

SMS via TFN (Toll Free Numbers) to US and CA

‌

Summary

• A bug was released to our SMS platform affecting the UDH of messages sent.
• Our TFN provider did not provide an invalid response for malformed messages, causing them to be retried by us blocking valid traffic and subsequently creating queues
• We received internal queue alerts for our TFN provider at 2023-05-04 10:51 UTC
• Upon identifying the issue, the initial SMS servers deployment was rolled back to correct UDH issue at 2023-05-04 13:19 UTC
• Upon escalating to the TFN provider, a hotfix was implemented to respond with an “invalid” error once an incorrect UDH is identified 2023-05-04 16:41 UTC

‌

Root Cause

On 2023-05-04 09:08:00 UTC we did a software release to our SMS platform that contained a bug, which created malformed UDH of SMS using those servers. 

UDH is a standard parameter, normally used to identify concatenated SMS parts. This bug would have affected concatenated messages towards our providers, which would have normally been rejected on their side.  

In particular, our TFN provider was unable to process malformed UDH and had no failure response. This caused our servers to retry those messages and create a queue, blocking concatenated as well as non-concatenated messages.  This caused some messages to timeout in the queue or to take too long to be delivered. 

Restoration Actions

The software release which contained the bug was rolled back at 2023-05-04 13:19 UTC. Additionally, the TFN provider implemented a hotfix to correct the response to traffic with invalid UDH on 2023-05-04 16:41 UTC, which ultimately helped to clear the queue.

‌

Next Steps

• We are adding further functional testing to ensure UDH issues are identified for future releases, to reduce the likelihood of recurrence.
• We will be enhancing our alerting towards our providers by adding new alerts for increased submission errors and timeouts, to reduce the time to detect and recover.
• Improve our supplier escalation process for critical issues.