Date/Time Impacted

Between Apr 21, 2023 01:20:00 UTC and Apr 21, 2023 10:11:00 UTC

‌

Services Impacted

10DLC traffic across all US networks

‌

Summary

Our Suppliers started rejecting 10DLC traffic across all US networks since 01:20 UTC 21 April, with almost all 10DLC MT messages failing (UNDELIV) with error code 61. This was recovered at 10:11 UTC 21 April.

‌

Root Cause

A routing rule was removed by our supplier which impacted Vonage traffic through them directly. The supplier was not aware that this would impact Vonage traffic.

‌

Restoration Actions

A fix was implemented by our supplier to restore Vonage traffic.

‌

Next Steps

To prevent similar issues from occurring in the future, our supplier will be implementing the following measures:

• Conducting a review of their change management processes to ensure that changes are thoroughly tested and validated before they are implemented.
• Ensure post-change monitoring is performed for several hours (or longer when required).
• Proactively communicate with Vonage on any change that could have a major impact on our messaging traffic.
• Add monitoring and alerting to better detect issues related to the Vonage account.
• Internal review of the support case with all stakeholders to highlight areas where improvements are required (particularly initial diagnosis and email/phone responsiveness).
• Our supplier’s Service Operations to introduce a validation step in their UI when similar API queries are triggered.