Date/Time Impacted

Dec 19, 2024 18:45 UTC - Dec 19, 2024 21:50 UTC

Services Impacted

Outbound 10DLC SMS and MMS

Summary

The issue observed with outbound rejections was the result of actions taken by Vonage during a response to a critical fraud incident. As the engineering team was implementing a fix to resolve the identified fraud, a misconfiguration was introduced that prevented the processing of some messages. The engineering team identified the issue and took actions to correct the rule, which restored normal service. 

Root Cause

This fraud rule update, introduced as part of a response to an earlier critical incident, inadvertently caused legitimate messages to be rejected with an error code of 99.

Restoration Actions

Once the configuration issue was identified, the engineering team adjusted the new rule to properly reject the identified fraud and process all other messages. 

Next Steps

As a result of this incident Vonage has identified several opportunities to improve our response and prevent similar fraud related response actions from impairing the system. 

• Improvements to the change management process to avoid deployment issues
• Updated Runbooks that will improve our deployment process
• Improvements to the incident management process to reduce resolution times
• Additional training for teams involved in the fraud resolution process
• Additional system alerting to quickly identify issues introduced by the fraud management system

‌

Vonage will continue to review and improve our processes, systems and platform monitoring to prevent these issues in the future. 

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/oHG3KinmGarhpvU78