Date/Time Impacted
Sep 25, 2024 18:02:00 until Sep 25, 2024 18:08:00
Services Impacted
Voice - Voice API
Summary
A configuration error inside Vonage systems resulted in high volumes of call retries. These retries triggered internal rate limiting to take effect.
Root Cause
Due to a configuration error for one of our major suppliers, calls on the affected trunk were retried 32 times. This excessive volume of calls triggered internal rate limiting which was designed to protect Vonage systems from high traffic volumes. The effect of this rate limiting was that the vast majority of calls, via the affected edge routers, were blocked.
Restoration Actions
A reduction in excess CPS allowed the system to correct within a 6-minute timeframe.
Next Steps
Retry logic has been changed in two areas of the platform to prevent this amplification of traffic from happening again.
A full review of carrier trunk configuration will be done to ensure the same configuration error does not exist elsewhere.
We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/BRDnAmmotJVzfp4e6