Voice API Failures Due to High CPS and Invalid Traffic

Date/Time Impacted

Beginning - October 18th 2023, 19:14 UTC

Resolved - October 18th 2023, 20:26 UTC

‌

Services Impacted

Voice API

‌

Summary

Recently, we experienced a higher-than-usual volume of voice traffic, which led to a temporary disruption in our services. This was primarily due to an unexpected increase in call attempts per second (CPS) beyond the standard limits. Our systems, designed to ensure fair usage and optimal performance for all our users, automatically adjusted to manage this surge.

‌

Root Cause

During the impact window, there was an unexpected increase in call attempts per second (CPS) beyond the standard limits, some of which were invalid. 

Our system is designed to ensure fair usage and optimal performance for all our users. However, as it attempted to make automated adjustments to manage this surge, it resulted in a higher-than-expected number of calls being rejected, leading to a temporary service disruption for some of our customers.

‌

Restoration Actions

As a corrective measure, we implemented additional checks to block invalid calls and are improving our automated system's ability to adjust to these types of surges.

‌

Next Steps

Since restorative actions have been taken, all systems are stable.