Date/Time Impacted

2024-06-24 15:06 UTC until 2024-06-24 16:16 UTC

‌

Services Impacted

Voice API

Programmable SIP

WebSockets

In-App Voice

SIP

‌

Summary

Our backend services responsible for managing service discovery across our APIs experienced a configuration problem that led to service disruptions in Messaging and Voice platforms.

‌

At 15:06 UTC our monitoring alerted us to a problem impacting multiple Voice services. 

Some customers observed call quality degradation resulting in some new call requests failing to be established. In-progress call control actions via API(s) were also impacted.

‌

Once the issues were identified we started mitigating actions at 15:16 UTC and a final recovery was observed at 16:16 UTC marking the end of customer impact.

‌

‌

Root Cause

This incident originated from an internal change that was deployed on June 24th at 14:57 UTC in our backend service that manages our service discovery across all APIs. This has created extensive service instability in multiple APIs, including voice products. As a result, some customers started to see failures on new calls and also experienced issues in controlling the in-progress calls using call control features.

Restoration Actions

 The mitigation activity started at 15:16 UTC and the change was rolled back over the next hour on all affected services.

‌

Next Steps

• To implement more robust pre-deployment testing in these services.
• To implement a rollout process for these services with per-region soaking.

‌

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/oHG3KinmGarhpvU78