Date/Time Impacted

2024-08-12 11:43 UTC until 2024-08-12 13:08 UTC

‌

Services Impacted

Voice API

‌

Summary

A third party service provider responsible for managing data across our APIs experienced failure and instability that led to service disruptions in processing the non SIP trunking calls.

‌

At 11:43 UTC our monitoring alerted us to a problem impacting multiple Voice products except for the SIP trunking.

Some customers observed intermittent call failures while creating new calls or while modifying active or in progress calls in the Singapore region during this incident.

‌

Once the issues were identified, we started mitigating actions at 12:03 UTC, and a final recovery was observed at 13:08 UTC marking the end of customer impact.

‌

‌

Root Cause

This incident occurred when one of our third-party managed nodes who is responsible to process the call data started to failover causing for a rebalancing to occur repeatedly.This led to the timeouts.As a result customers experienced call failures for new and ongoing calls in the Singapore region.

‌

Restoration Actions

 The mitigation activity started at 12:03 UTC by re-starting the service multiple times until we saw the services in the operational state.

‌

Next Steps

• A support case was raised with our third-party provider, so that the root cause on their side can be determined and they can apply a fix.
• To automate the remediation actions, so that they can immediately resolve any impact, if our third party has the same incident in the future.
• To check feasibility on adding failover capabilities to another region especially for customers not using any specific regional domains.This will not work for customers if they use the regional domains.

‌

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/oHG3KinmGarhpvU78