Date/Time Impacted

2023-01-23 01:42 UTC - 2023-01-23 04:49 UTC

Services Impacted

Programmable SIP and SIP Trunk outbound

Summary

On January 23rd 2024, between 01:42 and 04:49 UTC, some customers experienced issues attempting to make calls using our Virginia (us-3) Data Center. Specifically, customers using Programmable SIP and SIP Trunking services to connect to the PSTN may have experienced call cancellations or timeouts during this time. Approximately 3.07% of global voice traffic was affected by this incident. The root cause of the issue was identified to be a problem with the discovery of new and updated servers by our proxies.

Root Cause

While updating and deploying servers supporting Programmable SIP and SIP Trunking, a configuration issue with an internal service prevented the proxies from discovering and monitoring the status of these servers. Consequently, the servers rejected calls and caused failed or canceled calls.

Restoration Actions

The configuration issue of discovery of the new instances was fixed in the internal services, and calls from the new and updated servers began to be accepted.

Next Steps

• Improve deployment processes to reduce the chance of incidents and the impact of any incidents.
• Improve the process to include post-deployment checks during deployments on new instances.
• Implement new monitoring and alerting for call failures in the proxies.
• Implement enhanced monitoring on startup for new instances and deployment.

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/jtryntZLqfJ1jZDf6