Service Issue in Platform: Voice API

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

2024-03-12 14:33 UTC until 2024-03-12 16:36 UTC

‌

Services Impacted

Programmable SIP calls

‌

Summary

On 2024-03-12, from 14:33 UTC until 16:36 UTC, Customers sending inbound calls via Programmable SIP from a third-party shared platform were rejected with a 403 (Unauthorized)

Root Cause

Following a request to bypass SIP Trunking Digest Authentication for a customer, all calls from a third-party shared platform were routed through a single API key. All SIP Trunking and Programmable SIP calls from customers using the same originating shared platform failed due to being routed to the single API key, and calls were not routed to the intended Programmable SIP Domain or PSTN. Calls affected failed with a 403. Customers not originating calls from the same third-party shared platform were not affected.

‌

Restoration Actions

The IP addresses added for the Digest Authentication bypass were removed.

‌

Next Steps

To improve internal routing to classify Programmable SIP calls based on the domain instead of IP addresses.

To improve the SIP Trunking Digest Authentication Bypass process to not permit shared platform IP addresses to be added to the allow list.

Posted Mar 15, 2024 - 16:20 UTC

Resolved

The issue causing some inbound Programmable SIP calls to fail has been resolved.

The following services were impacted from [12-03-2024 14:33] UTC until [12-03-2024 16:36] UTC.
Programmable SIP: Some customers may have experienced inbound call rejections for the calls processed in US regions.

All services have been restored.
Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.

Posted Mar 12, 2024 - 17:45 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

The following services were impacted from [12-03-2024 14:33] UTC until [12-03-2024 16:36] UTC.
Programmable SIP: Some customers may have experienced inbound call rejections for the calls processed in US regions.

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Mar 12, 2024 - 17:14 UTC

Investigating

Our monitoring has alerted us to a potential service issue with our inbound Programmable SIP Calls. We are alerting customers immediately while our engineering teams investigate.

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://api.support.vonage.com/hc/en-us/articles/6749696044828-Vonage-API-Support-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted Mar 12, 2024 - 16:34 UTC

This incident affected: Voice (Programmable SIP & SIP Interconnect) and Multiple Services Affected (Multiple Services Affected).