[SMS and Verify Service Incident] Service disruption for SMS and Verify channels

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

2023-07-01 10:10:00 UTC - 2023-07-01 12:20:00 UTC

‌

Services Impacted

SMS
Messages API
Verify API

‌

Summary

Between 2023-07-01 10:10 UTC and 2023-07-01 12:20 UTC, customers experienced delayed outbound SMS via the SMS and Messages API. Verify API requests experienced delayed or expired requests due to increased latency on SMS and Voice channels.

‌

Root Cause

Our node health checks - responsible for determining service availability - incorrectly reported some nodes as unavailable. The incorrect health check reports caused the nodes to stop receiving requests. As a result, our capacity to process customer requests was reduced, ultimately causing increased latency and queuing customer requests.

‌

Restoration Actions

We removed the problematic node health checks from our systems

‌

Next Steps

The bug causing the erroneous health checks has already been fixed
Increase service resiliency by adding additional resources in other availability zones
Review and improve our automation and remediation processes

Posted Jul 06, 2023 - 14:35 UTC

Resolved

The issue our platform experienced has been resolved.

The list of services that were impacted has been confirmed as:
SMS, SMPP, Messages API, Verify, Subaccount API, Numbers API

The incident started at 10:10 UTC and the final service recovered was SMS at 12:20 UTC, with other services recovering much sooner.

All services have been restored.

Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.

Posted Jul 01, 2023 - 15:40 UTC

Monitoring

We have implemented a fix for the issue affecting multiple of our products.

The list of services that were impacted has been confirmed as:
SMS, SMPP, Messages API, Verify, Subaccount API, Numbers API

The incident started at 10:10 UTC and the final service recovered was SMS at 12:20 UTC with other services recovering much sooner.

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Jul 01, 2023 - 13:06 UTC

Identified

We have identified the root cause of the issue affecting our services.

We are currently working on getting the issue fixed and will post further updates here.

Posted Jul 01, 2023 - 11:02 UTC

Update

We are continuing to investigate this issue.

Posted Jul 01, 2023 - 10:39 UTC

Investigating

Customers are currently experiencing problems impacting our SMS and Voice service.

Our engineering team is currently investigating the impact of the issue.

We will update this status as soon as we have more information on this issue.

Posted Jul 01, 2023 - 10:35 UTC

This incident affected: SMS & Messages (SMS, SMPP, Messages API), Voice (Voice API), Auth & Identity (Verify), and Vonage Platform Management (Account APIs, Numbers API, SubAccount API).