Service issue on WhatsApp channel

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

Oct 14, 2024 23:57 UTC - Oct 15, 2024 04:48 UTC

‌

Services Impacted

Messages API - WhatsApp Channel

Summary

A major Meta Whatsapp Business API disruption impacted some inbound Whatsapp messages and delivered/read outbound message callbacks for Whatsapp messages. All affected messages and callbacks were lost and unrecoverable.

This impact was limited to Whatsapp messages sent via our Messages API. Other products, including Verify service which was originally reported as experiencing degraded performance, were not impacted.

‌

Root Cause

There was a major service disruption with our upstream partner, Meta. They pushed out a configuration change which affected the services. The issue was mitigated when Meta reverted that change.

‌

Restoration Actions

Our Engineering team continuously worked with Meta until the service was completely restored by Meta.

‌

Next Steps

Improve our alerting system to reduce detection time.

Improve our internal processes regarding the validation of impacted dependent products.

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/oHG3KinmGarhpvU78

Posted Oct 17, 2024 - 11:43 UTC

Resolved

This incident has been resolved.

Posted Oct 15, 2024 - 06:47 UTC

Update

We are continuing to monitor for any further issues.

Posted Oct 15, 2024 - 05:49 UTC

Monitoring

A fix has been implemented by Meta and we are monitoring the results.

Posted Oct 15, 2024 - 05:35 UTC

Identified

An issue with our partner, Meta, has been identified.
We aim to provide an update within 1 hour.

Posted Oct 15, 2024 - 05:24 UTC

Update

We are continuing to investigate this issue.

Posted Oct 15, 2024 - 04:20 UTC

Investigating

Our monitoring has alerted us to a potential issue with callback processing on the WhatsApp channel that may impact inbound traffic and delivered/read DLRs which started at 23:57 UTC.

We aim to provide an update on our investigation within 1 hour.

You can refer to this article - https://api.support.vonage.com/hc/en-us/articles/6749696044828-Vonage-API-Support-Incident-Handling - for an explanation of our approach to publishing incidents.

Please contact support@api.vonage.com if you have any questions.

Posted Oct 15, 2024 - 04:19 UTC

This incident affected: Multiple Services Affected (Multiple Services Affected), Auth & Identity (Verify, Verify V2), and SMS & Messages (Messages API, Dispatch).