Service Issue in Messages API Platform: WhatsApp

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

Dec 11, 2024 17:59 UTC - Dec 11, 2024 21:00 UTC

‌

Services Impacted

Messages API, Verify V2

‌

Summary

Meta had a global outage which caused Meta's graph-api to respond with HTTP 500 errors when submitting outbound messages. Outbound (WhatsApp on-premises and cloud-api) messages resulted in undeliverable callback statuses with error code 1030. Inbound WhatsApp messages were also delayed by up to 1 hour.

This impact was limited to WhatsApp outbound and inbound messages via our Messages API, as well as our Verify 2 service.

‌

Root Cause

There was a global outage with our partner, Meta.

‌

Restoration Actions

Our Engineering team continuously worked with Meta until the service was completely restored by Meta.

‌

Next Steps

Improve our internal processes regarding the validation of impacted dependent products.

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/oHG3KinmGarhpvU78

Posted Dec 16, 2024 - 03:38 UTC

Resolved

This incident has been resolved.

Posted Dec 12, 2024 - 01:07 UTC

Update

We are observing stable delivery of Verify and Messages traffic and are continuing to work with Meta on the most updated information as they work towards a resolution. We will provide an update in 2 hours.

Posted Dec 11, 2024 - 23:00 UTC

Monitoring

We are currently observing stable delivery of Verify and Messages traffic, and are continuing to work with Meta on the most updated information as they work towards a resolution. We will provide an update in 1 hour.

Posted Dec 11, 2024 - 22:01 UTC

Update

We are continuing to monitor delivery improvements from Meta and will provide additional updates as we have more information.

Posted Dec 11, 2024 - 21:24 UTC

Update

We are currently observing a recovery in Verify and Messages traffic and are monitoring the results to update towards a resolution.

We are monitoring delivery improvements from Meta and will provide additional updates as we have more information.

Posted Dec 11, 2024 - 20:53 UTC

Update

We are continuing to work on a fix for this issue.

Posted Dec 11, 2024 - 20:33 UTC

Identified

We continue to experience issues with our provider Meta. We will provide updates as soon as situation changes. We are working diligently with Meta to get additional/most updated information as they work through these issues.

Posted Dec 11, 2024 - 20:13 UTC

Update

We are continuing to investigate this issue.

Posted Dec 11, 2024 - 19:51 UTC

Update

We are currently experiencing disruption impacting message delivery on WhatsApp Cloud API. This issue started at 17:59 UTC UTC. Our service provider Meta is investigating the issue.

Impacted products: WhatsApp, Verify

Posted Dec 11, 2024 - 18:23 UTC

Investigating

We are currently experiencing disruption impacting message delivery on WhatsApp Cloud API. This issue started at 17:59 UTC UTC. Our service provider Meta is investigating the issue.

Posted Dec 11, 2024 - 18:17 UTC

This incident affected: Auth & Identity (Verify, Verify V2) and Global Connectivity (Whatsapp Connectivity).