Messages API - Service issue on WhatsApp channel

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

Aug 30, 2024 00:25 UTC - Aug 30, 2024 04:55 UTC

‌

Services Impacted

Messages API - WhatsApp Channel

‌

Summary

A problem in our internal Meta traffic processing application caused Meta to stop sending inbound messages and outbound traffic callbacks to one of our data centers. Unfortunately, these callbacks are unrecoverable. Inbound messages were lost during this period. Outbound messages continued to be sent correctly, but their delivery/read callbacks were lost.

Root Cause

One of our internal EU Meta applications’ data usage compliance checks expired. Vonage carries out this compliance check annually, and in this instance, we failed to complete it by the deadline, so it has expired.

‌

Restoration Actions

Our Engineering team completed the expired application's data check-up, and the affected callback traffic was finally resumed.

‌

Next Steps

  • Improve our internal process to prevent our Meta application from expiring 
  • Enhance alert notifications for such failures.
Posted Sep 03, 2024 - 14:36 UTC

Resolved

This incident has been resolved.
Posted Aug 30, 2024 - 05:38 UTC

Update

We are continuing to monitor for any further issues.
Posted Aug 30, 2024 - 05:11 UTC

Monitoring

We are monitoring the results.
Posted Aug 30, 2024 - 05:07 UTC

Identified

The issue has been identified and a fix has been implemented
Posted Aug 30, 2024 - 05:05 UTC

Investigating

Our monitoring has alerted us to a potential issue on callback processing on the WhatsApp channel that maybe impacting inbound traffic and DLRs.

We aim to provide an update on our investigation within 1 hour.

You can refer to this article - https://api.support.vonage.com/hc/en-us/articles/6749696044828-Vonage-API-Support-Incident-Handling - for an explanation of our approach to publishing incidents.

Please contact support@api.vonage.com if you have any questions.
Posted Aug 30, 2024 - 04:09 UTC
This incident affected: SMS & Messages (Messages API).