Service Issue in Platform : Messages API

Incident Report for Vonage API Status

Postmortem

Meta Business Manager service issues

Date/Time impacted:

2023-08-16 09:35:00 UTC - 2023-08-21T20:20:00 UTC

Services impacted:

The below services were impacted:

WhatsApp Accounts:

During this incident, we were unable to onboard new customers. This applied to partners or direct customers onboarded by us, and self-service customers who use the Vonage dashboard to set up their accounts.

WhatsApp Message delivery: Some existing customers' WhatsApp accounts may have been restricted due to this and they may have experienced issues in outbound and inbound WhatsApp message delivery.

Messenger Delivery: Some customers may have experienced issues with failed messenger delivery.

Blocked Facebook Account users: Some users may have been blocked from their Facebook account and therefore be unable to access their Meta Business Manager. You are advised to reset your password with 2FA and follow the unblock instructions shown on your account page. Please contact support@api.vonage.com if this is not possible for you.

Summary

On 16th August 2023, We identified a downstream service issue on Meta's business manager platform impacting onboarding new Messenger, Instagram and WhatsApp accounts. This was due to phishing attempts detected by Meta as intending to compromise their access security.

During this incident, most existing WhatsApp Business Accounts (WABAs) were able to send and receive messages as normal and were not impacted. A small number of WABAs have been blocked and as a result, a very small number of WABAs have had delivery issues, most of these services have been restored following a review with Meta.

Restoration actions

Following close collaboration with Meta regarding reviews of suspicious activity, accounts, and users, our normal service has resumed.

Root Cause

Attempts were made by unknown bad actors to compromise Meta’s 2fa access security via phishing emails. If you receive emails related to your account(s) being blocked, do not click on the links in the emails. Instead, please go to www.facebook.com or use the Facebook app to confirm and resolve.

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/vaeCotuCHUQCCqs49

Posted Aug 25, 2023 - 19:36 UTC

Resolved

This issue has been resolved.

The following services were impacted from 2023-08-16 09:35 UTC - 2023-08-21 20:20 UTC

WhatsApp Accounts:
During this incident, we were unable to onboard new customers. This applied to partners or direct customers onboarded by us, and self-service customers who use the Vonage dashboard to set up their accounts.

WhatsApp Message delivery: Some existing customers' WhatsApp accounts may have been restricted due to this and they may have experienced issues in outbound and inbound WhatsApp message delivery.

Messenger Delivery: Some customers may have experienced issues with failed messenger delivery.

Blocked Facebook Account users: Some users may have been blocked from their Facebook account and therefore be unable to access their Meta Business Manager. You are advised to reset your password with 2FA and follow the unblock instructions shown on your account page. Please contact support@api.vonage.com if this is not possible for you.

The below have not been impacted:
Viber Message delivery
SMS and MMS delivery using Messages API
Most customers' existing WhatsApp accounts and delivery were not impacted due to this incident

All services have been restored.
Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.

Posted Aug 23, 2023 - 15:13 UTC

Monitoring

This issue has been resolved and we are monitoring the results.

Services were impacted between 2023-08-16 09:35 UTC - 2023-08-21 20:20 UTC

The below services have been restored:

NEW WhatsApp for Business Accounts (WABAs) :
Access to create new WABAs has been restored. This includes:

Self-service for direct customers via embedded sign-up
Adding new WABAs through the managed process.
New WABAs can be created
New numbers can be added to WABAs

Sending Messages
Most existing WhatsApp Business Accounts (WABAs) are able to send and receive messages as normal and are not impacted.
As a result of the suspicious activity, we have had to block a small number of WABAs for suspected fraud. If you have been blocked, please escalate to your account manager Please include the customer’s WABA ID and Account Name.
If you do not have an account manager, email the Vonage Support team at support@api.vonage.com. Please include your WABA ID and account name.

Blocked Facebook Account users:
Some users may have been blocked from their Facebook account and therefore unable to access their Facebook Business Manager. You are advised to reset your password with 2FA and follow the unblock instructions shown on your account page. Please contact support@api.vonage.com if this is not possible for you.

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Aug 22, 2023 - 11:56 UTC

Update

We are continuing to work with our third-party provider on getting the issue fixed and will post further updates here.

The below services are still impacted:

WhatsApp Accounts:
- We are temporarily unable to onboard new accounts. This applies to partners or direct customers onboarded by us, and self-service customers who use the Vonage dashboard to set up their accounts.

Some existing customers' WhatsApp accounts may have been restricted due to this and they may experience issues in outbound and inbound WhatsApp message delivery.

Action point: Please tell us your email and WABA ID by emailing our support team at support@api.vonage.com and we will ensure your accounts are unblocked as soon as possible by raising with the Meta team.

Blocked WhatsApp users: Some blocked users may not be able to reset their access via 2FA so those users with blocked access to the Facebook business ID should also contact us so we can re-set the access.

Meta Business Manager: Customers may have issues accessing their Meta Business Manager because their Facebook account is blocked.

Action point: If customers are blocked from their Meta Business Manager, or Facebook account and are unable to reset with 2fA, please have them contact us at support@api.vonage.com with the Name, email address, and phone number associated with the account.

We no longer believe Messenger delivery is an issue at this time.

We will update this incident once we have more information.

Please see this article - https://api.support.vonage.com/hc/en-us/articles/6749696044828-Vonage-API-Support-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted Aug 21, 2023 - 09:31 UTC

Update

Our monitoring has alerted us that Messenger delivery is also impacted and customers may see failed Messenger delivery. We are continuing to work with our third-party provider on getting the issue fixed and will post further updates here.

Posted Aug 19, 2023 - 10:57 UTC

Update

We are continuing to work with our third-party provider on getting the issue fixed and will post further updates here.

The list of services that were impacted and the actions to be taken have been outlined below:

WhatsApp Accounts:
- Newly onboarded customers who used embedded signup may have issues sending messages as we can't attach credit lines.
- We are unable to onboard new accounts for partners or direct customers going through the manual onboarding process as we are blocked from creating new WABAs.

Some existing customers' WhatsApp accounts may have been restricted due to this and they may experience issues in outbound and inbound WhatsApp message delivery.

Action point: Please tell us your email and WABA ID by emailing our support team at support@api.vonage.com and we will ensure your accounts are unblocked as soon as possible by raising with the Meta team.

Blocked WhatsApp users: Some blocked users may not be able to reset their access via 2FA so those users with blocked access to the Facebook business ID should also contact us so we can re-set the access.

Meta Business Manager: Customers may have issues accessing their Meta Business Manager because their Facebook account is blocked.
Action point: If customers are blocked from their Meta Business Manager, or Facebook account and are unable to reset with 2fA, please have them contact us with the Name, email address, and phone number associated with the account.

Posted Aug 18, 2023 - 11:56 UTC

Update

We are continuing to work with our third-party provider on getting the issue fixed and will post further updates here.

Posted Aug 17, 2023 - 13:27 UTC

Identified

Our monitoring has alerted us to a downstream service issue on Meta's business manager platform that may impact onboarding for Messenger, Instagram and WhatsApp.
A subset of existing WhatsApp accounts may have been restricted due to this and they may experience issues in outbound and inbound WhatsApp message delivery.

We are currently working with our third-party provider to get the issue fixed and will post further updates here.

Please see this article - https://api.support.vonage.com/hc/en-us/articles/360015693092-Nexmo-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted Aug 17, 2023 - 12:35 UTC

This incident affected: Global Connectivity (Whatsapp Connectivity, Facebook Messenger Connectivity).