Service Issue in Platform

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

2022-07-08 09:20 UTC - 2022-07-08 12:02 UTC

Services Impacted

SMS API

Summary

Our backend services in the Asia Pacific region incorrectly responded to some SMS API requests with status code 2 ‘missing from parameter’ .

At 11:21 UTC our monitoring alerted us to the issue, investigations into root cause started at 11:56 UTC and mitigating actions were performed at 12:02 UTC.

‌

Root Cause

A deployment of our backend services in the Asia Pacific region indirectly triggered a bug which incorrectly parsed API parameters.

‌

Restoration Actions

We routed traffic to a different cluster of backend services.

Next Steps

Fix the bug that caused incorrect status code 2 responses
Add automated alerting for high rates of rejections.
Automatically fail-over to backup cluster when alerts are triggered.

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/oHG3KinmGarhpvU78

Posted Jul 15, 2022 - 09:47 UTC

Resolved

This problem has been resolved.

All services have been restored.

Please let support@api.vonage.com know if you continue to experience problems with this.

Posted Jul 08, 2022 - 15:08 UTC

Monitoring

This issue has been resolved.

The following services were impacted from 2022/07/08 08:28 UTC until 2022/07/08 12:02 UTC:

SMS API: Some customers may have experienced errors with status code 2 ‘missing from parameter’ on their outbound SMS requests in the Asia Pacific region.

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Jul 08, 2022 - 12:27 UTC

Investigating

Our monitoring has alerted us to a service issue with our platform.

If there is a customer impact, it is likely to be affecting the following products:

SMS API: Some customers may experience status code 2 ‘missing from param’ submitting outbound SMS

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://help.nexmo.com/hc/en-us/articles/360015693092-Nexmo-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted Jul 08, 2022 - 12:08 UTC

This incident affected: SMS & Messages (SMS).