Service Issue in Platform - Vonage API Dashboard

Incident Report for Vonage API Status

Postmortem

Date/Time Impacted

2022-12-13 14:40 UTC - 2022-12-13 15:08 UTC

‌

Services Impacted

Vonage API Dashboard

‌

Summary

On 2022-12-13 at 14:40 UTC, Vonage API Dashboard users began to experience authentication issues accessing the Vonage API Dashboard. At the time of initial customer impact, engineers were aware of a potential problem and we identified the specific customer impact at 14:50 UTC. We implemented a fix at 15:08 UTC and this corrective action prevented any further customer impact. This issue did not impact live customer traffic.

‌

Root Cause

The Vonage API Dashboard includes a security component for authenticating users that was subject to a routine configuration change at 14:40 UTC.

After the change was committed, the relevant component was restarted - as expected - triggering the Vonage API Dashboard service to fetch the latest software version. However, the software version included in this change was incorrect, causing the Vonage API Dashboard service to retrieve a software version incompatible with the authentication component. We discovered the potential issue immediately, however, upon identifying the specific cause, our normal practice of rolling back problematic changes failed. Instead, we reverted to a previous stable software release and manually pointed to the correct version.

Ultimately, the Vonage API Dashboard retrieved an incorrect software version resulting in the inability to authenticate Vonage API Dashboard users.

‌

Restoration Actions

Software was reverted back to a previous version and pointed to the correct version.

‌

Next Steps

Fix the problem that prevented an immediate rollback to pre-change configuration
We are introducing steps to duplicate the authentication component to ensure that there is always at least one spare instance running on each server. This will enable us to monitor changes made to the software before fully committing to updates.

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/CaiZE4ScG118ZmkS6

Posted Dec 15, 2022 - 17:23 UTC

Resolved

This problem has been resolved, and all services have been restored.

Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.

Posted Dec 13, 2022 - 16:00 UTC

Monitoring

We have implemented a fix for this issue.

The list of services that were impacted have been confirmed as:
Vonage API Dashboard

Between 2022-12-13 14:50 UTC and 2022-12-13 15:08 UTC customers may have experienced authentication errors attempting to access the Vonage API Dashboard. Live customer traffic was not impacted by this issue.

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Dec 13, 2022 - 15:19 UTC

Investigating

Our monitoring has alerted us to a potential service issue with our platform. Although It is not yet certain whether there is any customer impact, we are alerting customers immediately while our engineering teams investigate.

If there is a customer impact, it is likely to be affecting the following products:
Vonage API dashboard

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://api.support.vonage.com/hc/en-us/articles/360015693092-Nexmo-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted Dec 13, 2022 - 15:01 UTC

This incident affected: Vonage Dashboard (Account Access & Notifications).