Incident Statement - Messages API database issues

Date/Time Impacted

2023-01-27, 20:46 UTC - 2023-01-27 20:58 UTC

‌

Services Impacted

Messages API Platform

Dispatch 

Verify

‌

Summary

A software defect in our partner database resulted in connections being dropped causing errors 5xx to be returned by Messages API, Dispatch and Verify in EU and AP. For Messages API the connection drop caused also for outbound messages to be processed multiple times.

At 20:51 UTC our monitoring alerted us to the issue, the incident recovered at 20:58 UTC, and investigations into the root cause started at 21:04 UTC.

‌

Root Cause

The component in our partner database in charge of managing the compute resources triggered a known issue, causing all database resources to be restarted and dropping all connections from our Messages API, Dispatch and Verify in EU and AP.

‌

Restoration Actions

The 3rd Party Service owner has acknowledged the issue and is currently working on a resolution.

‌

Next Steps

Plans are underway to migrate the Messages API services in order to ensure this issue does not recur.  Timelines for completion of this work are currently TBC.