Date/Time Impacted

Sep 14, 2024 17:50 UTC - Sep 14, 2024 18:40 UTC

‌

Services Impacted

Verify API V1 (Legacy)

Verify API V2 

‌

Summary

Fraudulent traffic from one of our customers caused excessive memory usage in our database, which affected a small amount of Verify requests. For these requests, the PIN code may not have been sent, the validation of the received PIN may have failed, or the requests may have expired.

‌

Root Cause

A sudden surge in fraudulent traffic from one of our customers caused our database to reach its memory limit, which may have resulted in Verify traffic being expired or failed for some customers' Verify requests. This traffic should have triggered a network block for that API key/network in the Verify platform. But that didn't happen, resulting in high memory usage and affecting the other customers' verify requests. Our engineers are investigating why the autoblock didn't work for this fraudulent traffic.

‌

Restoration Actions

We have remediated the source of the fraudulent traffic by manually blocking it and increasing the memory limit on our database.

‌

Next Steps

• To create alerts based on traffic anomalies. 
• To investigate the cause of why the fraudulent traffic was not being blocked automatically.
• To review current database usage and to increase capacity accordingly.