Incident Statement -  Latency in studio insights API in the EU region

Date/Time Impacted (UTC)

Start Time:  Oct 19, 2023 11:43:00

End Time:  Oct 19, 2023 19:09:00

Services Impacted

• AI Studio Reports - EU Region

Summary

On Oct 19, 2023 11:43 (UTC),  the AI Studio Reports Service experienced delays in the EU region due to insufficient capacity following an unexpectedly high increase in demand. This triggered fallback call flows for some real-time AI Studio/Virtual Assistant customers. 

To prevent recurrence, we have increased capacity and will accelerate planned initiatives to fully replace the Reporting service with an auto-scaling architecture. We will also implement traffic controls to avoid overloading the system in the interim.

We are committed to enhancing monitoring and alerts to proactively detect issues moving forward.

In summary, this incident underscores the priority of upgrading our backend systems to be resilient at scale. We will increase investment to accelerate initiatives like the Reporting service replacement. Our goal is to uphold reliability and transparency for customers.

Root Cause

The root cause was an unexpected traffic spike that caused latency due to queuing in the AI Studio Reports service in EU, since it had manual components.

‌

Restoration Actions

• Added more resources to the current AI Studio reports service components in EU region.

‌

Next Steps

• Adjust the latency monitoring to enable detection in an earlier stage.
• Replace the reporting components of the system with new auto scalable components.

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey:

https://forms.gle/yVq3eKKaP3SHHaKa9