Publisher Failures related to token roles 

Date/Time Impacted

Start time: Oct 31, 2023 18:13:00 UTC

End Time: Oct 31, 2023 20:47:00 UTC

Services Impacted

Video - Session Connection

Summary

Following an update to the Video API platform, users experienced publisher failures when attempting to publish a stream to a session, resulting in error 1500. Affected Opentok users would have received a message stating “The token doesn't have permission".

Root Cause

A new version of the Video API platform was deployed to support a new token use case. However, the new version did not support mixed case role names, i.e., PUBLISHER vs. Publisher vs. PuBliSHer, which the previous version supported. This resulted in a request made to the API with all upper or mixed case role names failing.

Restoration Actions

The issue was corrected on Oct 31, 2023, 20:47:00 UTC, by rolling back the platform update.

Next Steps

• Create alerts on the client, server-side, and API logs when error rates increase to reduce time to detect an issue.
• Review and improve practices for progressive deployments, including but not limited to,  advanced testing, rollback automation, and advanced customer communication.

‌

We value your feedback and want to do the best we can for you. Please let us know how we did or your ideas for how we can improve our incident communication processes through the following short survey: https://forms.gle/DyYcm1otcZrt6k7s5