We’d like to provide a detailed review of the cause of this incident. It was caused by an unstable connection and limited failover between Chameleon’s CDN nodes and their backing Redis database.

To resolve the issue Chameleon engineers temporarily redirected traffic to a different CDN node, and rebuild the CDN nodes that were causing issue. The redirection took approximately 60 seconds and the rebuild completed in around 45 minutes.

To prevent future issues, the Chameleon engineering team has added specific alerting for this issue and new automation to redirect traffic in the event of issues in individual CDN notes. Prior to this incident, our engineers had started migrating away from our current CDN provider to a new (more robust) service not dependent on a real-time database. This work is continuing, and when completed (in the coming weeks), it will remove the risk of this issue recurring again. Additionally this upcoming change will also have the benefit of being more performant and cacheable for end-users.

Once again, we would like to thank any Chameleon customers for their patience and understanding as we resolved this issue.