Editor and User Experience Outage

Incident Report for Chameleon

Resolved

This incident is resolved and no related issues are present at this time. We will gather any remaining information and publish a postmortem by end of day tomorrow.

Posted Jul 15, 2020 - 21:53 PDT

Monitoring

The Chameleon Engineering team has deployed an update that switches Chameleon's caching services to another provider during issues such as this (a failover from Compose's services). Customers should see a return to service of Chameleon's Editor and Experiences (Tours, Surveys, Launchers and Tooltips).

We are monitoring this update to ensure it provides the same level of service as our primary provider.

Posted Jul 15, 2020 - 12:12 PDT

Identified

Our team has confirmed that the cause of this incident is an outage originating from our service provider Compose (https://status.compose.com/incidents/5qgqnq95wvc8).

We have implemented a failover to another service provider, and we are actively testing the services relying on this failover to ensure resiliency.

Posted Jul 15, 2020 - 11:03 PDT

Investigating

Editor and end-user Experience requests to Chameleon servers are being rejected due to an issue with Chameleon's caching layer.

The issue may be due to an outage of a Chameleon service provider Compose. Track Compose's ongoing outage here: https://status.compose.com/incidents/5qgqnq95wvc8

Chameleon's engineering team is actively investigating the issue, and working on sourcing another service provider.

We will make updates here regarding the continued investigation and a resolution.

Posted Jul 15, 2020 - 09:05 PDT

This incident affected: User profile API, JavaScript CDN, and Sidebar API.