The root cause of this incident was a database failover into a node that was performing a background index build -- the index on the previous master had completed in 'off hours' with not noticeable impact to query performance however when the index build had to keep up with peak load it could not quite cover the load. The result was a throttling effect on each connection causing requests that typically take 35ms to take around 5s-12s with some more intensive requests taking longer. After identifying the issue our engineers were able to shed load into a properly throttled background queue to drop the overall load.
End-users may have seen console errors relating to failed network requests or simply a slowdown in network activity to Chameleon.
Posted Oct 06, 2020 - 21:24 PDT
Monitoring
This incident has been resolved as of approximately 12 minutes ago -- we will post a more detailed final update when we have finished actively monitoring the issue
Posted Oct 06, 2020 - 14:21 PDT
Investigating
We're currently investigating an issue that causes Chameleon Server requests to fail due to a database slowdown. This can affect the creation to Chameleon Experiences, including Surveys and Tours, and possibly Tours not appearing to end-users.
Customers who are having difficulty accessing the Builder should contact Chameleon at Support@trychameleon.com, or submit a ticket through their Chameleon Dashboard.
Posted Oct 06, 2020 - 13:28 PDT
This incident affected: User profile API and JavaScript CDN.