Google incorrectly reporting "malware" on Intercom links (May 17th)
Incident Report for Chameleon
Postmortem

Late last week, Google flagged email messages sent by Chameleon as malicious or containing links to malicious sites.

This email is to notify you that the flag has been removed, and no Chameleon messages contained malicious content or links to sites containing malicious content.

The issue was due to Google identifying the domain of our email provider (Intercom) as malicious. This was caused by a series of attachments from other Intercom customers being flagged by Google as "dangerous" and in turn causing our emails to be marked the same way.

We take security seriously and launched our own investigation into the issue and worked with Intercom to gain more information and a quick resolution. We will be conducting a fresh vulnerability test, audited by a third-party, to further validate and strengthen our security practices.

We are also reviewing and re-evaluating our email delivery system so that we reduce the risk of this happening again.

We apologize for the alerts and want to re-assure you that Chameleon emails are safe and contain safe (and hopefully valuable) links.

If you have any concerns, please reply and we will do our best to address them.

Posted May 22, 2019 - 18:21 PDT

Resolved
With reference to the underlying Intercom => Google issue https://www.intercomstatus.com/incidents/7k8sq34rr2c4 we consider this closed.

The root cause was a series of attachments on Intercom-based email being flagged by Google as "dangerous" and in turn causing other emails to also be marked the same way.

Thanks for all the reports. Your diligence reflects how much we all care about the security of our platforms.

Brian
Posted May 20, 2019 - 08:22 PDT
Monitoring
Intercom has officially acknowledged this issue: "references to urls containing "intercom-mail.com" are appearing with a banner in Gmail indicating "This message seems dangerous". The root cause of this to be related to customer files sent via Intercom being flagged as malicious by Google."

Intercom has now removed references to these files and is monitoring for resolution. While we await confirmation of a resolution, Chameleon will not be sending any further emails from Intercom to avoid further Google alerts.

The issue can also be monitored on Intercom's status page: https://www.intercomstatus.com/incidents/7k8sq34rr2c4
Posted May 18, 2019 - 20:24 PDT
Update
We are no longer sending any emails from our service provider (Intercom) and are awaiting a response from them. From our internal testing, we believe the issue has been resolved but are awaiting formal acknowledgment and confirmation of resolution.
Posted May 17, 2019 - 17:05 PDT
Update
We have seen reports that *multiple Intercom customers* have been affected by this issue.

Intercom support has claimed it to be a bug in their system and are investigating
Posted May 17, 2019 - 13:31 PDT
Update
Generally speaking, when sending emails, Intercom replaces links (i.e https://www.trychameleon.com/blog/live-chat-user-onboarding) with links to https://chameleon.intercom-mail.com/via/ob=.... This is done so that Intercom can show "link click analytics" in their dashboard. We know now that Intercom is NOT CORRECTLY REDIRECTING links to their original target. In addition, old links that "previously linked correctly" (in past emails) have changed and are now redirecting with a 302 status code to to top-level domain https://app.intercom.io.

We suspect that Google is viewing this "change" as potentially malicious
Posted May 17, 2019 - 11:32 PDT
Investigating
We have been alerted to an issue arising from a marketing email sent via Intercom being flagged as containing malware. Specifically, Google is flagging and all links to the domain chameleon.intercom-mail.com.

Please be assured that the Chameleon website or the Chameleon application has not been hacked nor does it contain malware. We take security very seriously and you can learn more about our practices here => https://www.trychameleon.com/security.

We believe the issue is stemming from Intercom and we're working closely with them to investigate and remedy the issue.
Posted May 17, 2019 - 05:45 PDT