Introduction

Today, June 6, 2024, WellsFLIX was notified that users were unable to view the images within emails sourced from WellsFLIX. Unfortunately, this had impact to multiple users who utilize Google Gmail, Yahoo Mail, Hotmail, and others. This has been going on for quite some time. This would have impacted the weekly newsletters, as well as any email communication from the requests application.

I am very sorry for this inconvenience.

Background

WellsFLIX currently uses Cloudflare as the hosting DNS service for the WellsFLIX site and all immediate sub-domains. There are many perks to using Cloudflare such as DNS, Email DMARC Filtering, SSL/TLS security, Web Application Firewall (WAF), caching, Distributed Denial of Service (DDoS) protection, etc.

Roughly about 2 years ago, in an effort to reduce the amount of bot traffic, WellsFLIX implemented a Zone-level WAF rule for Known Bots. These known bots are updated and managed dynamically by Cloudflare to capture known bots, detect, and mitigate malicious requests across all traffic under the zone.

At some point since implementation (Guessing within the past year), image caching services for Google, Yahoo, and Microsoft were added to the list of known bots. This went un-noticed to WellsFLIX as I use my own self-hosted email service that does not include the caching like these other companies.

When the image caching services were placed on the known bots list, the Cloudflare WAF started blocking these bots with a managed challenge, and in turn, blocking images from being pulled from WellsFLIX servers onto the Email service clients.

Clients such as Gmail, and Yahoo Mail cache images that are included within the body of emails. So instead of the image just pulling directly from say https://stats.wellsflix.com/image/images/newsletter/newsletter-header.png, it will instead attempt to cache the image and show a URL such as: https://ci3.googleusercontent.com/meips/ADKq_NaGck1RNFqs8X9U4zPBaNrXLCsZTp7wBWmCFDg6HQRgHInOo5FsSPUSwV6kAFA7ZylQd3jMgtzBu44GZIWxvUBDoJFsttEqB2gU2xFvzY11pvLTxNaw9oH_d5mwNVWB=s0-d-e1-ft#https://stats.wellsflix.com/image/images/newsletter/newsletter-header.png

Google and Yahoo caches these images in an attempt to speed up the delivery to the end user. Cloudflare was including this caching service as a known bot.

Remediation and follow-up steps

This incident had a large impact to the user community and I am ashamed I did not catch this sooner.

Remedition: In the Zone-level WAF, a new rule was configured to allow picture caching for email services. I created multiple incoming requests, and if they match, action will be taken to skip WAF filtering.

I added incoming requests for the URI Path, that contains .png, .jpg, .svg, and .ico to be skipped from the WAF. This is applied across all of WellsFLIX sub-domains.

Follow-Up: WellsFLIX has reviewed all custom WAF rules and will continue to monitor the event logs to ensure they are capturing and only filtering malicious requests.

Conclusion

Although WellsFLIX has invested significantly in ensuring services are dependable, but also are safe from malicious traffic, this shows an opportunity on where I can do better. I clearly fell short of the expectations. I have already started working on the changes outlined above and will continue our diligence to ensure this cannot happen again.