How a major ISP built a new anti-abuse platform

1. How a major ISP built a new anti-abuse platform Mike O’Reirdan Comcast Distinguished Engineer Internet Systems Engineering Comcast National Engineering & Technical Operations

2. Outline • Comcast facts and figures • Why build a new platform • Fundamentals of anti spam • Size of the problem • Previous approach • Current solution • Migration methods • Current status

3. Why a new platform? • Moved from a hosted to an in-house platform • Need to improve customer experience by further reducing volumes of spam to the mailbox • Deploy a platform which can economically and easily scale • Emerging threats in abuse landscape • Image spam • Botnets • VoIP spam (SPIT) • Need to have a plug-and-play architecture • Firmly believe that no one vendor will be the best forever • We need a mix of vendors and approaches to hedge our bets and reduce risk • Somebody in this room may be our next vendor when you have gone from the lab to the VC and into beta 

4. Size of the problem • Volumes of spam are astronomical • 596 million connection attempts (Jan25th 2008) • 539 million connection attempts rejected • 93% spam • 76 million messages delivered • Connection attempts increases massively above this around holidays such as Thanksgiving. • The problems is criminality at massive scale

5. Fundamentals of anti-spam • Not much differentiation between major mail box hosters and other ISPs with regard to spam percentages and volumes • Three stages • Blocking based on IP (reputation and DUL space) • 5% of CPU cycles • Removes ~70% of the spam • Blocking based on message protocol and heuristics • 10% of CPU cycles • Removes ~15% of the spam • Blocking based on content • 85% of CPU cycles • Remove ~10% of the spam • Idea is to use the least cycles to remove the most messages

6. Previous approach • 100s of Linux blade servers • No site fail over • Multiple RBLs using BIND for DNS • Heuristics and protocol filtering • Spam content filtering using industry standard software • Virus filtering using industry standard software

7. New Approach • Fewer Linux Blade servers distributed over two sites • Full dual site redundancy with each site fully capable of carrying 100% of traffic • RBLs hosted on a specialised DNS based platform • Trend • Spamhaus • Return Path • Protocol and heuristics filtering performed on the Bizanga IMP MTAs which run on Linux • Spam content filtering technology • Anti-virus technology

8. Heuristics employed • Directory Harvest attack • Dictionary attack • rDNS check • Throttling • Dynamic space blocking • Non-existent user block

9. Content filtering-detecting spammy content • Cloudmark • Relies on multiple sources of data • Spam / no Spam reports from end users • Honeypots • Initially based on Vipul’s Razor • Applies algorithmically derived signatures to incoming email (Proprietary) • Zero hour anti virus • Trend Anti-virus • Signature analysis • Heuristics

10. Migration • Relatively simple process to migrate from old platform • Moved traffic across by re-pointing comcast.net MX records to new platform and making lots of involved highly planned DNS configuration changes • Performed a series of increasing short duration burst test scale • Then moved 5% of the traffic. After platform rules proved stable, traffic was moved across in slightly larger increments over several days to the new platform. • This method allowed us to quickly revert back (under 30 minutes) to old platform in the event of any issues without customer impact

11. Lessons learned • It always helps to be able to test the new platform against an existing live e-mail flow but this is difficult at our scale with a multi-Gbps mail flow • Failing that, heavy reliance has to be placed on cooperation with vendors and existing platform technology users • Rules used on an old platform do not always map across neatly to a new one