Exchange deployment at CERN and new ideas for SPAM fighting

1. Exchange deployment at CERN and new ideas for SPAM fighting Michel Christaller, Emmanuel Ormancey, Alberto Pace

2. CERN Mail infrastructure • 14 Servers • 8 “Mailbox” stores, 2 Public Folder Stores, 2 Front-end servers, 2 Spare • IMAP (secure), POP (secure), MAPI and secure HTTP • MAPI with Outlook on Windows/Mac • MAPI open (in theory) outside CERN using Microsoft ISA Server • IMAP and POP work with almost any client • HTTP works with any Web browser • Collaborative tools available with MAPI and HTTP • Office XP recommended for collaborative features • Not possible to switch Outlook 2000 from IMO to CW • Allows multi protocol (pop, imap, mapi, webdav) • All information stored at server level, no more PST file problems • Office 2003 being evaluated • MAPI over HTTP • Seamless connected/disconnected/online/offline feature • Optimized for slow network connections

3. webdav http mapi imaps pops Additional interfaces available Legacy Server New Server Migration overview • Nothing changes for the user Mail User Mail Client imap user.mailbox.cern.ch Mail Server The server is replaced, Nothing changes for the client

4. Migration: what is done • User are invited to migrate by filling a migration form • The password is kept on the new service and synchronized with the windows password • Unresponsive users are forced to migrate and the password is reset • All folders and mails are copied from the old servers to Exchange • Mail Forwarding configuration is kept if any • Mailbox is not functional during at most 10 minutes, while rebuilding configuration files

5. Migration Workflow “Ask for migration” mail Click on link No answer Accept / Delay Form Click on link Reminder Mail (3) Accept Migration Form After n reminders Force migration Mailbox migrated Password reset Nice and Mail password synchronized Mailbox migrated Keep password typed in migration form Nice and Mail password synchronized

6. Migration Status • 10000 Exchange Users, 14774 Total • Only inactive and a few “non cooperative” users remaining • Cleanup: More than 700 Mail accounts deleted following user approval

7. Current status • 1 year of production • Exchange software stable and scalable • No major disaster, only normal hardware failures, solved in operational delays • Usage: 50 % Outlook XP, other 50 % with IMAP, POP and HTTP access • 1’000’000 Incoming mails per week, 30% is Spam

8. Next step, currently in test • Move SMTP Gateways to Exchange • Implement automatic anti flood system • Any server, sender or recipient sending or receiving more than 500 mails in 5 minutes will be banned (numbers to define) • Only solution to improve quality of service, and reduce impact of loops on “regular” mails • Migrate Mailing lists system from majordomo to Exchange • You will hear about this next year

9. Spam Fighting at Cern Evolution

10. Legacy system • Sendmail checks: • Lists of banned IP addresses, domains, subject, senders or recipients, and words • Header “consistency” tests (i.e. message id format) • Mail rejected if identified as Spam • Heavy manual work: • Update local banned lists from abuse reports • Remove entries when users report false positive rejections

11. Current service • Existing market products were reviewed: • Technology too young • Results are not accurate • Missing a per user basis configuration • While the market consolidates … • CERN developed his own Anti-Spam filter • Based on SpamAssassin • Less effort than running after immature commercial technology • Now in production for 1 year • Easy to modify and update detection techniques

12. How it works • The anti-spam filter calculates the probability for a message to be spam • Regular expressions • “Intelligent” content parsing • Statistical heuristics (Bayesian Filters) • The user sets the threshold at which he wants spam to be rejected • Rejected message can be seen by the user (CERN Spam folder) • Per user configuration (!) • Allows rejection of foreign languages mail(Chinese, Korean, Russian, Japanese, Arabic, etc …)

13. User configuration Filtering level Language-based rejection

14. Efficiency • Roughly 160 000 Incoming mails per day • Spam filter detects from 25% to 35% as spam

15. Efficiency • False positives are very low • Except for commercial lists (spam that you want) • White lists at user level can be configured to prevent this • Good spam detection • Statistics are hard to build • Standard mailbox filtering statistics: • 30 to 40 Spams filtered per day • 1 or 2 Spams still go to the INBOX per week • Could still be improved with some optimization • Not enough for some users with “public” email address • Old email address or published email address are more targeted for Spam

16. Current evolution • Spammer techniques always follow anti-spam techniques • New detection mechanisms work only for a few months • Needs a full time work to have a constantly “up-to-date” filter • Only viable long term solution is to accept only mails from people you know: • ICQ (and other messenger systems) already have this feature • Accept only messages from people in my contact list • Adding someone to the contact list requires validation

17. New feature (in test) • Good Mails not matching the user’s white list are quarantined • Mail is sent to sender requiring action to validate himself • Once validated, sender is added to white list, mails are moved back to Inbox Delete Delete if evident spam level Move to Cern Spam Spam Filter level Move to Inbox.Quarantine Mail to sender for validation Quarantine level Inbox

18. What’s next ? • Join forces against Spam • Share rules, regular expressions patterns and Bayesian statistics dictionary with other organizations • Central antispam configuration with Live Update like antivirus definitions is the solution. Therefore … • Long term goal: use a commercial product • Like for antivirus products, only a full time working team will provide up-to-date filters

19. In addition … • Within Exchange, mail is authenticated • Not possible to forge To: or From: fields • Delivery and Read receipts are reliable • A platform for workflow application • Extend this towards the internet • Mail messages digitally signed with guaranteed origin and dates • (See my presentation on PKI this Thursday)

20. Conclusion • Users are profiting from the new collaborative services • Shared calendar (already used by 1500 accounts) • Tasks, workflow • Web and webdav interfaces • Spam is a serious issue • Towards accepting only authenticated/verified mail • There is a future for commercial products in this area