400 likes | 537 Views
Peter Eicher, Product Manager peter_eicher@sybari.com. Agenda. Anti-Spam Challenges Typical Anti-spam solutions Content filter, heuristics, Bayesian RPD™ (Recurrent Pattern Detection) Patent Pending Technology Implementation and Management ASD Evaluation Mode. Two Unique Anti-Spam Issues.
E N D
Peter Eicher, Product Manager peter_eicher@sybari.com
Agenda • Anti-Spam Challenges • Typical Anti-spam solutions • Content filter, heuristics, Bayesian • RPD™ (Recurrent Pattern Detection)Patent Pending Technology • Implementation and Management • ASD Evaluation Mode
Two Unique Anti-Spam Issues • The growing number of spam attacks • Over 500,000 unique spam attacks detected in our service center each DAY • Compare to virus technology (1000 per month) • Need for a real-time solution with minimal IT involvement • For the first time with a security product, the user must be involved in the decision-making • Spam is not black and white • Need flexibility to fine-tune solution to client’s needs
The Problem • “Spam is a rapidly growing problem for all email users. The traffic is doubling every 4 months, as are the associated costs” • Today : 40-60% of all e-mail is Spam • Unique spam attacks have increased 200% in 2002 (Osterman Research) • A study shows that the annual cost of spam is : $8.9-billion for U.S. corporations (Forrester research) • Typical user receives 14,500 spam emails each year
Market Trend: The Volume of Spam Total spam messages/day (Billions) 2007 Spammers will continue to improve infiltration tactics… 2006 …and demand will grow for a real-time adaptive solution 2005 2004 2003 2004: Enterprises cannot afford staying unprotected 2003: Damages exceed $500 per-employee, annually Early 2002: Annoyance-only level Graph source: The Radicati Group, Inc. 2003
Typical Anti-spam Solutions • Most anti-spam solutions rely on a combination of content filtering, heuristic scanning and/or Bayesian filtering • These techniques have numerous flaws • Spam detection rarely higher than 70% without extensive administrator attention • False positives extremely high
Content Filtering • Useful as a content management tool • Prevent certain words/topics from being sent to or from your employees • However, both inefficient and unsuccessful for spam management • Requires continuous administrator attention (multiple hours per day) • Simple spelling tricks defeat content filtering • Examples: $ave, V*i*a*gr*a, Chëὰρ • There are 105 variations available just for the letter A! • Results in numerous false positives • Impossible to use in certain industries
Content Filtering • Think your administrators can keep up? Here’s a few ways to spell Viagra… V I @ G R A , V--1.@--G.R.a, \./iagra, Viiagra, V?agr?, V--i--a--g--r-a, V!agra, V1agra, VI.A.G.R.A, vi@gra, vIagr.a, via-gra, Via.gra, Vriagra, Viag*ra, vi-agra, Vi-ag.ra, v-iagra, Viagr-a, V^I^A^G^G^A, V'i'a'g'r'a', V*I*A,G,R.A, VI.A.G.R.A..., Viag\ra!, Vj@GRA, V-i:ag:ra, V'i'a'g'r'a, V/i;a:g:r:a, V i a g r @, V+i\a\g\r\a, Viag[ra, V?agra, V;I;A*G-R-A, V-i-a-g-r-a, V*I*A*G*R*A , V-i-@-g-r-a, VI@AGRA, Vi@gr@, \/^i^ag-ra, VlAGRA, V\i\a.g.r.a, V1@GRA, v_r_i_a_g_r_a, V\i\a:g:r:a, V^i^a^g^r^a, V-i-@-g-r-@, Viag(ra.
Heuristic Scanning • A “scoring” technique that looks at thousands of “characteristics” to determine spam and creates a score • Level of “spaminess” must be constantly adjusted • Used in many spam products • Well understood by spammers • Spammer websites allow “testing” of spam vs. heuristic scanners • Extremely performance intensive • Every detection is a new event that doesn’t benefit from previous detections • Very high false positive rate • A “best guess” solution
Bayesian Filtering • A learning system that uses statistical analysis of vocabulary • Lists of “good” and “bad” words • Requires active user participation to be effective • Can be very effective for individual user • Far less effective in an enterprise setting • One user’s choice can negate another’s • Deliberately attacked by spammers • “Invisible” random text lowers spam score by increasing count of “good” words • High rates of false positives
Five anti-spam challenges • Catching spam and spammer evolution • Need a high detection rate today • Solution must overcome tomorrow’s spammers • What defines “spam” for the end-user? • Unsolicited emails – considered spam by almost everyone • Solicited commercial email – may or may not be considered spam • ‘Opt-out’ and unsubscribing are often tricky and users have been trained to avoid this • Anti-spam should handle all of these situations • Reaching a near-zero false positive level without compromising the detection level
Five anti-spam challenges • Real-time updates & filtering • Blocking from the first minute of an attack • Remove the “window of vulnerability” created by scheduled filter updates • Improving anti-virus filtering • International efficacy • Languages, encoding methods & double-byte can cut the effectiveness of content-based detection to zero
Outsmarting Spam All messages in a spam outbreak have a repetitive component – the attack “pattern” … and Sybari ASD knows how to trace it!
First, some statistics • The ASD Service Center detects on average over 600,000 unique spam attacks per day • Based on statistics from 12/07/03 to 1/06/04 • High of 799,000 to low of 340,000 Actual new outbreaks per hour from 12/29/03
The ASD Spam Detection Engine • Located at the ASD Service Centers, monitoring over 15 million message signatures daily • Automatically detects the repetitive component of each spam outbreak • Uses Recurrent Pattern Detection technology, or RPDTM • Powered by Commtouch Software • Identifies the identical or approximate patterns appearing in spam • Statistical analysis determines spam • Spam “signatures” created based on detection
Recurrent Pattern Detection • Identical match and approximate match techniques detect spam attacks • Every spam attack has some element of similarity • Checks sender, subject line, body SPAM! SPAM! Classification system, statistical analysis Mail Signatures Valid mail
The ASD Spam Detection Engine • Based on message prevalence, mail is rated as “not spam,” “bulk mail” or “confirmed spam” • Bulk Mail and Confirmed Spam can be handled differently • Spam is “confirmed” by human monitors to ensure complete confidence in rejecting confirmed spam messages
RPDTM Benefits • 95%+ detection rate - detects solicited & unsolicited spam • No false positive mistakes due to “suspicious” content in legitimate person to person messages • Does not rely on specific words • Critical for industries that use many “spam” words – financial, real estate, medical, retail, marketing, etc. • Immune to constantly evolving spammer tactics • Relies on the one factor that remains consistent for all spam – it is sent in volume
RPDTM Benefits • The fastest spam detection technology: • Blocks spam from the first minutes of an outbreak • Real-time spam signature updates ensure the highest detection levels • Content-agnostic – detects spam in: • All languages • All encoding methods, and double-byte • All file formats
Good Bulk = Spam = = Service Center/Gateway Interaction Internet • Real time signature updates from Service Center Local Signature Cache Local detection first, remote detection as needed. Internet Local Match If unknown Service Center Tag, Junk Folder or Reject Inbox Signature Database – over six million sigs Classifier Recurrent Pattern Detection Data Center
ASD Implementation Sample deployment scenario • Installed on Windows 2000/2003 server • Installs on SMTP Gateway or Exchange server • Supports Exchange 5.5, 2000, 2003 • Uses SQL MSDE database • Directory integration allows controlled deployment • One user/group at a time
Policy Flow and Spam Management Options The ASD Gateway and Service Center
: Strong anti-spam-filtering capabilities, flexible deployment options; easy to set up and manage. ASD Gateway Administration • Centralized administrator control of system-wide block and accept rules • Spam can be rejected, quarantined or sent to user • Maintains database of individual user preferences for delegated control • Easy to use browser interface
Gateway Administration Lists blocked mail received from specific Domain or From field
Gateway Administration User Decision – all future mail from sender will be sent to user for decision Approve (white list) – all future mail from sender will be allowed Quarantine – all future mail from sender will be sent to site Quarantine based on group/rule settings Reject (black list) – all future mail from sender will be rejected and treated based on group/rule settings
Gateway Administration Spam is identified as Confirmed or Bulk • Three actions for confirmed or suspected spam • User Decision – send to Junk Mail folder • Site Quarantine – send to quarantine for administrator decision • Reject – reject message Because spam is fluid and attacks happen quickly, mail with “low” or “moderate” chance of being spam can be held until Service Center is re-polled.
Gateway Status report An overview of system status Total number of Block and Approve rules created by users and Admins Total number of Users and Users in Exception group Total number of spam messages in given time period, and percentage of emails considered spam
Gateway General Traffic Reporting An overview of system traffic Total messages, spam and non-spam, processed by policy or detection Number of messages approved and blocked
About the ASD Junk Mail folder • Users make their own spam decisions • Users can white-list desired messages or black-list unwanted messages with one click • No need to impose system wide blocks • Completely private and secure • Relieves admin from constant decision making • The Junk Mail folder is automatically created in the user’s Outlook client • Does not disrupt the user experience • Junk Folder is self-cleaning, based on administrator defined life cycle
What the User Sees… Policy Manager: allows user to review and change existing rules, write new rules Approve Sender: all further emails from this sender go to Inbox Block Sender: all further emails from this sender will be blocked at the Gateway
What the User Sees… • The Policy Manager allows end users to modify or create rules • Provides support for POP3 accounts (clients that are not MS Outlook)
Non-Junk Folder users • Users who don’t use or want a Junk Folder can have spam “tagged” with admin-defined prefix • For example, Outlook Express users or other POP3 clients • A second ASD user group is defined in the Directory Services to support users that do not want/need a Junk Folder • Created using a simple utility
About the Site Quarantine • Administrator can direct spam to a Quarantine folder rather than the Junk Mail folder • Spam and/or suspected spam can be sent to the Quarantine folder • Depends on administrator settings • Administrator takes actions on quarantined messages • Reject message • Approve: release to user’s inbox • User Decision: send to user’s Junk Mail folder
Quarantine Folder Approve sender – mail is delivered to end user Inbox Reject sender – mail is deleted User Decision – mail is delivered to user’s Junk Mail folder
ASD Evaluation Mode • Run ASD in “Spam Analyzer” mode • Detects spam without taking any actions • No Junk Folders created • No stamping of email • End users are unaffected/unaware • Administrators receive full report data on number of spam messages detected, spam domains, etc. • Understand ROI potential of ASD
Summary – Sybari Advanced Spam Defense (ASD) • Manages spam as a background service • Minimal IT maintenance • External Service Center scales to increasing volume • Global view of Internet traffic • Gives IT control over inbound e-mail • Integrates directly into e-mail system • Fine-tune sensitivity when needed • Enforcement of enterprise policies • Keeps responsibility in the hands of end users • Only they know the real definition of spam for them • Reduces false positives and non-delivery complaints • Preserves confidentiality and security