1 / 19

Spam Reduction Techniques

Using greylisting and SpamAssassin. Spam Reduction Techniques. The problem. The vast majority of email today is Spam Some current statistics indicate over 90% of email Spam This matches my experience. Botnets. Vast majority of Spam comes from Botnets compromised home PCs

bat
Download Presentation

Spam Reduction Techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using greylisting and SpamAssassin Spam Reduction Techniques

  2. The problem • The vast majority of email today is Spam • Some current statistics indicate over 90% of email Spam • This matches my experience

  3. Botnets • Vast majority of Spam comes from Botnets • compromised home PCs • hundreds of thousands to millions, or even tens of millions of machines in a heard • Controlled by the owner of the heard via a centralised command and control structure • Typically don't have a “real” smtp server to actually send the email

  4. Spam Reduction with Greylisting and SpamAssassin • Currently > 99% effective (closer to 99.8%)‏ • In a recent week, only 11 out of 8,000 Spam messages made it through to the end user without being stopped or marked.

  5. Spam statistics as of: 16/09/2007 Total spam: 5459 Total greylisted: 4457(90.8%)‏ Total emails accepted (both spam and legitimate): 451 (9.2)% Total identified spam through to end users: 1002 (20.4%)‏ Emails greylist_delayed: 58 (1.2%), marked as spam 57 (96.6%), NOT marked as spam 2 (3.4%)‏ emails via backup mx: 991 (20.2%), marked as spam 944 (95.2%), NOT marked as spam 48 (4.8%)‏ Effectiveness of Greylisting / SpamAssassin: 99.0%. 50 out of 4908 not marked as spam Spam statistics as of: 23/09/2007 Total spam: 5167 Total greylisted: 4928(90.8%)‏ Total emails accepted (both spam and legitimate): 499 (9.2)% Total identified spam through to end users: 239 (4.4%)‏ Emails greylist_delayed: 99 (1.8%), marked as spam 98 (97.0%), NOT marked as spam 3 (3.0%)‏ emails via backup mx: 151 (2.8%), marked as spam 138 (90.2%), NOT marked as spam 15 (9.8%)‏ Effectiveness of Greylisting / SpamAssassin: 99.7%. 18 out of 5427 not marked as spam Spam statistics as of: 30/09/2007 Total spam: 6216 Total greylisted: 5950(91.2%)‏ Total emails accepted (both spam and legitimate): 573 (8.8)% Total identified spam through to end users: 266 (4.1%)‏ Emails greylist_delayed: 141 (2.2%), marked as spam 135 (95.1%), NOT marked as spam 7 (4.9%)‏ emails via backup mx: 151 (2.3%), marked as spam 128 (84.2%), NOT marked as spam 24 (15.8%)‏ Effectiveness of Greylisting / SpamAssassin: 99.5%. 31 out of 6523 not marked as spam Spam statistics as of: 07/10/2007 Total spam: 7901 Total greylisted: 7712(93.0%)‏ Total emails accepted (both spam and legitimate): 581 (7.0)% Total identified spam through to end users: 189 (2.3%)‏ Emails greylist_delayed: 135 (1.6%), marked as spam 134 (97.8%), NOT marked as spam 3 (2.2%)‏ emails via backup mx: 62 (0.7%), marked as spam 55 (87.3%), NOT marked as spam 8 (12.7%)‏ Effectiveness of Greylisting / SpamAssassin: 99.8%. 11 out of 7901 not marked as spam • Greylisting removes > 90% of incomming Spam • SpamAssassin catches > 90% of received spam • Total effectiveness > 99.5%

  6. Greylisting • Relies on Spammers not using a “proper” mail server. They just fire-and-forget • Give a temporary failure to any “suspect” messages. Spammers will not retry, but a mail server will

  7. Which messages to challenge • Look at (all of): • From address • To Address • IP of sending machine • If not seen before: • give temporary failure • record this “tuple” + time

  8. If seen before: • check if it is now past a “start time” (time + time to go live)‏ • time to live is typically a parameter passed to greylisting server. • many recommend 60 minutes • I use 60 seconds • OK – let through • record the time • Not OK • reject again • Any subsequent communication is let straight through

  9. Potential issues • Some delay first time someone new contacts you • Small chance of non delivery of some messages. • non compliant mail servers • ISPs with rotary pool of mail servers may get continually greylisted • email from web forms that doesn't go through a real mail server

  10. Risk minimisation • Can have various white lists • add mail server details for all regular / potential contacts to a white list • these emails are coming from a real mail server, so we don't need to use this test on them. • grep you mail server logs to determine who does conatct you. eg:egrep "client=.*mail.*|client=.*mx.*|client=.*smtp.*" /var/log/maillog*| awk '{print $7}' | awk -F = '{print $2}' | awk -F [ '{print $1}' | sort | uniq -u • can use regex in these whitelists

  11. Examples of server whitelist /^.*\.ebay\.com$/ /.*\.emailebay\.com$/ /^.*\.mx\.bigpond\.com$/ /^.*\.dell\.com\.au$/ /^.*\.mailguard\.com\.au$/ /^mailout.*\.pacific\.net\.au$/ /^mail-out.*\.netspace\.net\.au$/ /^mx.*\.phx\.paypal\.com$/ /^smtp.*\.bis\.ap\.blackberry\.com$/ /^.*\.server-mail\.com$/ /^vscan.*\.westnet\.com\.au$/ /^ihug-mail\.icp-qv1-irony?\.iinet\.net\.au$/

  12. Implementations • Available for many popular mail servers including MS Exchange

  13. SpamAssassin • Categorises email as either Spam or Ham (good stuff, not Spam), based on a number of tests • Each test may add to the overall score for this email • If the total score exceeds a (configurable) limit, it is marked as Spam • Highly configurable • personal limits, tests, scoring etc

  14. Tests • Tests to find words that look like viagra etc • Is the sender in a RBL • Does the sender match the SPF record • v=spf1 a mx mx:westnet.com.au include:westnet.com.au ~all • Does the body look like spam • The ratio of text to images • Bayesian analysis of the content • Many more tests • see: http://spamassassin.apache.org/tests_3_2_x.html for the full list

  15. Spam / Ham folders • can also set up folders containing Spam and Ham (non Spam) for SpamAssassin to learn from. As a large proportion of email is actually spam (if you are not using greylisting), doing this may not be a good idea, as eventually the Bayesian filter gets poisoned and everything ends up looking like spam.

  16. Implementations • Available for many popular mail servers including MS Exchange • Exchange implementations tend to be commercial offerings

  17. SMTP Conversation

  18. Greet - Pause • When the sender connects, delay the greeting • If the sender tries to continue the conversation, before the appropriate response, the conversation is stopped by the smtp server. • A “proper” smtp server will handle this, a Spam bot may just have a sequential script and fail this test. • About 10% of Spam can be eliminated this way

  19. Components (in my system)‏ • Postfix mta (postfix-2.3.3-2)http://www.postfix.org • postgrey greylisting server (v 1.30)http://postgrey.schweikert.ch/ • See also http://www.greylisting.org/ • SpamAssassin (spamassassin-3.2.2-1.el5.rf)http://spamassasin.apache.org/

More Related