Advanced Unix

1. Advanced Unix SMTP & Sendmail

2. Overview • Introduction to Email • Message Breakdown • Sample Messages • Extensions (MIME) • MTA’s and Mailbox Protocols

3. Email Statistics • 31 billion emails are sent daily • Email generates about one billion Gigabytes of new “information” per year • Spam accounts for about 40% of all email traffic http://www.spamfilterreview.com

4. Spam Statistics • The statistics that follow are current as of EOY 2006, but Spam statistics are quickly outdated. • These statistics were derived from a number of different reputable sources including: Google, Brightmail, Jupiter Research, eMarketer, Gartner, MailShell, Harris Interactive, and Ferris Research.

5. Spam Statistics • Email considered Spam - 40% of all email • Daily Spam emails sent - 12.4 billion • Daily Spam received per person - 6 • Annual Spam received per person - 2,200 • Spam cost to all non-corp Internet users - $255 million • Spam cost to all U.S. Corporations in 2002 - $8.9 billion • States with Anti-Spam Laws - 26 • Email address changes due to Spam - 16% • Estimated Spam increase by 2007 - 63% • Annual Spam in 1,000 employee company - 2.1 million • Users who reply to Spam email - 28% • Users who purchased from Spam email - 8% • Corporate email that is considered Spam - 20% • Wasted corporate time per Spam email – 4 to 5 seconds

6. Email Statistics • Daily emails sent - 31 billion • Daily emails sent per email address - 56 • Daily emails sent per person - 174 • Daily emails sent per coporate user - 34 • Daily emails received per person - 10 • Email addresses per person - 3.1 average • Cost to all Internet users - $255 million

7. SMTP • Originated in 1982 (rfc0821, Jon Postel) • Goal: To transfer mail reliably and efficiently

8. SMTP • SMTP clients and servers have two main components • User Agents – Prepares the message, encloses it in an envelope. (Eudora for example) • Mail Transfer Agent(MTA) – Transfers the mail across the internet User Agent Mail Transfer Agents

9. SMTP • SMTP also allows the use of Relays allowing other MTAs to relay the mail • Mail Gateways are used to relay mail prepared by a protocol other then SMTP and convert it to SMTP

10. What is Mail? • Mail is a text file • Envelope – • sender address • receiver address • other information • Message – • Mail Header – defines the sender, the receiver, the subject of the message, and some other information • Mail Body – Contains the actual information in the message

11. Post Office Mailbox Return-Path: <Jwatson@cis.udel.edu>Delivered-To: jwatson@cis.udel.edu Received: by mail.eecis.udel.edu (Postfix, from userid 62) id 17FBD328DE; Wed, 5 Nov 2003 11:27:02 Received: from mail.acad.ece.udel.edu (devil-rays.acad.ece.udel.edu [128.4.60.10]) by mail.eecis.udel.edu (Postfix) with ESMTP id 5F41832893 for <Jwatson@cis.udel.edu>; Wed, 5 Nov 2003 11:27:01 Received: by mail.acad.ece.udel.edu (Postfix, from userid 62)id 47509456C; Wed, 5 Nov 2003 11:27:01 Received: from stimpy.eecis.udel.edu(stimpy.eecis.udel.edu [128.4.40.17])by mail.acad.ece.udel.edu (Postfix) with SMTP id 7C2943D79 for <Jwatson@cis.udel.edu>; Wed, 5 Nov 2003 11:26:34 Message-Id: <20031105162634.7C2943D79@mail.acad.ece.udel.edu>Date: Wed, 5 Nov 2003 11:26:34 From: Jwatson@cis.udel.eduTo: undisclosed-recipients: ;MIME-Version: 1.0This is a test. Post office and mail route Receivers Mailbox

12. How SMTP works • The Essentials • How about a Demo?

13. How SMTP works • The Extras

14. Status Codes • The Server responds with a 3 digit code that may be followed by text info • 2## - Success • 3## - Command can be accepted with more information • 4## - Command was rejected, but error condition is temporary • 5## - Command rejected, Bad User!

15. Connection Establishment TCP Connection Establishment

16. Message Progress

17. Connection Termination TCP Connection Termination

18. Problems with SMTP • No security • Authentication • Encryption • Only uses NVT (Network Virtual Terminal) 7-bit ASCII format

19. E-mails can be forged….. HELO mail.rose.edu MAIL FROM: carberry@rose.edu RCPT TO: wrichards@rose.edu DATA From: Dr. Art Zenner To: Professor Richards Subject: CIT 2243 Professor Richards, By department decree all students in your CIT 2243 Introduction to Unix class are hereby to be given automatic A’s. Thank you, Dr. Art Zenner . QUIT

20. Extensions to SMTP • MIME – Multipurpose Internet Mail Extensions • Transforms non-ASCII data to NVT (Network Virtual Terminal) ASCII data • Text • Application • Image • Audio • Video

21. MIME Headers • Goes between the Email Header and Body • MIME-Version: 1.1 • Content-Type • Content-Transfer-Encoding • Content-Id • Content-Description

22. MIME Headers • Content-Type – Type of data used in the body of the message Text – plain, unformatted text; HTML Multipart – Body contains multiple independent parts Message – The body is whole mail message, part of a message, or a pointer to a message

23. MIME Headers Image – The message is a stationary image (JPEG or GIF) Video – The message is an animation (Mpeg) Audio – The message is 8 kHz standard audio data Application – The message is a type of data not previously defined

24. MIME Headers • Content-Transfer-Encoding – The method used to encode the messages 7 bit – no encoding needed 8 bit – Non-ASCII, short lines Binary – Non-ASCII, unlimited length lines Base64 – 6 bit blocks encoded into 8-bit ASCII Quoted-printable – send non-ASCII characters as 3 ASCII characters, =##, ## is the hex representation of the byte

25. Base64 Encoding • Divides binary data into 24 bit blocks • Each block is then divided into 6 bit chunks • Each 6-bit section is interpreted as one character incurs a 25% overhead 11001100 10000001 00111001 110011 001000 000100 111001 (51) (8) (4) (57) (z) (I) (E) (5) 01111010 01001001 01000101 00110101

26. Quoted-Printable Encoding • Used when the data has a small non-ASCII portion • Non-ASCII characters are sent as 3 characters • First is ‘=‘, second and third are the hex representation of the byte 01001100 10011101 00111001 (=) (9) (D) 00111101 00111001 01000100

27. MIME Headers • Content-Id – Uniquely identifies the whole message in a multiple message environment • Content-Description – defines whether the body is image, audio, or video

28. A Multipart, Encoded MIME Message From: joe_luthier@plucknplay.comTo: lchae@mfi.comSubject: Info on Gibson guitarMIME-Version: 1.0Content-Type: multipart/mixed; boundary=17 - 17Content-Type: text/enriched; charset="us-ascii"Content-Transfer-Encoding: 8bitContent-Description: Greetings As promised, I'm getting back to you about the Gibson Southern Jumbo guitar you were Interested in. I've enclosed a spec sheet on the guitar, which is in Microsoft Word. I guarantee that you'll love it! - 17Content-Type: application/octet-streamContent-Transfer-Encoding: base64Content-Description: Spec sheet saved as MS Word file - 17 -

29. MIME Example Date: Wed, 04 Apr 2001 00:11:37 -0400 From: Meghna Naik <mnaik@UDel.Edu> MIME-Version: 1.0 To: stoweg@hotmail.com Subject: =?gb2312?B?1tDOxA==?= title Content-Type: text/plain; charset=gb2312 Content-Transfer-Encoding: 7bit a body text, blah, blah

30. Mail Transfer Agents (MTA) • MTAs do the actual mail transfers • MTAs are not meant to be directly accessed by users. • Other MTA’s are: • Postfix • Qmail • MS Exchange • CC:Mail • Lotus Notes • ….etc.

31. O’Reilly’s “Bat Book”3rd Edition

32. This is what happens if you try to learn to much about Sendmail

33. Sendmail It's been said that you aren't a real Unix system administrator until you've edited a sendmail.cf file. It's also been said that you're crazy if you've done so twice.

34. What is Sendmail? • Definition: Sendmail is the most widely used Mail Transport Agent(MTA) on the internet • MTAs send mail from one machine to another. • Sendmail is not a client program, which you use to read your email. • Sendmail is one of the behind-the-scenes programs which move email over the Internet. • Normally it runs as a background daemon • Can even be run out of the super daemon (xinetd)

35. Implementations • SMTP Gateway • An SMTP gateway allows users on your network to communicate with others on the Internet without concern as to which local mail software package exists on your network. • All incoming mail for your network will pass through this gateway which converts the message into the appropriate format specific to your local mail software. • Similarly, all mail destined for the Internet from your network will pass through this gateway to be sent across the Internet via SMTP

36. Implemetations • SMTP Relay ”Warning Will Rogers” • An SMTP relay is a machine that actually sends the mail across the Internet. • A common misconception is that SMTP gateways are the same as SMTP relays. This is not always the case. • There are SMTP gateways that act as relays themselves, but there are also many that do not. If the latter is the case on your network, you'll need to bounce your mail off one of the relays.

37. Installation Methods • RPM installation • Obtained from installation CDs • Binaries (*.tgz) • Obtained from http://www.sendmail.org • Source Code • Obtained from http://www.sendmail.org

38. The Pieces • The binary: /sbin/sendmail • The configuration file: /etc/mail/sendmail.cf • Supporting files: /etc/mail/access /etc/mail/aliases …and many more

39. More Pieces • Email messages are stored in the directory: /var/spool/mail • There is a separate file for each user • Email waiting to be sent ./var/spool/mqueue • A log of Email sent and received: /var/log/mail

40. Sendmail Features • Sendmail uses DNS (Domain Naming System) • But not 100% dependent: [Joe@[192.168.1.1] • DNS provides Mail Exchange (MX) Info • Sendmail can do a DNS double-tap • Lookup up who the client says they are • Sendmail default is mail relay off • Realtime Blackhole Lists (RBL) • Mail Relay checkers - - Open Mail Relay Db http://www.ordb.org/submit/

41. Sendmail Anti-Spam Enhancements • Mailscanner • Minimal anti-spam • Anti-virus integration (scan in/outbound) • http://www.sng.ecs.soton.ac.uk/mailscanner/ • Or http://www.mailscanner.info • Spam Assassin • Rule based heuristic • Header and text analysis • Blacklist (RBL) • Vipul's Razor (http://razor.sf.net) • http://www.spamassassin.org

42. Mail Access Protocols • The MTAs place the email in the user’s mailbox • The Mail Access Protocols are used by the users to retrieve the email from the mailbox • POP3 • IMAP4

43. POP vs. IMAP POP3: All Messages Whole message IMAP: Mr Smith Friends …. headers

44. Post Office Protocol v3 • Simple • Allows the user to obtain a list of their Emails • Users can retrieve their emails • Users can either delete or keep the email on their system • Minimizes server resources

45. Internet Mail Access Protocol v4 • Has more features then POP3 • User can check the email header before downloading • Emails can be accessed from any location • Can search the email for a specific string of characters before downloading • User can download parts of an email • User can create, delete, or rename mailboxes on a server

46. References • RFC’s: • RFC821 - Simple Mail Transfer Protocol • RFC0822 - Standard for the Format of ARPA Internet Text Messages • RFC 1521 - MIME (Multipurpose Internet Mail Extensions) • E-mail Explained • http://www.sendmail.org/email-explained.html