Electronic Mail

1. Electronic Mail Chapter 22

2. Understand the basic steps in the mail delivery system. • Understand the role of the Mail User Agent (MUA) • Understand the role of the Mail Transport Agent (MTA) • Understand the basic strategies for handling email. • Understand the basic protocols used to deliver and transport mail. • Understand the basics of email security • Understand the basics of sendmail configuration and rulesets. • Parsing • Anti-spam • Anti-relay • Authentication Chapter Goals

3. Email Overview • Email is a “vital” service in users eyes. • Lost mail is not acceptable • But it happens daily! • Email is assumed to be confidential • It is not confidential! • Email delivery delays are not tolerated • Email is an unreliable service! • This leaves system administrators with huge problems. • How to ensure reliable service • How to secure the MTA/SMTP service • How to monitor and manage email services • How to secure MUA services Email

4. Email Overview • Typical user complaints/questions: • How do I send mail to my friend? I don’t know their address, but I need to send them mail. • I sent person X mail, why don’t they respond? • Why do I get all of this spam? • Why can’t you stop all of this spam? • Is OIT reading my mail? • Where did my mail go? It was there a minute ago! Email

5. Mail Overview (at the sending end) • A user creates a mail message using a mail user agent (MUA) (pine, mh, elm, netscape, eudora, mailx). • When done “creating”, they exit the MUA. • The MUA “contacts” a local mail delivery agent (MDA). • The MDA spools the message in the mail spool area and signals a program to process it. • The program that processes the message is called a Mail Transport Agent (MTA). The MTA implements the Simple Mail Transfer Protocol (SMTP). • The MTA parses the “To:” address, and attempts to contact the remote host’s MTA. • If contact succeeds, the local MTA transfers the mail to the remote end. • If contact fails, the local MTA retries for some finite period of time. Email

6. Mail Overview (At the destination end) • The destination MTA “receives” the message as per the SMTP protocol • sending end introduces itself • sending end tells who mail is from • sending end tells who mail is for • If destination user is valid, open a spool, and continue collection. More on this in a minute… • Sending end transfers data • Sending end closes connection Email

7. Mail Overview (”deliver it”) • The destination MTA checks the “To:” address to see if there is a system wide alias for the user. • If alias exists, deliver as per alias. • If no alias, check to see if account exists. • If no account, reject message. • If account exists, check to see if the user has a .forward file. • If .forward exists, deliver mail as per .forward • If no .forward, deliver to local spool. • When recipient invokes their MUA, it checks local spool, and informs user there is a message. Email

8. Mail Aliases • Unix: /etc/mail/aliases – a text file containing a list of recipients, and names of users to deliver mail to. root: curt, mmcnally postmaster: root epadmin: curt cfh: curt erp: curt, mark@co.org, jkeane@nd.edu mentor: suggest, henry tou: \tou@cse.nd.edu scott: \scott@cse.nd.edu Email

9. Mail Strategy • To implement the delivery of e-mail at a site, the administrator has to make a few decisions about how mail will be handled at the site. • There are two primary e-mail models in use: • the “every host” (mail host) model and • the “smart-hub/thin-client” model. Email

10. Mail Strategy • “Every Host” Mail host • In this model, every machine on the network is capable of sending and receiving e-mail. Although this requires the least setup, it also causes problems. • The administrator should add the mail spool to the backup schedule to ensure that a user’s messages are not lost. • The individual machines are all running the SMTP daemon, and could be used as open relays. • The fact that e-mail does not pass to a single host means that router filters and spam/virus filtering are more difficult to implement. • Troubleshooting mail problems is difficult, as every machine is (potentially) configured differently. Email

11. Mail Strategy • “Every Host” Mail host • Every time a new version of the software is released, the administrator has to update every host. • The good side of this model is that the configuration is pretty minimal. • Clients read mail from their own local disk, and therefore the mail file system does not have to be made available to other hosts. • The administrator may have to develop a standard delivery configuration file, and distribute it to all machines. Email

12. Mail Strategy • Smart Hub/Thin Client • This model of e-mail management requires a large central e-mail server. • The central server is the only machine that will accept mail messages for the entire enterprise. • This server is configured with plenty of disk space and connectivity to allow it to keep up with the load of all e-mail into and out of the enterprise. • If the enterprise decides to implement this model, the name service may also require some reconfiguration, to add MX records for all hosts in the enterprise. • The MX record would tell other hosts to send their mail messages to the smart hub, instead of to individual hosts within the enterprise. Email

13. Mail Strategy • This model requires much more configuration at the beginning, but it also brings a certain amount of sanity to the e-mail process. • For example, the enterprise router filters may be tuned such that they only allow SMTP connections to the mail hub (mail server). • All other SMTP connection requests can be denied. • Anti spam/virus filtering can be installed on the mail server to ensure that all messages are checked for harmful content. • The administrator only has to back up one mail host, instead of backing up the mail spool from every host at the corporation. • If an upgraded version of the mail software is released, the operator has to update only one mail server, instead of hundreds. Email

14. Mail Strategy • Mail client machines can also be greatly simplified using the mail hub model. • For example, you might determine that it is not necessary to run the SMTP daemon on client machines. • A simple cron job to check the queue periodically and process pending requests by sending them to the mail server for delivery may be all the client support your site requires. • For slightly more complicated scenarios, you may still need to build an SMTP daemon configuration file to control this process, and/or find a way to make the mail spool (on the server) available to the MUAs on mail clients. Email

15. Mail Strategy • But the news is not all good, as this model also has its downside. • The mail server is a great repository of mail messages, but it also has to make these messages available to users. • Although you could force the user to log in to the mail server to read mail, this is rarely acceptable. • Another problem with the mail hub model is user education. • Users like to personalize their mail. • Many users prefer to think that by having mail delivered to their desktop, it is more secure. • Some users want correspondents to think the mail goes directly to their desktop, instead of to a corporate mail server. • Quite often, the administrator has to convince the user to advertise the e-mail address on the corporate server, rather than the address on the user’s desktop. Email

16. Mail Strategy • Mail Servers often require substantial hardware to implement this mail management model. • A 20,000-employee operation could easily swamp a four-processor mail hub with 300 gigabytes of disk space reserved for the mail spool area. • If that single mail server ever experiences a problem that takes it off-line for an extended period of time, the users will be on the warpath! Email

17. Email Protocols • The heart of e-mail is the collection of protocols used to transport e-mail from server to server and make it available to users. • This collection of standard protocols allows the wide range of e-mail software to interoperate across the Internet. • These protocols can be split into three categories: • those used by MTAs as mail is moved between servers, • those used by MDAs to deliver the mail, and • those used by MUAs to allow the user to read, create, and reply to mail. Email

18. Email Protocols • MUAs typically allow the user several methods of accessing mail, depending on how and where messages are stored. • The three most common access methods are • plain files, • the Post Office Protocol (POP), and • the Internet Mail Access Protocol (IMAP). • These protocols, as well as the SMTP protocol used by MTAs and associated service daemons, have their own reserved ports Email

19. Mail User Agents (MUA’s) • Some MUA’s “read” mail directly from the spool (/var/mail, /var/spool/mail). (mailx) • Mail spool must be mounted on machine where user reads mail. • Some MUA’s move messages to an Inbox and operate on it there (pine, netscape). • Mail spool must be accessible, but not necessarily mounted on machine where user reads mail. Email

20. Mail User Agents (MUA’s) • Some MUA’s use delivery protocols to get mail to user • Post Office Protocol (POP) • Network based – cleartext passwords – copies mail to client, can remove from server, or leave copy on server (pointer into spool tells what has been read) • Internet Mail Access Protocol (IMAP) • Network based – cleartext passwords – uses RPC’s to act on messages. Displays message on client, does not copy to local disk. Email

21. MUA’s • Have their own transport languages. • Typical commands: • User • Pass • Get • Put • Delete • Purge • Write (Save) Email

22. Mail transport agents (MTAs) • Mail transport agents (MTAs) are the workhorses of the e-mail system. • Several MTAs exist for a wide range of platforms. Some common choices are Sendmail, Microsoft’s Exchange Server (exchange), postfix, qmail, and exim, PMDF. • This variety of choices means that the administrator needs to make a decision as to which MTA will work the best for the site. Email

23. Mail transport agents (MTAs) • Many factors influence the MTA selection process, including the following. • Security is one of the primary factors in choosing an MTA. Like a web server, a mail server will need to accept input from a wide variety of sources, including malicious sources. • A mail server needs to be capable of handling a large volume of data. • A mail server should be capable of using encrypted connections. • A mail server should be capable of controlling access. • Sendmail is the most commonly used MTA. It is shipped as the default MTA on nearly every UNIX variant and is available for Windows. Email

24. Sendmail • Sendmail, Inc. distributes an open-source version of the sendmail MTA. • The sendmail MTA is configurable on two fronts: • The Build utilities (shipped with sendmail) allow the administrator to configure the operation of the sendmail binary (email strategy, security). • The sendmail.cf file is used to customize the local delivery rules. Email

25. Sendmail Philosophy • Don’t actually deliver email – route it. • Too many local conventions. • Too hard to customize sendmail to fit these conventions. • Generalized crossbar switch • Individual MUA’s don’t need to know where mail goes. • All knowledge about mail routing is in SMTP daemon. • All routing is consistent. • Do some protocol conversion • Basic header munging • Do whatever is necessary to route message • Better to deliver twice, than not at all • Implemented via binary (typically /usr/lib/sendmail) • Main configuration file: /etc/[mail]/sendmail.cf) Email

26. SMTP protocol (spoken by MTA’s) • HELO/EHELO – introduce yourself, and your capabilities • MAIL FROM – who is this message from? • RCPT TO – who is this message to? • DATA – what is the body of the message? • VRFY – see if this user exists. • EXPN – expand this address and tell me who it is • HELP – display the sendmail.hf file • RSET – reset the connection • NOOP – do nothing • VERB – verbose mode • DSN – delivery status notice • AUTH – authenticate this user • QUIT – close the connection Email

27. Under the hood of MTA’s • The SMTP daemon places the “message” in an envelope for delivery. • There is a header on the “letter” • There is a header on the envelope. • The headers contain addresses, and other information about the message. • They should be the same, but are not always! • Envelope headers are (usually) assigned by system software (SMTP) • Message headers can be (and are) easily forged by user. Email

28. Under the hood of MTA’s • Users typically do not see the envelope. • These headers are stripped off by the SMTP daemon. • Every message is assigned a unique ID by EACH SMTP daemon that touches it. • This allows tracing the message from end to end (if log files are available). Email

29. The following are the files typically used to configure and support the Sendmail binary. • sendmail.mc: List of macro definitions used to create the sendmail.cf file. • sendmail.cf: Master Sendmail configuration file. It contains the rules that parse mail messages, and determine how, and if, a message gets delivered. • sendmail.cw: Contains a list of hosts the mail server will accept messages for. • sendmail.hf: Contains help information for the SMTP daemon. • sendmail.pid: Contains the process ID of the running Sendmail daemon. • aliases: Contains e-mail aliases (addresses) for users on this mail server. • access.db: Contains a database of sites/hosts/users that are, or are not, allowed to access this mail server. Email

30. Parts of a sendmail.mc file • Definitions • Configured via FEATURE and DEFINE statements in .mc file • Define variables • Define()dnl • Define macros to perform functions • Feature()dnl • Define classes (sets of names) • Rules • parse address, and re-write it for transport. • Use macros, classes, and variable definitions during re-write. • Apply rules to reject spam and other messages. • Mailers • Define the mailers that are available for mail delivery on this system. Email

31. Variables That Control Sendmail Configuration • Most of the Build options for Sendmail are implemented as a series of macro definitions, and/or variables the administrator can set. • There are tens (if not hundreds) of variables that might be used to customize Sendmail. • The following is a partial list of variables that may be set via the sendmail.cf configuration file, or via the siteconfig file. • confMAILER_NAME: Sender name used for internally generated messages. • confDOMAIN_NAME: Should only be defined if your system cannot determine your local domain name. • confCF_VERSION: If defined, this is appended to the configuration version name. • confCW_FILE: Name of file containing host names this system accepts mail for. • confCT_FILE: Name of file containing the list of trusted users. • confCR_FILE: Name of file containing the list of hosts allowed to relay. • confTRUSTED_USERS: Names of users to add to the list of trusted users. This list always includes root, uucp, and daemon. • confTRUSTED_USER: Trusted user for file ownership and starting the daemon. • confSMTP_MAILER: Mailer name used when SMTP connectivity is required. Email

32. Variables That Control Sendmail Configuration • confSEVEN_BIT_INPUT: Force input to seven bits. • confEIGHT_BIT_HANDLING: Enable 8-bit data handling. • confMAX_MESSAGE_SIZE: Maximum size of messages accepted (in bytes). • confMIME_FORMAT_ERRORS: Send error messages as MIME-encapsulated messages per RFC 1344. • confFORWARD_PATH: Colon-separated list of places to search for .forward files. • confLOG_LEVEL: Log level. • confPRIVACY_FLAGS: Privacy flags. • confTIME_ZONE: Zone info. Can be USE_SYSTEM to use the system’s idea, USE_TZ to use the user’s TZ environment variable, or something else to force that value. • confUNSAFE_GROUP_WRITES: If set, group-writable, :include: and .forward files are considered “unsafe.” That is, programs and files cannot be directly referenced from such files. World-writable files are always considered unsafe. • confDONT_BLAME_SENDMAIL: Override Sendmail’s file safety checks. This will definitely compromise system security and should not be used unless absolutely necessary. • confAUTH_MECHANISMS: List of authentication mechanisms for AUTH (separated by spaces). The advertised list of authentication mechanisms will be the intersection of this list and the list of available mechanisms as determined by the CYRUS SASL library. Email

33. Sendmail.mc file variables • use_cw_file: Reads the /etc/mail/sendmail.cw file to get a list of hosts the server will accept messages for. • use_ct_file: Reads the /etc/mail/trusted-users file to get the names of users that will be “trusted.” • stickyhost: This feature is sometimes used with LOCAL_RELAY, although it can be used for a different effect with MAIL_HUB. • When used with without MAIL_HUB, e-mail sent to user@local.host is marked as “sticky” and is not forwarded to LOCAL_RELAY. With MAIL_HUB, mail addressed to user@local.host is forwarded to the mail hub, with the envelope address remaining user@local.host. Without stickyhost, the envelope would be changed to user@mail_hub, in order to protect against mailing loops. • always_add_domain: Includes the local host domain even on locally delivered mail. • ldap_routing: Implements LDAP-based e-mail recipient routing according to the Internet Draft draft-lachman-laser-ldap-mail-routing-01. • Nullclient: A special case. Creates a configuration file containing nothing but support for forwarding all mail to a central hub via a local SMTP-based network. • promiscuous_relay: By default, the Sendmail configuration files do not permit mail relaying (that is, accepting mail from outside your local host and sending it to a host other than your local hosts). Email

34. Sendmail.mc file variables • relay_entire_domain: By default, only hosts listed as RELAY in the access db will be allowed to relay. • relay_hosts_only: By default, names listed as RELAY in the access db are domain names, not host names. • relay_mail_from: Allows relaying if the mail sender is listed as RELAY in the access map. • relay_local_from: Allows relaying if the domain portion of the mail sender is a local host. • accept_unqualified_senders: Normally, MAIL FROM: commands in the SMTP session will be refused if the connection is a network connection and the sender address does not include a domain name. • accept_unresolvable_domains: Normally, MAIL FROM: commands in the SMTP session will be refused if the host part of the argument to MAIL FROM: cannot be located in the host name service (e.g., an A or MX record in DNS). • access_db: Turns on the access database feature. • blacklist_recipients: Turns on the ability to block incoming mail for certain recipient user names, host names, or addresses. • delay_checks: The rule sets check_mail and check_relay will not be called when, respectively, a client connects or issues a MAIL command. • dnsbl: Turns on rejection of hosts found in an DNS-based rejection list. Email

35. Summary of the sendmail configuration file • /etc/sendmail.cf is built from the sendmail.mc file • /etc/mail/sendmail.cf - controls nearly everything: • sets global variables and options • defines macros and classes (sets of names) • describes syntax of message headers • defines Delivery Agents (mailers) that may be used • defines rule sets • based on a “production system” programming language • Line at a time syntax • Read only at startup Email

36. Creating a sendmail.cf • Used to be a guru function. • Hand editing of an existing sendmail.cf file • Complicated • Easy to mess up • Have to understand sendmail language • Better to create a sendmail.mc file, let the system build .cf file for you • Easier to port changes • Less knowledge of language required – just need to understand macros Email

37. Rewrite rules read “tokens” and make decisions based on contents of the token stream. The left hand side of rewriting rules contains a pattern. Normal words are simply matched directly. Metasyntax is introduced using a dollar sign. The metasymbols are: $* Match zero or more tokens $+ Match one or more tokens $- Match exactly one token $=x Match any phrase in class x $~x Match any word not in class x Email

38. Example: • curt@cse.nd.edu (user@host.domain) could become 7 tokens: curt @ cse . nd . edu $1 $2 $3 $4 $5 $6 $7 $* $1=curt@cse.nd.edu $+ $1=curt@cse.nd.edu $+ @ $+ $1=curt $2=cse.nd.edu $- @ $+ $1=curt $2=cse.nd.edu $+ @ $- . $D $1=curt $2=cse $+ . $+ . $=$T $1=curt@cse $2=nd $3=edu Email

39. Sendmail Operators • When the left hand side of a rewriting rule matches, the input is deleted and replaced by the right hand side. Tokens are copied directly from the RHS unless they begin with a dollar sign. Metasymbols are: $n Substitute indefinite token n from LHS $[name$] Canonicalize name $(map key $@arguments $:default $) Generalized keyed mapping function $>n "Call" ruleset n $#mailer Resolve to mailer $@host Specify host $:user Specify user • The $n syntax substitutes the corresponding value from a $+, $-, $*, $=, or $~ match on the LHS Email

40. HSubject: $>CheckSubject D{MPat}Important Message From D{MMsg}This message may contain the Melissa virus. D{HPat}Homeworkers Needed D{HMsg}Go away spammer SCheckSubject R${MPat} $* $#error $: 553 ${MMsg} RRe: ${MPat} $* $#error $: 553 ${MMsg} R${HPat} $* $#error $: 553 ${HMsg} RRe: ${HPat} $* $#error $: 553 ${HMsg} Email

41. LOCAL_RULESETS SLocal_check_mail R$* $:$1 $| $>Local_check_numb $1 R$* $| $#$* $#$2 R$* $| $* $@ $>Local_check_bull $1 R$* $| $#$* $#$2 # SLocal_check_numb R$* $: $>Parse0 $>3 $1 R$+ < @ $+. > $* $: $(allnumbers $1 $) R@MATCH $#error $: 553 Header Error # SLocal_check_bull R$* $: $>Parse0 $>3 $1 R$+ < @ $+. > $* $: $(SpamFriend $1 $) R@MATCH $#error $: 550 We no longer accept spam email from you # # now call Basic_check_mail to continue processing # R$* $| $* $@ $>"Basic_check_mail" $1 Email

42. Reading/understanding rewrite rules requires a lot of practice! • Sendmail has a debug mode that allows the administrator to see how the rule set works. • User can use mail -v to see what happens to mail they send. • Recent additions • New rules allow user authentication. • Authentication • Block relaying as a general rule • Enable authentication • User authenticates to server. • If user is valid, allow relaying. • Can also use SSL encryption as part of this process. Email

43. Recent additions • As of sendmail 8.8 there are also new macro’s and rewrite rules that allow administrators to build in rules to block spam. • Spammers use several tricks to get their mail sent with bogus addresses: • Connect to someone else’s SMTP service and send a flood of stuff (Relaying). • Guerrilla mailers that munge the envelope and headers to hide the real point of origin. • Mailers that use bogus domains or user names in the mail from line. Email

44. Statistics • A small ISP reported that in 80 days they received 257,000 pieces of email at their SMTP server. • Of that, 2,500 messages were real mail (<1% of total). • Another 2,500 messages were automated reports (<1% of total). • The rest was SPAM! (>98% of total). • Yahoo (purportedly) delivers 1 BILLION pieces of spamperday. • This is after their filters remove 4 out of 5 spam messages that they receive at their SMTP servers! • If you multiply the number of Internet users in the USA by the average number of spam messages that they receive per day, and use minimum wage as a cost factor, spam costs the US $622 billion a year! Email

45. Relaying • A “hack” typically used to deliver spam • I create message • I send message to user@somewhere_else@your_server • Your server accepts message and delivers it for me. • You take the heat for the spam. • If I forge the headers correctly, you cannot trace it to me. • Message MIGHT be a mobile user • Mail from a staff member on the road • Mail from a staff member’s “home” account • Need to control relaying • Detecting Relaying - is incoming message destined for a user on this host? • If so, accept. • If not, this is a relay attempt. Email

46. The anti-spam rules can check: • Valid sender hostname - look up sending host in DNS. • If it does not exist, do not accept mail. • Access databases - Local and global lists may be checked: • RBL - network services which tracks spammers, relay sites, and dialup ISP’s. • Works like DNS – look up host at the RBL service, if an answer comes back from RBL server, do not accept this message. Email

47. The anti-spam rules can check: • access.db - a local database of sites/users you will not accept mail from. • Create a text file with a key, and an action: • Key is one of: a user name, a domain name, or a user@domain • Action is one of: OK, REJECT, or RELAY Email

48. Access Database nd.edu RELAY cselab.nd.edu RELAY cse.nd.edu RELAY austin.ibm.com RELAY 129.74 OK nobody@ REJECT largetvs@ REJECT customersupportrep@ REJECT bttb.net REJECT free-virtualweb.org REJECT thefreevirtual.org REJECT friend.of.you@ REJECT Email

49. anti-spam rules: • Rulesets can return fatal, or transient error messages • Fatal error - sender gets a bounce message. • transient error - ties up resources, no bounce message. • Check Subject line and other headers • Useful for virus rejects and anti-spam. Email

50. When building the sendmail binary, there are several options that need to be considered: • By default, sendmail will use the OS’s simple database facilities. • Often, this is not a good choice. You may need to install a more robust database package for use with sendmail. • You may need to compile name service or directory service (LDAP, NIS, …) information into your sendmail. • Generally want to put localisms in the devtools/Site directory in a file called siteconfig.m4 • Use the Build utilities to include the localisms into the sendmail binary. Email