Design and Management of Email Service

1. Design and Management ofEmail Service 張傑生Jason Chang Computer Center National Taiwan University 2005/07/08 北區區網訓練課程 jsc@ntu.edu.tw http://ccms.ntu.edu.tw/~jsc

2. Outline • A brief view of NTU email service • Introduction to the architecture and operation of SMTP • Design of a suitable email system • Webmail solutions • Postfix and simple configuration samples • Spam and virus filtering • Conclusion

3. Overview • Electronic mail service has already evolved into one of the major Internet applications. • It is not only fundamental, but also a must. • Users may become impatient when mails were delayed, not to mention failed to access their emails. • Imagine we meet the situation of power failure or cut of water supply

4. A Brief View of NTU Email Service • Current Status • Email statistics • Average daily incoming: 800,000 • Truly delivered: 200,000(25%) • Account statistics • Faculty/staff/official use: 4200 • Student: 30000 • Alumni: 24500 • Account policy • Lifelong preserved for retired faculties/staffs and alumni • Server design • Incoming relay/filter: 3 • Outgoing relay: 1 • Webmail: 2 • Pop3: 5 for students and 1 for others

5. A Brief View of NTU Email Service • Problems we faced • Account creation and remove • Non-decreasing number of users • Spam • What we have done • Database synchronization with campus mis systems • Single Sign On • Totally encrypted transmission • Spam and virus filtering • Ugly but work architecture for scalability concern

6. Architecture of a Simple Mail System • Consists of the following components • MTA - Mail transfer agent • Sending and forwarding emails • Server end • MDA - Mail delivery agent • Delivering emails to recipients’ mailbox • Server end • Pop3/Imap4 Daemons • For users to download their mailboxs • Server end • MUA - Mail user agent • Reading and composing emails • Client end

7. Architecture of a Simple Mail System

8. Protocols Used for Mail System • Protocols • For computer programs to communicate with each other • Similar to languages that human beings speak • SMTP • Simple Mail Transfer Protocol • Too simple to provide any “advanced features” • Authentication • Authorization • POP3 • Post Office Protocol version 3 • Simple • IMAP4 • Internet Message Access Protocol version 4 • Fully compatible with internet message standards, e.g. MIME. • Allow messages to be accessed from more than one computer. • Provide support for online, offline, and disconnected modes. • Multiple and share folders.

9. Mail Forwarding Between Servers

10. How to Find the Way to the Destination? • How do we find the way tojsc@ntu.edu.tw?

11. DNS: The Key to All Internet Services • Query DNS server by the address part of email address.(jsc@ntu.edu.tw) • Query for MX records • Query for A records • Give up! • Not required to exist a real machine hosting the address.(ntu.edu.tw) • Adding MX records to improve query performance and to reduce unnecessary query overhead.

12. DNS: Query Results • Precedence value of MX records • Lower valueswith higher priority

13. DNS: Query Results • When MX record is absent, second query is required to issue.

14. The Use of Mail Relay Gateway • Load sharing and balance • Prevent the occurrence of single point of failure • Hide your backend mail servers and prevent from outside attack • Frequently used when you have M$ Exchange servers • Make use of multiple network link connections (different ISPs) • Spam and virus filtering • The easiest way for administrators to release current bottlenecks and to leverage performance. • It’s the way how most commercial products work.

15. The Architecture of Mail Relay Gateways

16. Planning of a Modern Mail System • This is what system administrators care. • Scalability • User / spool capacity • Virtual user management • Availability • Multiple relay servers • Multiple pop3 servers • Security • Avoid creating system accounts • No access to shell • Performance • Make the right decision • Architecture • OS/Software

17. Features Required By Modern Mail Systems • This is what end users care. • Ease of use interface • IMAP4 • Webmail • Security • Authentication • Encrypted transmission • Spam and virus filtering • The only effective way to prevent from virus infection is to remind and force your users to install antivirus softwares. • Stop using M$ systems is alternative solution.

18. Scalability of a Mail System • Traditional way: • Additional servers with different address. • ms1.cc.ntu.edu.tw • ms2.cc.ntu.edu.tw • ms3.cc.ntu.edu.tw • Ugly and not professional • Modern and Clever way: • Multiple backend servers • Make use of relay gateways to dispatch and deliver mails • A uniform email address for all of the users. • user@ntu.edu.tw • Virtual user management by central database

19. A Scalable Mail System Architecture

20. Improving Scalability and Availability By Adding Multiple POP3 Servers • POP3/IMAP4 accesses are more frequent than incoming mail delivery. • Users may easily aware of the failure of pop3 servers in comparison to smtp/relay servers.(in:out = 20:1) • NFS protocol is used to share the common storage. • Mail delivery efforts can be dealt by other servers. • Totally rely on the stability of storage server. • Single point of failure • Could be replaced by high-end NFS servers or NetApp filers.

21. Architecture of Multiple POP3 Servers

22. NFS Concerns And Bottlenecks • NFS has its own limitation. • Do not expect NFS to scale well. • Use gigabit network with high quality switch. • UDP is preferred over TCP when network environment is local and qualified. • Consider to apply the Maildir format • Avoid NFS lock problem • Severe enough in Linux and BSD systems • Spin lock and dead lock may block further accesses • Manual interference often requires

23. Make the Right Decision For the Promise of Performance • Capacity planning • Reasonable system design and architecture • Operating system • Linux, BSD • Always the best and wise choice. • Solaris • It’s been called Slow……laris. • M$ • Are you kidding?

24. Make the Right Decision For the Promise of Performance • MTA • Postfix • Our suggestion! • Sendmail • Not scalable! • Lack of new features. • Not efficiency in resource management. • Rich of documents and tutorials • Qmail • Termination of development. • Exim • Immature. • M$ Exchange • Seeking trouble for yourself? • POP3/IMAP4 • UW-IMAP • Courier-IMAP • Supports Maildir format. • Dovecot • Supports both mbox and Maildir format. • Still under development. • Cyrus • Proprietary spool format. • Not easy to convert between other formats.

25. Webmail: The Trend of Browser Based Mail Reader • Your determination depends on the system architecture. • IMAP4 based • Access the mailbox via IMAP4 protocol locally or remotely. • Flexible, scalable • Speed may be limited by protocol overhead and network transmission. • Horde IMP project • Locally direct access • Access the mailbox locally, proprietary protocol. • Could only run on local mail storage server. • NFS could be applied when deploying multiple servers • Extremely fast!! iff system not overloaded. • Not scale well when you want to expand. • Openwebmail • Excellent project developed by NCKU, Taiwan • Marvelous support to Chinese encodings • Only support mbox format currently

26. Webmail Architecture – IMAP4 based

27. Webmail Architecture – Locally Direct Access

28. Postfix: The Modern and Advance MTA • http://www.postfix.org • Developed by Wietse Venema • Features • Easy replacement of existing sendmail system by binaries with same file name and functionality . e.g. sendmail, newaliases • Human readable configuration file • Multiple small programs with limited execution privilege • Multiple queues to avoid block of normal mails • Backend database lookup table supported • Better support for virtual host • Virtual user system supported • Content filtering interface • Simple spam check/block mechanism

29. Postfix - the Big Picture • http://www.postfix.org/big-picture.html • Yellow ellipsoids are mail programs. • Yellow boxes are mail queues or files. • Blue boxes are lookup tables.

30. Postfix Installation • Debian Linux • apt-get install postfix-tls libsasl7 libsasl-modules-plain courier-imap • Redhat/Fedora Linux • rpm -ivh postfix-2.2.x.i386.rpm • rpm -ivh cyrus-sasl-2.1.21.i386.rpm • BSD • Use the ports system • /usr/ports/mail/postfix • /usr/ports/security/cyrus-sasl2 • /usr/ports/mail/courier-imap

31. Post Installation • Postfix Configuration • master.cf • Similar to inetd.conf • Control the behavior of small programs • In contrast against sendmail, with one binary and one config file • main.cf • The main configuration of the mail system • In general cases, no modification is required for a simple setup. • Commands to control the postfix program • postfix start • postfix stop • postfix reload

32. Test For Receiving Email • Send an email to yourself. • Check system logs • /var/log/maillog • An simple test from jason@dolphin.cc.ntu.edu.tw to jason@freebsd.csie.nctu.edu.tw

33. myhostname = mail.cc.ntu.edu.tw mydomain = cc.ntu.edu.tw myorigin = $mydomain mydestination = $myhostname, localhost.$mydomain On a mail domain gateway,you should also include $mydomain. mynetworks_style = host relay_domains = $mydestination home_mailbox = Maildir/ message_size_limit = 20971520 smtpd_recipient_limit = 1000 Frequently Used Configuration Options In main.cf

34. Configure to Enable SSL • main.cf • smtpd_enforce_tls = yes • smtpd_use_tls = yes • smtpd_tls_cert_file = /usr/local/etc/ssl/smtp.cert • smtpd_tls_key_file = /usr/local/etc/ssl/smtp.key • smtpd_tls_CAfile = /usr/local/etc/ssl/ntuca.crt • smtpd_tls_loglevel = 1 • smtpd_tls_received_header = yes • smtp_enforce_tls = yes • smtp_tls_cert_file = $smtpd_tls_cert_file • smtp_tls_key_file = $smtpd_tls_key_file • smtp_tls_CAfile = $smtpd_tls_CAfile • smtp_tls_loglevel = 1 • smtp_use_tls = yes • smtp_tls_note_starttls_offer = yes • tls_random_exchange_name = /var/run/prng_exch • tls_random_source = dev:/dev/urandom • tls_daemon_random_source = dev:/dev/urandom • master.cf • tlsmgr fifo - - n 300 1 tlsmgr

35. Test for SSL/TLS Connection • telnet to mail server and check for welcome messages.

36. Test for SSL/TLS Connection • Send an email to yourself. • From mx1 to ccms • Check system logs • /var/log/maillog

37. SMTP/Authentication • Traditional mail servers could only determine their relay policy based on connected ip addresses. • Insufficient in face of the modern network • DHCP, NAT • ADSL and dialup users • Roaming users • Mobile, wireless • SMTP Authentication • Username/password authentication for each mail delivery. • Reduce the chance to relay spam mails for anonymous.

38. Setting Up SMTP/Authentication • Install Cyrus-SASL2 library • It could be configured to authenticate via /etc/passwd, PAM, LDAP, backend databases • Modify /usr/local/lib/sasl2/smtpd.conf • pwcheck_method: saslauthd • mech_list: plain login • Make sure saslauthd is running.

39. Configure to Enable SMTP/Authentication • Since the username/password negotiation is not encrypted. We must configure our server to accept incoming mail authentication only on TLS encrypted connections. • main.cf • smtpd_sasl_auth_enable = yes • smtpd_sasl_security_options = noanonymous • smtpd_tls_auth_only = yes • smtpd_recipient_restrictions = reject_unknown_recipient_domain, reject_non_fqdn_recipient, permit_sasl_authenticated, reject_unauth_destination • master.cf • smtps inet n - n - - smtpd -o smtpd_tls_wrappermode=yes • The smtpd will listen on port 465 instead of 25.

40. Test For SMTP/Authentication Connection • The username/password is required to be encoded in BASE64. • The Perl MIME::Base64 module could be installed. • /usr/ports/converters/p5-MIME-Base64 • perl -MMIME::Base64 -e 'print encode_base64("user\0user\0passwd");‘ • dGVzdDEAdGVzdDEAcGFzczE=

41. Test For SMTP/Authentication Connection • Send an email to yourself. • Check system logs • /var/log/maillog

42. SMTP Transaction Example

43. Email Headers • Envelope header • Negotiated between mail servers • Typical tricks that SPAM mail plays • Why do I receive mails that are not destined to me? • Content Text • Mail servers will not examine the content in general cases.

44. Real World Snail Mail Example • Postman could only deliver the mail by envelope information. • Sender address is not verified when email delivery. • Trick of SPAM mails. • Content may be irrelevant or conflict with envelope.

45. Postfix: Spam Filtering Functions • SMTP access restriction • Refer to the previous SMTP transaction example • http://www.postfix.org/uce.html

46. Postfix: Spam Filtering Functions • Access maps • Similar to sendmail access map • Control relay permission based on IP addresses • main.cf • smtpd_client_restrictions = check_client_access hash:/usr/local/etc/postfix/access • Execute postmap access after update • Dynamic IP ranges should be blocked • http://www.twia.org.tw/SPAM-930412(news).htm

47. Postfix: Spam Filtering Functions • Realtime black list • Use with caution • The definition and meaning of each rbl • Too many TW hosts are on the lists • main.cf • maps_rbl_domains = bl.spamcop.net, relays.ordb.org, orbs.dorkslayers.com • smtpd_client_restrictions = reject_maps_rbl

48. Postfix: Spam Filtering Functions • SMTP access restriction • smtpd_client_restrictions • maps_rbl_domains = bl.spamcop.net, relays.ordb.org, orbs.dorkslayers.com • smtpd_client_restrictions = reject_rbl_client relays.ordb.org reject_rbl_client bl.spamcop.net reject_maps_rbl check_client_access regexp:/usr/local/etc/postfix/access reject_unauth_pipeliningreject_unknown_client(use with caution) • smtpd_helo_restrictions • smtpd_helo_required = yes • smtpd_helo_restrictions = reject_invalid_hostname reject_unknown_hostname reject_non_fqdn_hostname

49. Sample Rejected Logs

50. Postfix: Spam Filtering Functions • SMTP access restriction • smtpd_sender_restrictions • smtpd_sender_restrictions = reject_unknown_sender_domain reject_non_fqdn_sender check_sender_access hash:/etc/postfix/sender-access • Useful to block SPAM mails that fake envelope mail from header