1 / 36

Lessons Learned while Writing a Java Mailserver

Lessons Learned while Writing a Java Mailserver. by Richard O. Hammer. about me. 70s – BSEE, coded in Fortran and APL. 80s – went to CS graduate school at UNC-CH, ABD. Started a building business. 90s – started a libertarian think tank.

paul
Download Presentation

Lessons Learned while Writing a Java Mailserver

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Lessons LearnedwhileWriting a Java Mailserver by Richard O. Hammer

  2. about me • 70s – BSEE, coded in Fortran and APL. • 80s – went to CS graduate school at UNC-CH, ABD. Started a building business. • 90s – started a libertarian think tank. • Y2K – returned to programming to make a living, focus on Java and Internet programming.

  3. The Familiar Catch-22 • You can’t get a job without experience. • You can’t get experience without a job. • So I needed a project to work on at home.

  4. Why this Project? • Spam is an interesting problem to me. It is a “tragedy of the commons”, an example of lawless behavior in a public space. • An email service could eliminate spam in a novel way – by charging unlisted senders. • Along the way I would learn about programming at the level of sockets and network protocols – where the problem of Internet lawlessness seems to start.

  5. Minimal Specification:to offer an email service whicheliminates incoming spam • Customers can manage a list of “accepted” senders • All messages from other “unlisted” senders are waylaid. Automatic email is sent back to these senders, explaining the situation, giving them the option to pay for delivery • Works with usual email client, such as Mozilla Thunderbird or Outlook Express.

  6. Wish-List Specification:(postponed) • Transform payment into bond, with easy way for customer to order refund. • Add other ways for unlisted senders to authenticate themselves. • Make the service scalable, to handle high volume of traffic. • Install defenses against denial of service attacks. • And many more features.

  7. This Presentation Is Not • Entirely up to date (by a few years) • Comprehensive Yet It Is • Basic material that anyone who programs email will need to know

  8. Protocols Specification • I want people to be able to have email accounts with me. What do I have to offer? • Protocols required? • Services required?

  9. How an Email message is Transmitted Sender Email Client such as Thunderbird or Outlook Express simplified SMTP Recipient’s ISP Mailserver or Mail Transfer Agent SMTP Mailserver or Mail Transfer Agent Sender’s ISP POP3 Email Client such as Thunderbird or Outlook Express Recipient

  10. A look at SMTP • RFC (821, 1982), superseded by 2821, 2001 • client opens a socket to port 25 on the SMTP server • SMTP server identifies itself

  11. SMTP in Actiontelnet from DOS C:\>telnet XDomain.com 25 220 gork.XDomain.com ESMTP Postfix helo mailscreen.net 250 gork.XDomain.com mail from:<sue@yrox.net> 250 Ok rcpt to:<fred@XDomain.com> 250 Ok data 354 End data with <CR><LF>.<CR><LF> Here is a message to you. . 250 Ok: queued as 63235619F2ED quit 221 Bye Connection to host lost.

  12. Content of SMTP dataRFC (822) 2822 data 354 End data with <CR><LF>.<CR><LF> Received: from BILL ([73.175.198.24]) by vm024.mailsvcs.com for rHammer@mailscreen.net; Wed, 03 Jan 2007 07:07:50 -0600 (CST) Date: Wed, 03 Jan 2007 08:07:49 -0500 From: "Bill Hill" <bHill@Verizon.Net> Subject: RE: UFO sighting To: "Richard Hammer" <rHammer@mailscreen.net> MIME-version: 1.0 Content-type: text/plain; charset=windows-1250 Content-transfer-encoding: 7bit Hi Rich, Interesting! Maybe E. Jackson should take a look at it :) Bill . 250 Ok: queued as 63235619F2ED HEADERS MIME HEADERS RFC 2045 RFC 2822 headers end with a blank line

  13. Addresses in SMTP envelope may differ from Addresses in headers SMTP (RFC 2821) envelopevsRFC 2822 headers mail from:<sue@yrox.net> 250 Ok rcpt to:<fred@XDomain.com> 250 Ok data 354 End data with <CR><LF>.<CR><LF> From: "Bill Hill" <bHill@Verizon.Net> To: "Richard Hammer" <rHammer@mailscreen.net> My message to you. . 250 Ok quit 221 Bye

  14. Socket Programming • java.net.Socket • java.net.ServerSocket • On top of TCP

  15. public class GetterServer implements Runnable{ boolean gracefulShutdownOrdered; ServerSocket receiverServerSocket; Socket receiverConnectionSocket; public GetterServer() throws Exception{ receiverServerSocket = new ServerSocket(25); } public void run(){ while(!gracefulShutdownOrdered){ try{ try{ receiverConnectionSocket = receiverServerSocket.accept(); }catch (IOException e){ if(gracefulShutdownOrdered) break; else{ logStackTrace("Exiting GetterServer.", e); break; } } Getter myMan = new Getter(); myMan.handleConnection(receiverConnectionSocket); }catch (RuntimeException runt){ logStackTrace(runt); } } } ... } Top Level Code for a SMTP server

  16. protocol programming • We will look at three methods from class Getter, adapted from: org.apache.james.smtpserver.SMTPHandler

  17. method handleConnection(Socket) class Getter{ //adapted from void handleConnection(Socket socket) { remoteHost = socket.getInetAddress().getHostName(); remoteIP = socket.getInetAddress().getHostAddress(); in = new BufferedInputStream(socket.getInputStream(), 7024); smtpCommandLineReader = new CommandLineReader(in); out = new InternetPrintWriter(socket.getOutputStream(), "Getter says: ", true); out.println("220 " + helloName + " ready "); String clientCommand = null; try { boolean getAnotherCommand; do { clientCommand = smtpCommandLineReader.readLine(); getAnotherCommand = parseCommand(clientCommand); } while (getAnotherCommand); } catch (Exception e) { if (shutdownOrdered) { //log normal closing return; } else if (e instanceof SocketException) { //log unusual condition } else {// might be major //log error } } closeConnections(); }

  18. private boolean parseCommand(String command) throws Exception { if (command == null) return false; String argument = null; String argument1 = null; command = command.trim(); if (command.indexOf(" ") > 0) { argument = command.substring(command.indexOf(" ") + 1); command = command.substring(0, command.indexOf(" ")); if (argument.indexOf(":") > 0) { argument1 = argument.substring(argument.indexOf(":") + 1); argument = argument.substring(0, argument.indexOf(":")); } } if (command.equalsIgnoreCase("HELO")) doHELO(command, argument, argument1); else if (command.equalsIgnoreCase("EHLO")) doEHLO(command, argument, argument1); else if (command.equalsIgnoreCase("MAIL")) doMAIL(command, argument, argument1); else if (command.equalsIgnoreCase("RCPT")) doRCPT(command, argument, argument1); else if (command.equalsIgnoreCase("AUTH")) doAUTH(argument); else if (command.equalsIgnoreCase("DATA")) { doDATA(); } else if (command.equalsIgnoreCase("QUIT")) doQUIT(command, argument, argument1); else doUnknownCmd(command, argument, argument1); return !(command.equalsIgnoreCase("QUIT") == true || shutdownOrdered); } method parseCommand(String)

  19. method doMAIL(String, String, String) private void doMAIL(String command, String argument, String argument1) { if (!gotEhlo) { out.println("503 bad sequence of commands, send ehlo or helo first"); return; } if (argument == null || !argument.equalsIgnoreCase("FROM") || argument1 == null) { out.println("501 Usage: MAIL FROM:<sender>"); return; } String sender = argument1.trim(); if (sender.length() < 2 || sender.charAt(0) != '<' || sender.charAt(sender.length() - 1) != '>') { out.println("501 Usage: MAIL FROM:<sender>"); return; }

  20. //method doMAIL concluded MailAddress senderAddress = null; sender = sender.substring(1, sender.length() - 1); if (sender.length() > 0) {// the usual case try { senderAddress = new MailAddress(sender); } catch (ParseException e) { out.println("501 Failure to parse InternetAddress from " + "\"" + argument1.trim() + "\". " + e.getMessage()); return; } // screen senders with mailscreen addresses if (Services.isToLocalDomain(senderAddress)) { if (authenticatedUser == null) { out.println("530 Authentication Required"); return; } else if (!CustomerSet.isMsAddress(senderAddress)) { out.println("550 No such address"); return; } } } else { // sender.length() == 0, must have been <> // let senderAddress remain null, as that is our signal of mail from <> } setEnvelopeFrom(senderAddress); out.println("250 Sender " + sender + " OK"); }

  21. Mailscreen.netas it now runs • A free offering of the service has been available since March 2004

  22. SMTP CLIENT TOMCAT JSP POP3 MAILBOXES POSTGRES SMTP SERVER Unlisted Sender T 1.0 PAYPAL Unlisted Sender Pays HTTPS T 0.0 HTTPS URL T 1.1 TO: JOE@Mailscreen.net FROM: Unlisted Sender T 1.2 Payment Required Message TO: Unlisted Sender FROM: customer-agent@mailscreen.net T 1.3 T 0.2 MAILSCREEN Instant Payment Notificaton HTTP POST Echo Parameteres "VERIFIED" T 2.0 T 1.4 T 0.1 TO: JOE@Mailscreen.net FROM: Unlisted Sender

  23. Payment Required Message Dear Somebody@somewhere.org: Mailscreen.net has received your email for a Mailscreen customer, service@mailscreen.net. Unfortunately, since your email address is not registered as an accepted sender to this customer, Mailscreen will not deliver your message to the customer until you make a payment as described in the next paragraph. The charge to send this message will be $0.50. To make this payment click on the link below. It will take you to PayPal.com where you can make payment, and where you can open a new PayPal account if you do not already have one. Mailscreen does not accept any other form of payment at present. https://www.paypal.com/xclick/business=cashier@mailscreen.net&item_name=accept+email&invoice=060410.204815c5i0&amount=0.50&return=http%3A//mailscreen.net/thanks.jsp Please make your payment promptly. We cannot promise fulfillment if your payment comes after four calendar days, because we must limit the number of messages that we hold at Mailscreen. If you would like more specific information to identify the message which we received from you and which we are holding awaiting payment, here are some identifying headers from that message. Date: 10 April 2006 From: Somebody@somewhere.org Received: FROM nc-60-41-56-62.dyn.yahoo.net ([60.41.56.12]) BY mailscreen.net; Mon, 10 Apr 2006 20:48:15 -0400 (EDT) Sincerely, Mailscreen.net

  24. Instant Payment Notification, in the JSP that PayPal calls <% Enumeration en = request.getParameterNames(); StringBuffer replyBuf = new StringBuffer("cmd=_notify-validate"); while(en.hasMoreElements()){ String paramName = (String)en.nextElement(); String paramValue = request.getParameter(paramName); replyBuf.append("&").append(paramName).append("=") .append(URLEncoder.encode(paramValue,"UTF-8")); } // post back to PayPal system to validate URL url = new URL("http://www.paypal.com/cgi-bin/webscr"); URLConnection uc = url.openConnection(); uc.setDoOutput(true); uc.setRequestProperty("Content-Type","application/x-www-form-urlencoded"); PrintWriter pw = new PrintWriter(uc.getOutputStream()); pw.println(replyBuf.toString()); pw.close(); BufferedReader in = new BufferedReader( new InputStreamReader(uc.getInputStream())); String res = in.readLine(); in.close(); if (!"VERIFIED".equals(res)){ log("Payment failed. res not VERIFIED but "+ res); return; }

  25. JSP Control Consoles • Administrator • Customer

  26. PROBLEMHow can calls from Web Appreach the objects in the mailserver? SOLUTION CHOSEN Run Web App and Mailserver in the same JVM

  27. Starting the Mailserver It Is a Servlet public class StartServlet extends HttpServlet { public void init(){ ServerManager.start(); } public void destroy(){ ServerManager.myOneServerManager.startGracefulShutdown(); } }

  28. mailscreen web.xml file <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd"> <web-app> <servlet> <servlet-name>do it damn it</servlet-name> <servlet-class>common.StartServlet</servlet-class> <load-on-startup>3</load-on-startup> </servlet> <!-- security for regular customers --> <security-constraint> <web-resource-collection> <web-resource-name>customerResource</web-resource-name> <url-pattern>/customer/*</url-pattern> </web-resource-collection> <auth-constraint> <role-name>customer</role-name> </auth-constraint> <user-data-constraint> <transport-guarantee>CONFIDENTIAL</transport-guarantee> </user-data-constraint> </security-constraint> ...

  29. Tomcat server.xml file <Server port="8005" shutdown="SHUTDOWN"> <Service name="Catalina"> <Connector port="80" maxThreads="15" minSpareThreads="1" maxSpareThreads="2" enableLookups="false" redirectPort="443" acceptCount="6" connectionTimeout="20000" disableUploadTimeout="true" /> <Connector port="443" maxThreads="10" minSpareThreads="1" maxSpareThreads="2" enableLookups="false" disableUploadTimeout="true" acceptCount="5" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" /> <Engine name="Catalina" defaultHost="localhost"> <Host name="mailscreen.net" appBase="/usr/website/WebRoot" unpackWARs="true" autoDeploy="true"> <Alias>www.mailscreen.net</Alias> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="msLogs" prefix="access." suffix=".log" pattern="common" resolveHosts="false"/> <Realm className="org.apache.catalina.realm.JDBCRealm" debug="99" driverName="org.postgresql.Driver" connectionURL="jdbc:postgresql://localhost:3246/pqlms" connectionName="mailData" connectionPassword="e3$fe4uii0p" userTable="uLogin" userNameCol="uName" userCredCol="cred" userRoleTable="uRole" roleNameCol="power" /> <Context path="" docBase="." /> </Host> </Engine> </Service> </Server>

  30. Complexity of SMTP sending, Classification draft M4 - a message as received in SMTP, with potentially: • many domains among the recipients • many delivery attempts (try or retry) necessary for each domain • many servers (IP addresses) to attempt during each delivery attempt • many recipient addresses to attempt when a connection to a server succeeds M3 - part of an M4, with recipients at only one domain, but with potentially: • many delivery attempts (try or retry) necessary for each domain • many servers (IP addresses) to attempt during each delivery attempt • many recipient addresses to attempt when a connection to a server succeeds M2 - one of attempts (try or retry) to deliver a M3, but with potentially: • many servers (IP addresses) to attempt during each delivery attempt • many recipient addresses to attempt when a connection to a server succeeds M1 - part of an M2, the attempt to deliver an M3 to one server (IP address), but with potentially: • many recipient addresses to attempt in this connection M0 - part of an M1, the attempt to deliver an M3 to one user address at one server (IP address) during one try or retry attempt

  31. (abstract) M4 List: envelopeTo MailAddress: envelopeFrom String: id File: dataFile MailscreenKey Classes 1..* 1 DistinctDataMessage DDM String: ddmId, status boolean: disposable (abstract) (interface) MessageToOneRemoteDomain MORD List: recipients String: domainName Remote Recipient RR MailAddress: address String: status MessageToMsRecipient MSR void deliverTermsMet() void stopWaiting() 1..* 1 MessageToNonMs MsMailBoxEntry void writeTo(OutputStream) features ForwardedMessage toCustomer features 1 1 1 1 MsMessageFeatures Customer: customer MailHeaders: headers

  32. Delivery Status Notification This notification pertains to a message you sent. You may identify that message by these headers taken from it: To: susie@foggybottom.net Subject: coming through Date: Sun, 28 Mar 2004 20:22:21 -0500 This message could not be delivered to one or more recipients. A permanent failure was encountered while attempting to deliver to: susie@foggybottom.net No further action will be taken here on your message.

  33. Lessons about Spam • Spam is not bad enough to drive people to the tactless solution offered by Mailscreen. • People do not want to risk the possibility that a cherished contact might receive a “payment required” message. • The problem is not just technical, because there are many technical solutions. • The problem is sociological.

  34. References • RFC Index http://www.rfc-editor.org/rfc-index.html • Delivery Status Notifications RFC 1891, 1894 • Java Network Programming, 2nd ed., Elliotte Rusty Harold, 2000 • Programming Internet Email, David Wood, 1999

  35. Libraries Used • org.xbill.DNS • javax.mail Source Available to You at • http://mailscreen.net/jug.zip • for one week, until Jan. 22, 2007

  36. Thanks To • Sun Microsystems, for Java, the Java API • Apache James project • Ralph Cook • IETF email list, ietf-smtp@imc.org • Triangle Java Users Group

More Related