LuaSocket behind the scenes

1. LuaSocketbehind the scenes Diego Nehab

2. Short Bio • Graduated from PUC in CS & E, 1999; • Worked in Tecgraf 1995-2002; • MSc in PL with Roberto, 2001; • 3rd year PhD candidate at Princeton; • Computer Graphics.

3. Outline of talk • A few historical notes • Case study: SMTP support • Protocol abstraction • Message abstraction • Implementation highlights • Conclusions

4. Historical notes • 1.0, 1999, 1.5k C, 200 man • 1.1, 2000, 1.5k C, 1.3k Lua, 500 man • added protocol support for HTTP, SMTP, FTP • 1.2, 2001, 2k C, 1.3k Lua, 900 man • buffered input and non-blocking I/O • UDP support • object oriented syntax

5. Historical notes • 1.3, 2001, 2.3k C, 1.6k Lua, 1.2k man • streaming with callbacks • added select function • 1.4, 2001-2, 2.2k C, 2.2k Lua, 1.9k man • LTN7 • added URL module • named parameters

6. David Burgess Current version • 2.0, 2005, 4.6k C, 2.5k Lua, 4.7k man • Extensible C architecture, split in modules • LTN12 (sources, sinks and filters) • MIME support (partial but honest) • Multipart messages support • LTN13 (finalized exceptions) • Package proposal • Improved non-blocking code, robust to signals...

7. Outline of talk • A few historical notes • Case study: SMTP support • Protocol abstraction • Message abstraction • Implementation highlights • Conclusions

8. from rcpt body SMTP (RFC2821) [lua:roberto] telnet mail.tecgraf.puc-rio.br 25 220 tecgraf.puc-rio.br ESMTP Sendmail 8.9.3/8.9.3 helo lua 250 tecgraf.puc-rio.br Hello lua, pleased to meet you mail from: <roberto@inf.puc-rio.br> 250 <roberto@inf.puc-rio.br>... Sender ok rcpt to: <diego@tecgraf.puc-rio.br> 250 <diego@tecgraf.puc-rio.br>... Recipient ok data 354 Enter mail, end with "." on a line by itself Subject: World domination: instructions. Commence stage two. . 250 RAA10452 Message accepted for delivery quit 221 tecgraf.puc-rio.br closing connection

9. Protocol abstraction status, error = smtp.send { from = "<roberto@inf.puc-rio.br>", rcpt = "<diego@tecgraf.puc-rio.br>", body = "Subject: World domination: instructions.\r\n\r\n" .. "Comence stage two." } • What if body is large?

10. LTN12 sources • Use callback function that produces data; • Returns one chunk each time called; • Signals termination returning nil. function ltn12.source.file(handle) return function() local chunk = handle:read(BLOCKSIZE) if not chunk then handle:close() end return chunk end end

11. Using sources status, message = smtp.send { from = "<roberto@inf.puc-rio.br>", rcpt = "<diego@tecgraf.puc-rio.br>", body = ltn12.source.file(io.open("/mail/body", "r")) } • What if body is complicated?

12. Message Format (RFC2822) From: Roberto Ierusalimschy <roberto@inf.puc-rio.br> To: Diego Nehab <diego@tecgraf.puc-rio.br> Subject: World domination: roadmap. Content-Type: multipart/mixed; boundary=part This message contains attachments --part Content-Type: text/plain Please see attached roadmap. --part Content-Type: text/html; name="roadmap.html" ... --part-- headers headers part 1 body body part 2

13. Message abstraction declaration = { headers = { subject = "World domination", from = "Roberto <roberto@inf.puc-rio.br>", to = "Diego <diego@tecgraf.puc-rio.br>" }, preamble = "This message contains attachments.", [1] = { headers = { ... }, body = "Please see attatched roadmap." }, [2] = { headers = { ... }, body = ltn12.source.file(io.open("/plans/roadmap.html", "r")) } }

14. Our message API status, message = smtp.send { from = "<roberto@inf.puc-rio.br>", rcpt = "<diego@tecgraf.puc-rio.br>", body = smtp.message(declaration) } • Transform declaration into an LTN12 source; • Pass source as body to sending function.

15. How hard is it? • Message structure is recursive; • Need to return chunks but mantain context; • Nightmare to write in C!; • Use coroutines; • Write function recursively, naturally; • Call yield with each chunk; • Next call resumes wherever we left.

16. Zoom in on attachments [2] = { headers = { ["content-type"] = 'text/html; name="roadmap.html"', ["content-disposition"] = 'attachment; filename ="roadmap.html"' }, body = ltn12.source.file(io.open("/plans/roadmap.html", "r")) } • Would like to send PDF; • Binary data has to be encoded (Base64); • Want to encode on-the-fly.

17. LTN12 filters and chains • Filters process data one chunk at a time; • MIME module provides common filters: • base64, quoted-printable, stuffing, line-wrap... • Can chain two filters together: factory • Produce a filter with the composite effect • Can chain a filter with a source: factory • Produce a source that returns filtered data.

18. Zoom in on attachments [2] = { headers = { ["content-type"] = 'application/pdf; name="roadmap.pdf"', ["content-disposition"] = 'attachment; filename ="roadmap.pdf"', ["content-description"] = 'Detailed world domination plan', ["content-transfer-encoding"] = 'BASE64' }, body = ltn12.source.chain( ltn12.source.file(io.open("/plans/roadmap.pdf", "r")), ltn12.filter.chain( mime.encode("base64"), mime.wrap("base64") ) ) }

19. Creating filters: high-level • Chunks can be broken arbitrarily; • Filters have to keep context between calls; function ltn12.filter.cycle(low, ctx, extra) return function(chunk) local ret ret, ctx = low(ctx, chunk, extra) return ret end end function mime.normalize(marker) return ltn12.filter.cycle(mime.eol, 0, marker) end

20. Creating filters: low-level int eol(lua_State *L) { int ctx = luaL_checkint(L, 1); size_t isize = 0; const char *input = luaL_optlstring(L, 2, NULL, &isize); const char *last = input + isize; const char *marker = luaL_optstring(L, 3, CRLF); luaL_Buffer buffer; luaL_buffinit(L, &buffer); while (input < last) ctx = translate(*input++, ctx, marker, &buffer); luaL_pushresult(&buffer); lua_pushnumber(L, ctx); return 2; }

21. Creating filters: low-level #define candidate(c) (c == CR || c == LF) int translate(int c, int last, const char *mark, luaL_Buffer *buffer) { if (candidate(c)) { if (candidate(last)) { if (c == last) luaL_addstring(buffer, mark); return 0; } else { luaL_addstring(buffer, mark); return c; } } else { luaL_putchar(buffer, c); return 0; } }

22. SMTP dependencies socket tp smtp ltn12 mime

23. Error checking • Function return convention • Return nil, followed by message on error; function metat.__index:greet(domain) local r, e = self.tp:check("2..") if not r then return nil, e end r, e = self.tp:command("HELO", domain) if not r then return nil, e end return self.tp:check("2..") end • Tedious, error prone, virotic, not finalized.

24. LTN13 exceptions • try = newtry(finalizer): factory; • On success, try returns all arguments; • On failure, throws the second argument; • Calls finalizer before raising the exception. • foo = protect(bar): factory; • foo executes bar in a protected environment; • Returns nil followed by any thrown error.

25. No 'if' statements function metat.__index:greet(domain) self.try(self.tp:check("2..")) self.try(self.tp:command("HELO", domain)) return self.try(self.tp:check("2..")) end • Internal functions throw exceptions; • try calls tp.close() on error; • External functions can be protected.

26. Conclusions • Hope you like our API, we do; • It is easy to implement; • Function factories + clusures, coroutines • It is fast; • Time critical in C, management in Lua; • Questions?