An OCaml-based Network Services Platform Chris Waterson waterson@liveops

1. An OCaml-based Network Services PlatformChris Watersonwaterson@liveops.com October 4, 2007

2. Motivation • Highly-interactive self-scheduling system • Workers use site to book work throughout the week • Non-trivial application logic • Extreme peaks • First generation system was LAMP-based and stateless • M=MySQL, P=Perl • Each interaction generated an HTTPS request • Client polled for updates • Issues with scaling led to a session-based redesign • Client maintains persistent SSL connection with server • No need to re-authorize or reconstruct state to handle client requests • Two-way communication allows events to be propagated to client • But...how do we scale to millions of users? (== millions of SSL connections)

3. Designing a Scalable Server • Conflicting requirements • Manageable programming model • Scalable implementation • Some Alternatives • "Naive" threading (one thread per session) over blocking I/O. • This is the model the application programmer wants • Conventional wisdom says: hard to scale past several hundred threads • Arbitrary pre-emption requires synchronization for shared structures • Event-based I/O with worker thread pooling • Parallelism is still limited by the number of worker threads • You still have synchronization issues • Event-based I/O with a single thread & explicit state management • Hand-coded state machines are difficult to maintain over time

4. Event-based Model, Redux • Start with this: let rec echo fd = let str = readln fd in writeln fd str ; echo () • Transform into continuation passing style • Each I/O operation takes a continuation routine that will be called when the I/O completes let rec echo fd sk = readln fd s (fun str s -> writeln fd str s (fun _ s -> echo fd sk)) • The underlying I/O state is maintained in the s argument • If readln blocks, it can store k and use s to resume another I/O operation • We've encapsulated our application's "state machine" in the lexical closure • Transform into monadic style • Define bind and sequence operators to combine functions exactly as above let echo fd = readln fd >>= writeln fd >> echo fd

5. Mcom • Monadic Communication Library • Continuation passing to encapsulate session state • An additional argument to maintain underlying I/O state • Layered monads to make it manageable • Provides basic I/O operations (read, write, connect, accept, ...) • Provides basic concurrency operations (fork, wait, notify) • Advantages • "Feels" like naively threaded code over blocking I/O • Runs over scalable I/O primitives (kqueue, epoll) • Pre-emption occurs at well-known points • OCaml • First-class functions to support continuations • Function currying to support monadic style • Production-quality compiler and runtime

6. Results • Scales well • Two Xeon-class dual core servers • Supports 5,000 simultaneous users connecting via SSL • Sustains 700+ TPS (with bursts of 1,500 TPS) during peaks • Two major feature releases since initial deployment (mid-2006) • Issues • Requires non-blocking protocol implementations • Monadic style doesn't mix well with exceptions • Conclusion • FP techniques allowed us to have our cake...and eat it, too! • Hope to release as open source soon