Server Design

1. Server Design

2. Roadmap • Design goals • Server design • Java NIO • Reactor Pattern

3. Design Goals • Meet availability and performance goals (response time, throughput … etc) • Scalability: • Graceful degradation under increasing load (more clients) • Continuous improvement with increasing resources (CPU, memory, disk, bandwidth)

4. Server Design Spectrum Server Design Concurrent Iterative + Simple to implement. • Low performance • Not scalable • Harder to implement (manage concurrency, data sharing) + Higher performance + Better scalability (more efficient use of resources and ability to handle more clients)

5. Server Design Spectrum Server Design Concurrent Iterative Processes Threads + More secure. (separation of memory space) + More fault tolerant (a crash of one process often does not impact other processes) - Higher overhead • Less secure. • Less reliable ( a crash in a single thread may lead to a complete server crash) + Lower overhead + Easier to program.

6. Server Design Spectrum Server Design Concurrent Iterative Processes Threads Thread-per-request (blocking IO) Event based (Non blocking IO)

7. Server Design Spectrum – Thread-per-request Figures source: “SEDA: An Architecture for Well-Conditioned, Scalable Internet Services”, Matt Welsh, et. al. SOSP 2001.

8. Server Design Spectrum – Event Based Figures source: “SEDA: An Architecture for Well-Conditioned, Scalable Internet Services”, Matt Welsh, et. al. SOSP 2001.

9. Server Design Spectrum Server Design Concurrent Iterative Processes Threads Thread-per-request (blocking IO) Event based (Non blocking IO) + Easier to implement. • Higher overhead • Less scalable • Harder to implement (requires maintaining communication state machine and implement handlers for set of events) + Less overhead (requires few threads, more efficient use of resources) + More scalable.

10. Roadmap • Design goals • Server design • Java NIO • Reactor Pattern

11. Java Non-Blocking IO Server Side Figure from O’Reilly OnJava.com

12. Java NIO support • Channels • Connections to files, sockets etc that support non-blocking operations (read, write) • Buffers • Array-like objects that can be directly read or written by Channels • Selectors • Tell which of a set of Channels have IO events • SelectionKeys • Maintain IO event status and bindings

13. Java NIO Program Structure • Create Selector and one or more Channels • Register Channels to a Selector • Mark the events of interest to detect in each channel. • Loop: • Call to the selectoperation of Selector. • Returns a number of the keys whose ready set has changed. (each key corresponds to a channel) • Using the selected keys: • Check ready operations for each key. • Remove the key from the selected keys set. • That clears the ready operations set of the key. • Do appropriate processing.

14. Java NIO Program Structure // Create the server socket channel ServerSocketChannel server = ServerSocketChannel.open(); // nonblocking I/O server.configureBlocking(false); // host-port 8000 server.socket().bind(new java.net.InetSocketAddress(host,8000)); // Create the selector Selector selector = Selector.open(); // Recording server to selector (type OP_ACCEPT) server.register(selector, SelectionKey.OP_ACCEPT);

15. Java NIO Program Structure for(;;) { // Waiting for events selector.select(timeout); // Get the keys Set keys = selector.selectedKeys(); Iterator i = keys.iterator(); // For each keys... while(i.hasNext()) { SelectionKey key = (SelectionKey) i.next(); // Remove the current key i.remove();

16. Java NIO Program Structure // if isAccetable == true - then a client required a connection if (key.isAcceptable()) { // get client socket channel SocketChannel client = server.accept(); // Non Blocking I/O client.configureBlocking(false); // recording to the selector (reading) client.register(selector, SelectionKey.OP_READ); continue; } // if isReadable = true - then the server is ready to read if (key.isReadable()) { SocketChannel client = (SocketChannel) key.channel(); try { client.read(buffer); } catch (Exception e) {… }

17. Java NIO - Buffer • Buffer a container for a fixed amount of data. • Similar to a byte [] array • But encapsulated such that internal storage can be a block of system memory. • Direct memory mapping • Zero-copy read/receive  high performance communication

18. Java NIO – Non blocking operations • socket.configureBlocking(false) ; • A read() operation transfers data that is immediately available. If no data immediately available, returns 0. • If data cannot be immediately written, a write() will immediately return 0. • For a server socket, if no client is currently trying to connect, the accept() method immediately returns null. • connect() is more complicated • generally connections would always block for some interval waiting for the server to respond. • In non-blocking mode connect() generally returns false. But the negotiation with the server is nevertheless started. The finishConnect() method on the same socket should be called later. It also returns immediately. Repeat until it return true.

19. Java NIO – Key Attachments • Problem: when it.next() returns a key, there is no convenient way of getting information about the context in which the associated channel was registered with the selector. • Solution: specify an arbitrary object as an attachment to the channel when you register it; • Later when you get the key from the selected set, you can extract the attachment, and use its content in to decide what to do.

20. Java NIO – Key Attachments Example channel1.register (selector, SelectionKey.OP_READ, new Integer(1) ) ; // attachment … channel3.register (selector, SelectionKey.OP_READ | SelectionKey.OP_WRITE, new Integer(3) ) ; // attachment … while(true) { … Iterator it = selector.selectedKeys().iterator() ; … SelectionKey key = it.next() ; if( key.isReadable() ) switch( ((Integer) key.attachment() ).value() ) { case 1 : … action appropriate to channel1 … case 3 : … action appropriate to channel3 … } }

21. Java NIO Tutorial Introducing Non blocking IO http://www.onjava.com/pub/a/onjava/2002/09/04/nio.html?page=1 A good overview (including NIO.2) http://chamibuddhika.wordpress.com/2012/08/11/io-demystified/

22. Roadmap • Design goals • Server design • Java NIO • Reactor Pattern

24. Reactor Design Pattern

25. Reactor Design Pattern • Handle • Receives events; E.g. a network connection, timer, user interface device • Synchronous Event Demultiplexer • select() waits until an event is received on a Handle and returns the event. • Often implemented as part of an operating system. • Initiation Dispatcher • Uses the Synchronous Event Demultiplexer to wait for events. • Dispatches events to the Event Handlers. • Event Handler • Application-specific event processing code.

26. Reactor Design Pattern Dynamics • Setup • Create Initiation Dispatcher. • Register Event Handlers with Initiation Dispatcher. • Main loop • Call handleEvents in Initiation Dispatcher repeatedly. • Initiation Dispatcher calls select in Synchronous Event Demultiplexer, blocking until an event is received. • The Initiation Dispatcher calls handleEvent in the corresponding Event Handler, passing it the event. • End • Unregister Event Handlers from Initiation Dispatcher.

27. Thank you