Reliable Distributed Systems

1. Reliable Distributed Systems RPC and Client-Server Computing

2. Remote Procedure Call • Basic concepts • Implementation issues, usual optimizations • Where are the costs? • Reliability and consistency • Multithreading debate

3. A brief history of RPC • Introduced by Birrell and Nelson in 1985 • Pre-RPC: Most applications were built directly over the Internet primitives • Their idea: mask distributed computing system using a “transparent” abstraction • Looks like normal procedure call • Hides all aspects of distributed interaction • Supports an easy programming model • Today, RPC is the core of many distributed systems

4. More history • Early focus was on RPC “environments” • Culminated in DCE (Distributed Computing Environment), standardizes many aspects of RPC • Then emphasis shifted to performance, many systems improved by a factor of 10 to 20 • Today, RPC often used from object-oriented systems employing CORBA or COM standards. Reliability issues are more evident than in the past.

5. The basic RPC protocol client server “binds” to server registers with name service

6. The basic RPC protocol client server “binds” to server prepares, sends request registers with name service receives request

7. The basic RPC protocol client server “binds” to server prepares, sends request registers with name service receives requestinvokes handler

8. The basic RPC protocol client server “binds” to server prepares, sends request registers with name service receives requestinvokes handlersends reply

9. The basic RPC protocol client server “binds” to server prepares, sends request unpacks reply registers with name service receives requestinvokes handlersends reply

10. Compilation stage • Server defines and “exports” a header file giving interfaces it supports and arguments expected. Uses “interface definition language” (IDL) • Client includes this information • Client invokes server procedures through “stubs” • provides interface identical to the server version • responsible for building the messages and interpreting the reply messages • passes arguments by value, never by reference • may limit total size of arguments, in bytes

11. Binding stage • Occurs when client and server program first start execution • Server registers its network address with name directory, perhaps with other information • Client scans directory to find appropriate server • Depending on how RPC protocol is implemented, may make a “connection” to the server, but this is not mandatory

12. Data in messages • We say that data is “marshalled” into a message and “demarshalled” from it • Representation needs to deal with byte ordering issues (big-endian versus little endian), strings (some CPUs require padding), alignment, etc • Goal is to be as fast as possible on the most common architectures, yet must also be very general

13. Request marshalling • Client builds a message containing arguments, indicates what procedure to invoke • Due to the need for generality, data representation a potentially costly issue! • Performs a send I/O operation to send the message • Performs a receive I/O operation to accept the reply • Unpacks the reply from the reply message • Returns result to the client program

14. Costs in basic protocol? • Allocation and marshalling data into message (can reduce costs if you are certain client, server have identical data representations) • Two system calls, one to send, one to receive, hence context switching • Much copying all through the O/S: application to UDP, UDP to IP, IP to ethernet interface, and back up to application

15. Schroeder and Burroughs • Studied RPC performance in O/S kernel • Suggested a series of major optimizations • Resulted in performance improvments of about 10-fold for Xerox firefly workstation (from 10ms to below 1ms)

16. Typical optimizations? • Compile the stub “inline” to put arguments directly into message • Two versions of stub; if (at bind time) sender and dest. found to have same data representations, use host-specific rep. • Use a special “send, then receive” system call (requires O/S extension) • Optimize the O/S kernel path itself to eliminate copying – treat RPC as the most important task the kernel will do

17. Fancy argument passing • RPC is transparent for simple calls with a small amount of data passed • “Transparent” in the sense that the interface to the procedure is unchanged • But exceptions thrown will include new exceptions associated with network • What about complex structures, pointers, big arrays? These will be very costly, and perhaps impractical to pass as arguments • Most implementations limit size, types of RPC arguments. Very general systems less limited but much more costly.

18. Overcoming lost packets client server sends request

19. Overcoming lost packets client server sends request Timeout! retransmit duplicate request: ignored ack for request

20. Overcoming lost packets client server sends request Timeout! retransmit ack for request reply

21. Overcoming lost packets client server sends request Timeout! retransmit ack for request reply ack for reply

22. Costs in fault-tolerant version? • Acks are expensive. Try and avoid them, e.g. if the reply will be sent quickly supress the initial ack • Retransmission is costly. Try and tune the delay to be “optimal” • For big messages, send packets in bursts and ack a burst at a time, not one by one

23. Big packets client server sends request as a burst ack entire burst reply ack for reply

24. RPC “semantics” • At most once: request is processed 0 or 1 times • Exactly once: request is always processed 1 time • At least once: request processed 1 or more times ... but exactly once is impossible because we can’t distinguish packet loss from true failures! In both cases, RPC protocol simply times out.

25. Implementing at most/least once • Use a timer (clock) value and a unique id, plus sender address • Server remembers recent id’s and replies with same data if a request is repeated • Also uses id to identify duplicates and reject them • Very old requests detected and ignored by checking time • Assumes that the clocks are working • In particular, requires “synchronized” clocks

26. RPC versus local procedure call • Restrictions on argument sizes and types • New error cases: • Bind operation failed • Request timed out • Argument “too large” can occur if, e.g., a table grows • Costs may be very high • ... so RPC is actually not very transparent!

27. RPC costs in case of local destination process • Often, the destination is right on the caller’s machine! • Caller builds message • Issues send system call, blocks, context switch • Message copied into kernel, then out to dest. • Dest is blocked... wake it up, context switch • Dest computes result • Entire sequence repeated in reverse direction • If scheduler is a process, context switch 6 times!

28. RPC example Dest on same site O/S Source does xyz(a, b, c)

29. RPC in normal case Destination and O/S are blocked Dest on same site O/S Source does xyz(a, b, c)

30. RPC in normal case Source, dest both block. O/S runs its scheduler, copies message from source out-queue to dest in-queue Dest on same site O/S Source does xyz(a, b, c)

31. RPC in normal case Dest runs, copies in message Dest on same site O/S Source does xyz(a, b, c) Same sequence needed to return results

32. Broad comments on RPC • RPC is not very transparent • Failure handling is not evident at all: if an RPC times out, what should the developer do? • Reissuing the request only makes sense if there is another server available • Anyhow, what if the request was finished but the reply was lost? Do it twice? Try to duplicate the lost reply? • Performance work is producing enormous gains: from the old 75ms RPC to RPC over U/Net with a 75usec round-trip time: a factor of 1000!

33. Contents of an RPC environment • Standards for data representation • Stub compilers, IDL databases • Services to manage server directory, clock synchronization • Tools for visualizing system state and managing servers and applications

34. Closely Related Topic • Multithreading is a common performance-enhancing technique • Idea is that server is often idle while doing I/O for one client, so use extra threads to allow concurrent request processing • In the limit, leads to database transactional concurrency model, but many non-transactional servers use threads for enhanced performance

35. Multithreading debate • Three major options: • Single-threaded server: only does one thing at a time, uses send/recv system calls and blocks while waiting • Multi-threaded server: internally concurrent, each request spawns a new thread to handle it • Upcalls: event dispatch loop does a procedure call for each incoming event, like for X11 or PC’s running Windows.

36. Single threading: drawbacks • Applications can deadlock if a request cycle forms: I’m waiting for you and you send me a request, which I can’t handle • Much of system may be idle waiting for replies to pending requests • Harder to implement RPC protocol itself (need to use a timer interrupt to trigger acks, retransmission, which is awkward)

37. Multithreading • Idea is to support internal concurrency as if each process was really multiple processes that share one address space • Thread scheduler uses timer interrupts and context switching to mimic a physical multiprocessor using the smaller number of CPU’s actually available

38. Multithreaded RPC • Each incoming request is handled by spawning a new thread • Designer must implement appropriate mutual exclusion to guard against “race conditions” and other concurrency problems • Ideally, server is more active because it can process new requests while waiting for its own RPC’s to complete on other pending requests

39. Negatives to multithreading • Users may have little experience with concurrency and will then make mistakes • Concurrency bugs are very hard to find due to non-reproducible scheduling orders • Reentrancy can come as an undesired surprise • Threads need stacks hence consumption of memory can be very high • Deadlock remains a risk, now associated with concurrency control • Stacks for threads must be finite and can overflow, corrupting the address space

40. Threads: can spawn too many SCHED event

41. Threads: can spawn too many Thread spawned, but blocks SCHED event

42. Threads: can spawn too many SCHED Eventually, application becomes bloated, begins to thrash. Performance drops and clients may think the server has failed event

43. Upcall model • Common in windowing systems • Each incoming “event” is encoded as a small descriptive data structure • User registers event handling procedures • Dispatch loop calls the procedures as new events arrive, waits for the call to finish, then dispatches a new event

44. Upcalls combined with threads • Perhaps the best model for RPC programming • Each handler can be tagged: needs thread, or can be executed “unthreaded” • Developer must still be very careful where threads are used

45. Recent RPC history • RPC was once touted as the transparent answer to distributed computing • Today the protocol is very widely used • ... but it isn’t very transparent, and reliability issues can be a major problem • Today the strongest interest is in Web Services and CORBA, which use RPC as the mechanism to implement object invocation