The TIME-WAIT state in TCP and Its Effect on Busy Servers

1. The TIME-WAIT state in TCP and Its Effect on Busy Servers By Theodore Faber, Joe Touch & Wei Yue Ed Neto 18-845 Internet Services Tuesday, February 27, 3001

2. The TIME-WAIT state • Prevents delayed packets from being accepted by a later connection • TCB kept in memory until 2MSL timeout, then state moves to CLOSED • MSL ~ 30s. – 2min.

3. The TIME-WAIT state

4. TIME-WAIT Loading • Protocols such as HTTP and FTP specify server for the active close • Busy server may be handling thousands of requests • Each connection in TIME-WAIT state has a TCB in memory • Thousands of connections in TIME-WAIT consume memory, slow down active connections, lowering throughput

5. TIME-WAIT Loading • Obvious solution: Shift the TCB overhead to the client, by having it perform the active close instead • Not so fast: TCP close is used as an end-of-transaction marker. • How will client know when to close? Server could send transaction length explicitly in the beginning of the connection. But what about dynamic content?

6. TIME-WAIT Negotiation • New TCP option: TW-Negotiate • Client adds TW-Negotiate to <SYN> packet, data is the IP address of TCB holder • Server returns <SYN,ACK> with TW-Negotiate set to its choice (if supported) • Client acks <SYN,ACK> if server choice is accepted, sends <RST> otherwise • If no TW-Negotiate option, use normal TCP semantics

7. TIME-WAIT Negotiation • The Good: • More control over endpoint resources • More general solution than reversing active & passive roles or modifying upper layers • The Bad: • Significant change to very well established protocol (see RFC-1263 for reasons why) [RFC-1263] O’Malley, S. and L. Peterson, “TCP Extensions Considered Harmful”, University of Arizona, October 1991 RFC Fun: http://www.rfc-editor.org/rfcsearch.html

8. Transport Level (TCP) Solution • Exchange TW state when connection is closed • Modify client to send <RST> after passive close and put itself into TW state • Server removes TCB when it receives <RST> • Only works on implementations that accept <RST> while in TIME-WAIT • Susceptible to TIME-WAIT Assassination • [RFC 1337] R. Braden, “TIME-WAIT Assassination Hazards in TCP”, USC/Information Sciences Institute, May 1992. • Won’t work with RFC1337 “short-term” fix which ignores <RST> while in TIME-WAIT

9. Transport Level (TCP) Solution

10. Transport Level (TCP) Solution • The Good: • Not necessary to know transaction length or establish anything during <SYN> • Only clients/proxies need to change • The Bad: • <RST> isn’t being used the way it was intented • Changes to TCP state machine • Changes to OS to support flag for close_swap()

11. Application Level Solution for HTTP • New HTTP command: CLIENT_CLOSE • No reply required – client does active close • Modified client - compatible with HTTP 1.1 • May allow server to request that client performs active close

12. Application Level Solution for HTTP • The Good: • No changes to TCP stack needed • Server doesn’t necessarily need to understand CLIENT_CLOSE • The Bad: • Clients need to be modified • Persistent connection always necessary? • What about other App. Level protocols?

13. Performance Improvements • Unmodified TCP throughput cut in half under heavy load. Modified TCP throughput constant • WebSTONE • TCP Solution: ~25% faster • HTTP Solution: ~50% faster • Unmodified fails under very high connection rates – run out of memory

14. Misc. Issues • How are newer OS’s dealing with this? SunOS 4.1.3 is ancient…. • Are newer OS’s using linear search for demultiplexing? • TCP Solution: What if I send a <RST> to mean “connection messed up”…then both sides don’t go into TIME-WAIT • HTTP Solution: Would all transactions have to occur as persistent connections? • TW-Negotiate: Any better alternatives to using the higher IP address for conflict resolution?