HTTP-Tester- A web appliance testing tool

1. HTTP-Tester- A web appliance testing tool Kiran Komaravolu Dept of Computer Science and Engg Lehigh University

2. What is the need to test Web Appliances • Need to use best components. • Compare performance of different web devices. • Verify if a device is performs as per specifications. • Choosing between different service providers.

3. Related Work • The Ethereal network analyser (www.ethereal.com). • TCPDUMP/PCAP (www.tcpdump.org). • Httpflow (Grevnin and Davison) • Simultaneous proxy evaluation. (Davison). • Medusa Proxy. (Koletsou and Voelker).

4. Http-Tester • Replays captured requests to an existing server to one that is being evaluated. • No active deployment needed. • Does not affect existing services. • Can be used to evaluate any Web appliance running HTTP.

5. HTTP-Tester • Topology setup

6. HTTP-Tester • Components • The Sniffer. • Logs all packets onto a packet database. • Request Replicator. • Replays all requests to the “tested device” • The Post Processor. • Reads the packet logs, extracts HTTP requests and responses and correlates them.

7. HTTP • HTTP Message Structure • A command line. (GET, POST, OK etc.) • GET http://www.google.com / HTTP 1.0 • Zero or more header lines. • Content-Type: text/html; charset=iso-8859-1 • Control Return + Line Feed. • Optional Message Body • Response message

8. HTTP Transactions. • HTTP 1.0 • No persistent connections. • A new connection is opened up for every request made, and is closed down after response sent. • HTTP 1.1 • Supports Persistent Connections. • Client may make a series of requests on a connection. Server returns responses in same order. • Supports chunk encoding.

9. The Sniffer • Blindly captures all relevant network packets and logs them. • The pcap packet capture library has been used to read packets off the wire. • Berkeley DB package has been used to log the packets into a ‘db’ database. • The job of efficient memory and disk usage has been delegated to DB.

10. Request Replicator • Replays all requests flowing into original device onto the tested device. • Works in parallel with the sniffer module. • Looks for TCP SYN packets and determines the beginning of a request flow. And opens up a new connection to the second device. • The payload every next packet sent to on this connection is sent to the second device.

11. Request Replicator. • This way the second server sees the same workload as the first one. • The replies sent by the server are ignored. They are flushed out as the socket buffers overflow. The sniffer module though logs the TCP packets.

12. Post Processing. • Extracts HTTP requests/responses from the packet logs. • Every request is mapped onto its response. The same request response pair on the second connection is also identified. • Httpflow routines used to extract HTTP requests from packet stream.

13. Identifying Requests. • We know which way requests flow. • Assumption: First TCP packet of every request has one of GET, HEAD,OPTIONS, DELETE, POST, PUT. • End of headers is determined by a CRLF. • content-length field is specified in case of PUT/POST requests.

14. Identifying responses • Response headers extrated by looking for a CRLF. • End of Response message determined in three ways: • Server sends a FIN or RST packet. • Response headers contain a “content-length” field. • Response is chunk encoded. Each chunk has its own length. Zero length chunk indicates end of stream.

15. Correlating requests and responses • HTTP 1.0