Building HTTP clients in PHP

1. Building HTTP clients in PHP

2. A PHP package for sending HTTP requests and getting responses • A PHP package for handling HTTP requests/responses is available • It is called HTTP_Request and is part of the PEAR repository of PHP extensions and applications -- see http://pear.php.net/ • The HTTP_Request documentation is here: http://pear.php.net/manual/en/package.http.http-request.php • PEAR uses the object-oriented programming paradigm which is supported by PHP • Before looking at HTTP_Request, we will review a few details of OOP in PHP

3. OOP in PHP (contd.) • The object model in PHP was rewritten for PHP 5 • The PHP manual contains two main section on OOP: • Chapter 18. Classes and Objects (PHP 4) • Chapter 19. Classes and Objects (PHP 5) • A full treatment of OOP in PHP is beyond our scope here

4. Object-oriented programming in PHP • Example class and its usage <?php class widget { var $x; function say_hello() {echo "Hello, World!";} function set_x($value) {$this->x = $value;} function get_x() {return $this->x;} } $thing1 =& new widget; $thing1->say_hello(); $thing1->set_x(98); $y = $thing1->get_x(); echo "<br>y is $y"; ?>

5. Example HTTP client program in PHP <?php require_once "HTTP/Request.php"; $req = & new HTTP_Request('http://www.rte.ie/'); if (!PEAR::isError($req->sendRequest())) { $contents= $req->getResponseBody(); echo $contents; } ?> • Note we have not handled fact that some URLs on RTE site are relative • Thus, output (top image) not quite right; should be as in bottom inage

6. HTTP_Request package • We will now examine details of the HTTP_Request package • Information on the package is here: http://pear.php.net/package/HTTP_Request/docs/ • Some end-user documentation is available here: http://pear.php.net/manual/en/package.http.http-request.php

7. HTTP_Request methods • Current description of the HTTP_Request class is here: http://pear.php.net/package/HTTP_Request/docs/latest/apidoc/HTTP_Request-1.2.4/HTTP_Request.html (This may become outdated as new versions of the class are produced) • The list of methods provided by the class appears to be HTTP_Request setMethod setURL setHttpVer addHeader addCookie removeHeader addQueryString addRawQueryString addPostData addRawPostData addFile setBasicAuth setProxy sendRequest getResponseCode getResponseHeader getResponseCookies getResponseBody

8. HTTP_Request (contd.) • The set of methods provided by HTTP_Request corresponds to the structure of requests and responses defined in the HTTP protocol

9. Structure of HTTP requests/responses • Requests: a three-part structure request line (Method URL[+query] HTTP-version) headers body • Responses: also a three-part structure status line (HTTP-version NatLangPhrase Response-Code) headers body

10. Structure of HTTP requests and corresponding HTTP_Request methods • Request structure request line (Method URL[+query] HTTP-version) headers body • HTTP_Request methods • Methods for manipulating the Request Line setMethod setURL setHttpVer addQueryString addRawQueryString • Methods for manipulating the Headers addHeader addCookie removeHeader • Methods for manipulating the body addPostData addRawPostData addFile

11. cs 4408 got here on 1 nov 2005/

12. Structure of HTTP responses and corresponding HTTP_Request methods • Response structure status line (HTTP-version NatLangPhrase Response-Code) headers body • HTTP_Request methods • Methods for reading the status line getResponseCode • Methods for reading the Headers getResponseHeader getResponseCookies • Methods for reading the body getResponseBody

13. Creating a request object • Method for creating a request object HTTP_Request Format of call: HTTP_Request HTTP_Request( $url $url [,array() $params]) • Params is asssociative array which can include: • method - Method to use, GET, POST etc • http - HTTP Version to use, 1.0 or 1.1 • user - Basic Auth username • pass - Basic Auth password • proxy_host - Proxy server host • proxy_port - Proxy server port • proxy_user - Proxy auth username • proxy_pass - Proxy auth password • timeout - Connection timeout in seconds.

14. Creating a request object (contd.) • Example usages of method for creating a request object $req =& new HTTP_Request(); $req =& new HTTP_Request("http://www.rte.ie/"); $req =& new HTTP_Request("http://www.rte.ie/", array(method=>'GET',http=>'1.1')); • In the first example, no aspect of the new request is specified; they can all be specified later, using other methods, before the request is sent to the server • In the second example, the URL is specified; other aspects can be specified later, using other methods • In the third example, the method, the URL and the HTTP version are specified; other aspects can be specified later, using other methods

15. Sending a request object • Method for sending a request object sendRequest Format of call: mixed sendRequest() • Method returns: • a PEAR error message if there is an error, • true otherwise • Example usage of method for creating a request object • Example 1: $request =& new HTTP_Request("http://www.rte.ie/"); $request->sendRequest();

16. Sending a request object (contd.) • HTTP_Request is an object class which extends a base class called the PEAR base class • The PEAR base class contains a range of methods -- see Chapter 26 of the PEAR manual, which is available at http://pear.php.net/manual/index.php • One of these is the isError method which checks for a PEAR_Error object • Format of call boolean PEAR::isError (mixed $data [, mixed $msgcode]) • It can be used to see whether an error is produced when a HTTP request is sent • Example 2 $request =& new HTTP_Request("http://www.rte.ie/"); if ( ! PEAR::isError( $request->sendRequest()) ) { ......... }

17. Revisiting the example client program in PHP <?php require_once "HTTP/Request.php"; $req = & new HTTP_Request('http://www.rte.ie/'); if (!PEAR::isError($req->sendRequest())) { $contents= $req->getResponseBody(); echo $contents; } ?> • Note we have not handled fact that some URLs on RTE site are relative • Thus, output not quite right -- some image URLs are not quite right

18. Revisiting, again, the example client program <?php require_once "HTTP/Request.php"; $req = & new HTTP_Request('http://www.rte.ie/'); if (!PEAR::isError($req->sendRequest())) { $contents= $req->getResponseBody(); $contents= str_replace('src="/', 'src="http://www.rte.ie/', $contents); echo $contents; } ?> • We have handled some of the relative URLs on RTE site • But, output still not quite right

19. Revisiting, yet again, the example client program <?php require_once "HTTP/Request.php"; $req = & new HTTP_Request('http://www.rte.ie/'); if (!PEAR::isError($req->sendRequest())) { $contents= $req->getResponseBody(); $contents= str_replace('src="/', 'src="http://www.rte.ie/', $contents); $regexp = '%src="(?!http://)%'; $contents= preg_replace($regexp, 'src="http://www.rte.ie/', $contents); echo $contents; } ?>

20. A utility which shows all data from incoming requests is here: http://www.cs.ucc.ie/j.bowen/cs4408/resources/showRequest.php • It is defined as follows: <?php echo "<strong>SERVER variables:</strong><br>"; foreach ($_SERVER as $name => $value) { echo "$name = $value <br>"; } echo "<strong>GET variables:</strong><br>"; foreach ($_GET as $name => $value) { echo "$name = $value <br>"; } echo "<strong>POST variables:</strong><br>"; foreach ($_POST as $name => $value) { echo "$name = $value <br>"; } echo "<strong>COOKIE variables:</strong><br>"; foreach ($_COOKIE as $name => $value) { echo "$name = $value <br>"; } ?>

21. When called by a MSIE browser

22. Consider this "browser" calling the same showRequest.php • It is a PHP program at http://cosmos.ucc.ie/~jabowen/cs4408/myBrowser.php • It is implemented as follows: <?php require_once "HTTP/Request.php"; $req = & new HTTP_Request("http://www.cs.ucc.ie/j.bowen/cs4408/resources/showRequest.php"); if (!PEAR::isError($req->sendRequest())) { $contents= $req->getResponseBody(); echo $contents; } ?>

23. Output from showRequest.php when called by "browser"

24. Sending headers

25. A modified "browser" • It is a PHP program at http://cosmos.ucc.ie/~jabowen/cs4408/myBrowser2.php • It is implemented as follows: <?php require_once "HTTP/Request.php"; $req = & new HTTP_Request("http://www.cs.ucc.ie/j.bowen/cs4408/resources/showRequest.php"); $req->addHeader('ACCEPT-LANGUAGE','en-ie'); if (!PEAR::isError($req->sendRequest())) { $contents= $req->getResponseBody(); echo $contents; } ?>

26. Output from showRequest.php when called by "browser"ACCEPT_LANGUAGE header now appears

27. Handling cookies

28. Modified utility to show all data from incoming requests: http://www.cs.ucc.ie/j.bowen/cs4408/resources/showRequest2.php <?php ob_start(); setcookie('dummyCookie','baconAndEggs'); ob_end_flush(); echo "<strong>SERVER variables:</strong><br>"; foreach ($_SERVER as $name => $value) { echo "$name = $value <br>"; } echo "<strong>GET variables:</strong><br>"; foreach ($_GET as $name => $value) { echo "$name = $value <br>"; } echo "<strong>POST variables:</strong><br>"; foreach ($_POST as $name => $value) { echo "$name = $value <br>"; } echo "<strong>COOKIE variables:</strong><br>"; foreach ($_COOKIE as $name => $value) { echo "$name = $value <br>"; } ?>

29. In first call by a MSIE browser, no cookie comes in request

30. In second call by a MSIE browser, a cookie is in request

31. A "browser" to read showRequest2.php • It is a PHP program at http://cosmos.ucc.ie/~jabowen/cs4408/myBrowser3.php • It is implemented as follows: <?php require_once "HTTP/Request.php"; $req = & new HTTP_Request("http://www.cs.ucc.ie/j.bowen/cs4408/resources/showRequest2.php"); $req->addHeader('ACCEPT-LANGUAGE','en-ie'); if (!PEAR::isError($req->sendRequest())) { $contents= $req->getResponseBody(); echo $contents; } ?>

32. cs 4408 got here on 4 nov 2005

33. In first call, no cookie comes in request

34. No cookie comes in 2nd call either

35. Why no cookie in 2nd request? • We know that showRequest2.php sent a cookie to our "browser" when it sent a response to the first request • Why did that cookie not come back in the 2nd request? • Because our "browser" does not maintain a "cookie jar" and never sends cookies

36. Cookie "jars" • A cookie "jar" is the mechanism used by a HTTP client to store cookies it receives from servers

37. In MSIE, the cookie jar is a directory on the hard-disk of the client machine; each cookie is file

38. In Netscape 8.0, the cookie jar appears to be a single file managed by an interface which allows you to see cookie properties

39. In Firefox, the cookie jar appears to be a single file managed by an interface which allows you to see cookie properties

40. Detecting cookies • Before we implement a cookie jar in our web client, let's first make our "browser" report cookies that it receives from the server-side program

41. A "browser" to read showRequest2.php and which can show cookies and other headers that are sent by the server • At http://cosmos.ucc.ie/~jabowen/cs4408/myBrowser4.php <?php require_once "HTTP/Request.php"; $req = & new HTTP_Request("http://www.cs.ucc.ie/j.bowen/cs4408/resources/showRequest2.php"); if (!PEAR::isError($req->sendRequest())) { $headers = $req->getResponseHeader(); echo "<br><strong style='color:red'>Headers</strong>"; foreach ($headers as $name => $value) { echo "<br> $name = $value"; } echo "<br><strong style='color:red'>Cookies</strong><br>"; $cookies = $req->getResponseCookies(); foreach ($cookies as $fields) { foreach ($fields as $name => $value) { echo "$name = $value; "; } echo "<br>"; } $contents= $req->getResponseBody(); echo "<br><strong style='color:red'>Body</strong><br>"; echo $contents; } ?>

43. Adding a cookie jar to our web client • We could use a simple text file as our cookie jar • Initially, let's have a simple jar, which stores only the names and values of cookies, without storing such attributes a expiry date, web-address, etc • Let's call the file cookieJar

44. A "browser" to read showRequest2.php and which remembers and returns a cookie sent by the server • At http://cosmos.ucc.ie/~jabowen/cs4408/myBrowser5.php <?php require_once "HTTP/Request.php"; $req = & new HTTP_Request("http://www.cs.ucc.ie/j.bowen/cs4408/resources/showRequest2.php"); $fp=fopen("tmp/cookieJar","r"); $cookie = fread($fp,100); fclose($fp); if ($cookie) { $pieces=explode("=",$cookie); $req->addCookie($pieces[0],$pieces[1]); } if (!PEAR::isError($req->sendRequest())) { echo "<br><strong style='color:red'>Non-cookie Headers received by Browser</strong>"; $headers = $req->getResponseHeader(); foreach ($headers as $name => $value) { echo "<br> $name = $value"; } echo "<br><strong style='color:red'>Cookies received by browser</strong><br>"; $cookies = $req->getResponseCookies(); $fp=fopen("tmp/cookieJar","w"); foreach ($cookies as $fields) { foreach ($fields as $name => $value ) { echo "$name = $value; "; } echo "<br>"; fwrite($fp,"$name=$value"); } fclose($fp); $contents= $req->getResponseBody(); echo "<br><strong style='color:red'>Message Body rec'd by Browser</strong><br>"; echo $contents; } ?>

45. Before 1st call, cookieJar is empty; so, no cookie is sent

46. However, after the request the cookieJar is not empty, as we can see in the vi session below

47. Since cookieJar is no longer empty, a cookie is sent the next time we run our "browser"; its arrival is reported by server-side program