Chapter 7 Web Usage Mining Part I

1. Chapter 7Web Usage MiningPart I L. Malak Bagais

2. Web Usage Mining It’s main goal is to: Discover usage patterns from web data in order to understand and better serve the needs of web based applications

3. Web Usage Mining Web usage mining consists of three phases Preprocessing Pattern discovery Pattern analysis

4. Web-Usage Mining Generated by users’ interaction with the Web, data sources include: • web-server access logs • proxy-server logs • browser logs • user profiles • registration data • user sessions and transactions • cookies • user queries • bookmark data • mouse clicks and scrolls

5. Web-Log Processing A server log: • set of files consisting of the details of an activity performed by a server • files are automatically created and maintained by the server • The World Wide Web Consortium (W3C) has specified a standard format for web-server log files • There are other proprietary formats for web-server logs.

6. Web-Log Processing Most web logs contain: • IP address of the client making the request • date and time of the request • URL of the requested page • number of bytes sent to serve the request • user agent (such as a web browser or web crawler) • referrer (the URL that triggered the request) • Logs can all be stored in one file • A better alternative is to separate: • access log • error log • referrer log

7. Format of Web Logs Common log format (http://www. W3.org/Daemon/User/Config/Logging.html#common-logfile-format)

8. Examples of Common Log Format 140.14.6.11 - pawan [06/Sep/2001:10:46:07 -0300] "GET /s.htm HTTP/1.0" 200 2267 140.14.7.18 - raj [06/Sep/2001:11:23:53 -0300] "POST /s.cgi HTTP/1.0" 200 499 • GET request that retrieves a file s.htm • POST request sends data to a program s.cgi • Fields: • client machine’s IP address (140.14.6.11) • RFC 1413 identity of the client is missing (-) • Date and time • Request • Error code • Number of bytes transferred

9. Examples of Common Log Format An example of a log file in extended format

10. Format of Web Logs #Version: version of the extended log file format used #Fields: fields recorded in the log #Software: software that generated the log #Start-Date: date and time at which the log was started #End-Date: date and time at which the log was finished #Date: date and time at which the entry was added #Remark: Comments that are ignored by analysis tools

11. Format of Web Logs • The directives #Version and #Fields are mandatory and must appear before all the entries • Each field in the #Fields directive can be specified in one of the following ways: • an identifier; e.g., time • an identifier with a prefix separated by a hyphen; e.g., cs-method • a prefix following a header in parentheses; e.g., sc(Content-type)

12. Format of Web Logs • No prefixes for date, time, time-taken, bytes, cached • Prefixes for ip, dns, status, comment, method, uri, uri-stem, uri-query, host • Prefixes can be: csclient to server scserver to client srserver to remote server (this prefix is used by proxies) rsremote server to server (this prefix is used by proxies) x application-specific identifier

13. Analyzing Web logs

14. Analyzing Web Logs General Summary from Analog

15. Analyzing Web Logs Monthly report from Analog

16. Analyzing Web Logs Daily summary from Analog

17. Hourly summary from Analog

18. Analyzing Web Logs

19. Organization report from Analog

20. Search-word report from Analog

21. Operation-system report from Analog

22. Status-code report from Analog

23. File size report from Analog

24. File type report from Analog

25. Directory report from Analog

26. FRequest report from Analog

27. Analysis of Clickstream: Studying Navigation Paths

28. Analysis of Clickstream: Studying Navigation Paths Clickstream using Pathalizer with seven link specification

29. Analysis of Clickstream: Studying Navigation Paths Clickstream using Pathalizer with twenty link specification

30. Visualizing Individual User Sessions A brief on-campus session identified by StatViz that browses the bulletin board

31. Visualizing Individual User Sessions A brief off-campus session identified by StatViz with three distinct activities

32. Visualizing Individual User Sessions A long on-campus session identified by StatViz with multiple activities

33. Caution in Interpreting Web-Access Logs • Requests may not always reach the server as they may be served from a proxy server’s cache • You do not really know: • Identity of readers • Number of visitors • Number of visits • User’s navigation path through the site • Entry point and referral • How users left the site or where they went next • How long people spent reading each page • How long people spent on the site

34. Turner (2004) I’ve presented a somewhat negative view here, emphasizing what you can’t find out. Web statistics are still informative: it’s just important not to slip from “this page has received 30,000 requests” to “30,000 people have read this page.” In some sense these problems are not really new to the web—they are present just as much in print media too. For example, you only know how many magazines you’ve sold, not how many people have read them. In print media we have learnt to live with these issues, using the data which are available, and it would be better if we did on the Web too, rather than making up spurious numbers.