Evaluation: Preliminary Results from the Server Side

1. Evaluation: Preliminary Results from the Server Side Frank A. Settle Elizabeth Blackmer Thomas Whaley The Alsos Digital Library for Nuclear Issues Washington & Lee University Lexington, Va

2. Evaluation from the Server Side Who? What? How? Why? Server log data Server User

3. Preliminary Observations from the Server Side • Robots, crawlers, and other automated searches produce over 50% • of the visits. While they inflate the number of visitors, they do • produce human visits indirectly. • Search engines (Google and Yahoo) provide the majority of referrals. • Sites that have embedded, directed links to specific bibliographical • information provide over 10% of the referrals. • Referrals from the NSDL portal as well as other sites that include Alsos • in their list constitute a small percentage of referrals. However, users • may be using a search engine to follow up on an Alsos link they saw in • a list on another site.

4. Overall Usage - September 2002 – September 2003

5. Top Referral Sites – September 2003 • Visitors • www.google.com 687 • search.yahoo.com 289 • chemcases.org 227 • search.msn.com 138 • www.google.ca 54 • aolsearch.aol.com 39 • www.atomicarchive.com 38

6. Embedded Link from Atomic Archive to Alsos

7. Preliminary Observations from the Server Side • Printing (hardcopy or files) indicates real engagement of the user. • However there are also serious users who do not print so this criteria • for estimating engaged users is low. • Almost all of the visitors listed as referrals from other sites are • human users, not robots, crawlers, etc. • Based on a typical day (Tuesday, September 30, 2003) a large • number of users (~72%) who are referred directly to an annotation • by another site, print the annotation and exit Alsos immediately. • User access to the six general browse areas (Issues, Warfare, etc.) • indicates an equal distribution of interests. This validates the • selection of these areas and shows the broad interests of users.

8. EngagedUsers’ Behavior Tuesday, Sept. 30, 2003 Entry Point Did user move beyond printing the annotation? Homepage Search Annotation

9. Distribution of Browse Topics for September 2003 (human visitors)

10. Home Page

11. Browse Issues Area

12. Results for Nuclear Waste

13. Complete Citation with Annotation

14. Processed Information from Server • Tabularsession displays – summary information for each session • Individual session display – tracks the user through a session • Suffix statistics – provides information on types and also locations • of international users • Hit counts for individual references – skewed by robots and multiple • hits during a single session

15. Processed Server Log Data

17. Problem – hits are skewed by robots, crawlers, etc.

18. Types of Users • Robots, crawlers, etc. • Humans who scan topics, site information, etc., but don’t access • specific annotations • Single reference users referred to Alsos from a search engine or • website to a specific annotation. User gets information and exits. • Often prints. • Human browsers who go to a browse area, obtain a list of references, • and then examine specific annotations. May or may not print. • Human searchers who enter keywords(s), title, or creator to trigger a • search. Most users in this category do not use the advanced search • feature.

19. Single reference user from Google

20. Searcher enters keyword “Iran”

21. Browser in People Area

22. Casual Visitor – Scans Information

23. Summary • Massive amounts of raw server log data must be • transformed into smaller, organized sets of • information. • This information can provide useful information • on who is using the collection and how they use it. • This information is useful in improving the utility • of the collection and the effectiveness of the • user interface. • This information is critical in seeking funds to • sustain and expand the collection