Transaction Log Analysis & Michigan Teacher Network

1. Transaction Log Analysis & Michigan Teacher Network Marcia A. Mardis Merit Network at the University of Michigan

2. Working Definitions • User centered design is a methodology that concentrates on product design and development from user, user task, and user interface perspectives. A UCD process functions in parallel with a product development process to: • Collect Data: About users, their tasks, and their current work processes. • Analyze Data: To determine how to redesign the workflow, and to identify usability requirements and goals. • Design and Develop: Scenarios, information architectures and grammars, and user interface prototypes. • Test: Design and conduct iterations of usability assessments and tests. • Deploy: Follow up with measures of usability and user satisfaction • Transaction log analysis (TLA) uses transaction logs to discern attributes of the search process, such as the searcher’s actions, the interaction between the users and the system, and the evaluation of the results by the searcher. (Jansen & Pooch, 2000).

3. What can you do with TLA? • Characterize user sessions • Length of time and time of day • Number of pages viewed • Where coming from • Characterize of user queries • Number of revisions • Number of terms used • Use of logic and advanced search techniques • Characterize search terms • Topic • Frequency of terms

4. Key Authors & Studies • Spink, Jansen, Wolfram & Saracevic • Excite studies • Meta-evaluation • Frameworks, general taxonomies • Bryan-Kinns & Blandford • Digital library TLA (vs. Web TLA) • Interactional traps (misunderstanding; premature bailout) • Silverstein & Henziger • AltaVista search logs • Comparison of theoretical vs. actual user models • Abbas • Teachers & Children’s info seeking (her own work and lit review) • Influence of metadata and search scaffolds

5. Consistent Findings from Studies • Very few queries use advanced search features like boolean; half of those that do contain mistakes • Highly skewed use of terms: few used a lot, a lot used infrequently • Distribution of terms bears no relation to distribution of content • Users look mainly at the first page of results only • Most sessions contain only 1 query • Most queries are very short (3 terms or less)

6. Caveats • Are the exhibited searching characteristics due to some unique aspect of the system or collection? • Are the user samples from the same or different populations? • Are the searching characteristics really applicable to all users? • What are the implications of users who do not understand what they are searching (metadata vs. full-text) • What is the tradeoff between user privacy and data collection? • Real searchers have different information seeking processes • Are the topic classification schemes used consistently? • Is the data collection tool accurate?

7. What are information seeking behaviors? • Actions and expressions exhibited (recursively) during: • Query formation • Topic and concept identification • Keyword generation • Information collection • Source selection • Source navigation (Searching/browsing) • Results evaluation • Quality reflection • Query revision • Source reselection (choose another tool)

8. Why is it important to look at these behaviors? • For tool developers • User interface design • Training/Help content and design • Knowledge of where need to ask questions might arise • For questioners • Understand where asking questions may be integrated into information seeking process • For responders • Phrasing and types of answers given • Understanding of barriers to use

9. What’s consistent in the literature? • Time is consistently a big factor in teacher information seeking (Fitzgerald, 2001; Pittsley, 2000, etc.) • Teachers say quality is important (Sutton, 1999; Fitzgerald, 2001; Mardis, 2003, etc.) • Teachers don’t take time to evaluate quality (Lazonder, 2001) • Teachers are willing to interact (Fitzgerald, 2001; Shumar, 2003; Renninger, 2003) • Teachers consult colleagues (Sutton, 1999; Mardis & Hoffman, 2003, etc.)

10. Higher ed faculty and K-12 teacher behaviors: compared • Instructors • Primary research (use of email, discussion groups) fieldwork data, archival materials, and gray literature • Citation rhetoric (Bates, 2001) • Very little instruction on using Web for course resources • Teachers • Information tends to be for self-generated queries • Teachers adapt resources (Sutton, 1999; Fitzgerald, 2001) • Teachers use each other as primary sources of information

11. Adults and K-12 teacher behaviors: compared • Topic familiarity determines amount of time spent searching and reading. (Kelly & Cool, 2002) • Both groups are not familiar with search engine structure and function (Bradley, 1999) • No structured search models exist for either group

12. Children and K-12 teacher behaviors: compared • Little extensive, controlled study has been done (Dresang, 1999) • Children overperceive their abilities to use search tools; teachers bail out more quickly (Dresang, 1999; Fidel, 1999) • Both groups excel in systems designed for them (Hirsh, 1997; Dresang, 1999) • Younger students and teachers favor browsing (Dresang, 1999)

13. What are some key ideas? • Save time • Describe richly (Sutton, 1999) • Quick response is essential • Recommend ready-to-use resources with clear application and evaluation/assessment • Assure quality (rating/peer review) (anecdotal) • Question formation is still weak link • Don’t assume technology skills can/will be gained(Weglinksy, 2000) • Users need a lot of support (documentation, online help) (Pollock, 1999)

14. About Michigan Teacher Network (MTN) • Founded in 1998 as state clearinghouse • Contains over 8500 professional learning, professional support, and curriculum resources correlated to the Michigan Curriculum Framework • Receives additional funding from the NSF NSDL • Receives 500,000 (summer)-700,000(fall/winter) page views per day • Executes 750 (summer)-950(fall/winter) search strings per day • Governed by Merit privacy policy

15. Research Questions • What do MTN users look for? • Do MTN users look for classroom planning information at home? • Do users use too many or too few search terms? • Do users seem to understand what they’re searching?

16. Characteristics of Windows Examined

17. Data collected from April 29-May 14, 2003

18. Top Searches 2/19-3/5/2003

19. Top Searches 3/26-4/9/2003

20. Top Searches 4/29- 5/14/2003

21. Data collected from April 29-May 14, 2003

22. 2 1 3 Data collected from April 29-May 14, 2003

23. Conclusions • Many users exhibit “Search Engine Habits” • Search for items beyond scope of collection • Seem to be expecting to search site text, not metadata • Perform searches on browsing categories

24. Conclusions • Most searches occur after school • MTN users seek classroom management help the most • Science is the most searched curriculum area • Users search very broad or very narrow concepts with few terms • Users seem to want objects • Users repeatedly type in the same search

25. Users misunderstand sometimes… • Ozzy Osbourne • Motown Ho Down • Health priorities in Australia • Bay City Boating and Fishing • Recipes (food) • Spices and herbs • Buying a car • SARS • Mazda rx-8 • Bush declares war