Instrumenting the Learning Commons

1. Instrumenting the Learning Commons Eugene Agichtein, Qi Guo and Ryan Kelly Intelligent Information Access Lab, Math & CS Department Arthur Murphy, Selden Deemer, Kyle Fenton Emory Libraries

2. Goals/Motivation • Evaluate effectiveness of search/discovery with behavioral metrics (task-specific) • Perform aggregate, longitudinal studies • Tools for usability studies “in the wild” • Scale (hundreds/thousands of “participants”) • Realistic behavior and tasks • On-demand playback of “interesting” sessions • Unified analysis/query framework for internal and external resource access and usage statistics • Web-based query and statistics interface • Access auditing, privacy, anonymity enforced Intelligent Information Access Labhttp://ir.mathcs.emory.edu/

3. Approach: Client-side instrumentation • Implementation within the Emory Installation of the LibX Toolbar: (http://www.libx.org) • Extended LibX to track UI events: JavaScript patch to sample the mouse movements and other events on pre-specified web search pages. Events are encoded into a string and buffered, and periodically sent to the server (on internal library network). Intelligent Information Access Labhttp://ir.mathcs.emory.edu/

4. How it works • On login, firefox is started with • http://irlib.library.emory.edu/consent.cgi?user=USERID • If user has previously opted in (or out) • Redirect to Euclid homepage • If new user, show consent form • Store choice in database; if opted in, also store salted hash string for user log in • Can track user behavior over “lifetime” • No way to recover login id by dictionary attack • Can be removed at any time by deleting mapping • LibX sends http requests to server with encoded event strings. Intelligent Information Access Labhttp://ir.mathcs.emory.edu/

5. User Opt-in (new Learning Commons Users) http://irlib.library.emory.edu/ Intelligent Information Access Labhttp://ir.mathcs.emory.edu/

6. Events captured (v0.4, deployed) • Button/link clicks/Url changes • Name of the button, link, other meta-info • Mouse movements • (x,y) coordinates sampled ~every 10ms • Scrolling • Start, stop position, ~ every 10ms • Text entry • Query text, options changes • Keypress events • Menu item events • Print, bookmark, save (all of them) Intelligent Information Access Labhttp://ir.mathcs.emory.edu/

7. Example: Mouse Movement 5 segments: initial, early, middle, late, and end. • “Cheap” proxy for eye-tracking: • Capture physiological characteristics of the mouse trajectories • Segment Properties: • Speed • Acceleration • Rotation • Drift • Hover For each: avg. speed, avg. acceleration, rotation etc. Intelligent Information Access Labhttp://ir.mathcs.emory.edu/

8. Example applications: Classifying Search Intent • Initial exploration: • Standard supervised machine learning classification techniques • WEKA implementation of SVM and decision trees. Intelligent Information Access Labhttp://ir.mathcs.emory.edu/

9. Summary of Features Used Intelligent Information Access Labhttp://ir.mathcs.emory.edu/

10. Example Informational query: “spanish wine” Intelligent Information Access Labhttp://ir.mathcs.emory.edu/

11. Emory User Behavior Analysis System • EUBA: • Client-side instrumentation, Data mining/machine learning (Qi Guo) • Log DB parsing, indexing, web-based interface for querying, playback, annotation (Ryan Kelly) • Plan: to release the system to research/library community Intelligent Information Access Labhttp://ir.mathcs.emory.edu/

12. Demo Prototype: http://ir.mathcs.emory.edu/library/private/index.pl user: test password: notsafe Intelligent Information Access Labhttp://ir.mathcs.emory.edu/