Hypothesis (conceptual)

1. Consent Form Background Questionnaire Demo 1 Pre-Search Questionnaire Search (10 minutes) Post-Search Questionnaire 4 Tasks 4 Tasks Post-System Questionnaire BREAK Demo 2 Pre-Search Questionnaire Search (10 minutes) Post-Search Questionnaire Post-System Questionnaire Exit Questionnaire Results Purpose of the Study Investigate the role that the layout of search results plays in supporting human searchers executing topic distillation tasks. • Experimental Design • Two alternative user interfaces for the Panoptic search engine • - the “linear” interface/system displays a simple list of hits, ranked by Panoptic • the “hierarchical” interface/system clusters the search results based on common URL • (sub-domain/ path). Between and within cluster ranking is based on Panoptic scores. • Participants: 16 volunteers • Each subject conducted eight searches, four on each system. • The order of systems and topics was rotated to minimize the learning and fatigue effects. Rutgers’ Web Interactive Track at TREC 2003N. J. Belkin, H.-J. Lee, Y.-L. Li, G. Muresan, M.-C. Tang Hypothesis (conceptual) A match between the layout of the search output and the nature of the task reduces the searcher’s effort and improves the retrieval performance. Hypothesis (operational) A hierarchic layout of the search output performs better than a list of hits in terms of user effort and search performance. • XML-based approach • to interaction modeling • model the interaction using UML diagrams; this • allows the whole research team to contribute to • the design of the user interfaces, and supports the • documentation of the interface; • derive an XML/DTD coding for the states of the • user interface, the valid user actions in every state • and the state transitions that take place based on • user actions; • use XML to log the user actions and their • outcomes based on the interaction DTD; • based on the DTD, generate a log parser for the • log analysis; • - the log analysis provides interaction • information and can also be used for generating • a visual / graphic representation of the interaction Log DTD <?xml version='1.0' encoding='US-ASCII'?> <!ELEMENT log (record)*> <!ELEMENT record (date, millis, message)> <!ELEMENT message (StartSession|EndSession|RequestURL| FollowLink|SaveDoc|UnsaveDoc|ViewSavedDoc|DisplayDoc| BackToDoc|SelectDocViewer|SelectDoc|SeeHits|SearchResults |SelectSearchResults|QuerySubmitted)> <!ELEMENT SaveDoc (Doc)> <!ELEMENT UnsaveDoc (Doc)> <!ELEMENT ViewSavedDoc (Doc)> <!ELEMENT DisplayDoc (Doc)> <!ELEMENT SelectDoc (Doc)> <!ELEMENT SeeHits (Hit)*> <!ELEMENT SearchResults (Hit|Count)*> <!ELEMENT QuerySubmitted (#PCDATA)> <!ELEMENT RequestURL (#PCDATA)> <!ELEMENT FollowLink (#PCDATA)> … Method Collection: .gov Search Engine: Panoptic Task: Topic distillation Participants: 16 volunteers Conclusions The hierarchic interface seems to be conducive to less effort for the searcher: fewer iterations, shorter search sessions, fewer documents seen, selected and viewed. Users perceived the hierarchic system as easier to use and better at supporting the Topic distillation task. Subjects perceived the two systems equally easy to learn; they preferred the hierarchic display.