The Impact of Alternative Search Mechanisms on the Effectiveness of Knowledge Retrieval

1. The Impact of Alternative Search Mechanisms on the Effectiveness of Knowledge Retrieval Ryan C. LaBrie Department of Information Systems W. P. Carey School of Business Arizona State University Dissertation Defense Presentation Thursday 27 May 2004 10:00 a.m. – Noon BA 253

2. Today’s Agenda • Motivation • Literature Review • Model and Hypotheses • Method • Results • Discussion

3. Why Study Knowledge Management? • 75% of organizational assets are now intangible (knowledge) assets. - Alan Greenspan • The 500 largest firms in the United States had intangible assets valued at $7.3 trillion (70% of their total value). • These 500 companies employ more than 21.6 million employees and generate over US$ 6.1 trillion in revenue. (Stone, 2002)

4. Why Knowledge Management? • An Arthur Anderson study revealed: • In 1978 the balance sheet explained 95% of the market value of the firms. • In 1998 the balance sheet only explained 28% of the market value of those firms. • Today, the balance sheet explains less than 15% of the market value of the average firm (Stanfield, 2002). • Impending requirements of the Sarbanes-Oxley Act • Legislation requiring improved access to documentation and other knowledge objects.

5. Knowledge Management Research • Alavi & Leidner (2001) provide a KM research framework creation – “storage/retrieval” – transfer – application • This dissertation focuses specifically on the retrieval aspect of knowledge management research addressing the following question posed by Alavi & Leidner: “What retrieval mechanisms are most effective in enabling knowledge retrieval?”

6. Research Question Does the cognitive loading of search mechanisms impact the effectiveness of knowledge retrieval?

7. KNOWLEDGE INFORMATION DATA The Data, Information, Knowledge Hierarchy

8. What is Data? • Definition of data: • Sets of symbols not necessarily understood by, found meaningful by, or causing a change of state in the destination (Meadow & Yuan, 1997). • Connotes: raw facts, direct observations • Example: • 65

9. What is Information? • Definition of information • A set of symbols that does have meaning or significance to the recipient (Meadow & Yuan, 1997). • Connotes: structure, definition, semantics • Example: • Age: 65

10. What is Knowledge? • Definition of knowledge: • The accumulation and integration of information received and processed by a recipient (Meadow & Yuan, 1997). • Information bound together by structure, assumptions, justification, and process (Edgington et al., in press). • Connotes: relationships, models, and usage • Example: • At age 65 you may retire from work as a United States federal employee.

11. The Difference between Data and Knowledge Retrieval • Blair (2002) outlines five major differences • Additional work by Blair identifies 12 differences

12. Keyword Retrieval Mechanisms • Gorla & Walker (1998) • Ambiguity biggest problem • Call for a controlled vocabulary (classification) • LaBrie & St. Louis (2003) • Mirrored Gorla & Walker results with a different data set.

13. Did Classifications Improve Retrieval Effectiveness? • LaBrie & St. Louis (2003) • Classifications mirror keyword results • Classification schemes are almost immediately outdated and require updating • Weber (2003) • Ended 15 years of ISRL classification usage in MIS Quarterly

14. The Human Factor • Knowledge Management and KM retrieval involves people as part of the equation • People have cognitive limitations • Cognitive processing size limitations • People have inherent cognitive retrieval processes • Recall versus recognition retrieval • People have thresholds for decision-making • Effort versus accuracy • People have varying degrees of experience (or previous knowledge)

15. Recall and Recognition • Recall: retrieval mechanism with no cues or hints to help the individual make a decision or judgment (Driscoll, 2000). • Example: What does the word esoteric mean? • Recognition: involves a set of pre-generated stimuli presented to the individual for a decision or judgment (Driscoll, 2000). • Example: Which of the following words is the best synonym for esoteric? • Essential • Mystical • Terrific • Evident

16. Milestones in Recall & Recognition Research • Ebbinghaus’ Forgetting Function (1885) • Beginning of the recall research, exponential decay function. • Miller’s Magical Number Seven (1956) • There is a cognitive limit (7+/- 2) of how much information we can remember, use. • Simon’s Hierarchical Systems (1962, 1974) • Humans store information hierarchically. • Bower et al’s Recall versus Recognition Experiments (1969) • Recall is a serial process, recognition is a parallel process. • Anderson’s Adaptive Character of Thought – ACT-R (1995, 1997) • Retrieval of information is facilitated if it is organized hierarchically. • Theory of the way declarative and procedural knowledge interact in complex cognitive processes.

17. Effort versus Accuracy The saliency of effort is much greater than that of accuracy. • Todd & Benbasat (1992) • Effort and decision quality with decision aides • Vessey (1991, 1994) • Cognitive fit, Information presentation (graphs vs. tables) and decision making • Galleta et al. (1996) • Effort and accuracy in spreadsheet evaluations • Speier & Morris (2003) • Query interface design on decision making performance

18. The Role of Experience • Dhaliwal & Benbasat (1996),Gregor & Benbasat (1999), Mao & Benbasat (2000) • Provide theoretical support that experience plays a role in KB/IS usage • Markus (2001) • Differentiates between novice and expert users in her theory of knowledge reuse

19. Reexamining the Research Question Does the cognitive loading (recall or recognition, small to large sets, user experience) of search mechanisms (keyword or visual) impact the effectiveness (accuracy, timeliness, work effort, satisfaction) of knowledge retrieval?

20. Research Model (Figure 6, p.35)

21. Measuring the Variables • Automatically, electronically by the system • Captures time and articles • Measuring accuracy • Expert raters using a psuedo-Delphi method (Buckley, 1995) • Measured both omitted relevant (Type I errors) and included irrelevant (Type II errors) articles • Validated survey instruments captured work effort, and satisfaction data • Work Effort NASA-TLX (Hart & Staveland, 1988) • Mental, Physical, Temporal, Frustration, Performance, Effort • End-User Computing Satisfaction (Doll & Torkzadeh, 1988) • Content, Accuracy, Format, Ease of Use, Timeliness

22. Summary of Hypotheses (Main Effects)

23. Summary of Hypotheses (Interaction Effects)

24. H4: Retrieval Method – Results Set Size Interactions Visual Visual TIME Low High ACCURACY Low High Keyword Keyword Small Medium Large RESULT SET SIZE Small Medium Large RESULT SET SIZE Visual Keyword WORK EFFORT Low High SATISFACTION Low High Visual Keyword Small Medium Large RESULT SET SIZE Small Medium Large RESULT SET SIZE

25. H5: Retrieval Method – User Experience Interactions Visual Visual TIME Low High ACCURACY Low High Keyword Keyword Low High USER EXPERIENCE Low High USER EXPERIENCE Keyword Visual WORK EFFORT Low High SATISFACTION Low High Visual Keyword Low High USER EXPERIENCE Low High USER EXPERIENCE

26. Method • A 2 x 3 x 2 mixed factor, repeated measures (on one factor), analysis of variance (ANOVA), experimental design. • Independent Variables • Retrieval Method (keyword or visual – between subjects) • Result Set Size (small, medium, and large – within subjects) • User Experience (low or high – between subjects) • Dependent Variable: Retrieval Effectiveness • Accuracy • Time • Work Effort • Satisfaction

27. Method (cont.) • Laboratory Experiment Characteristics: • Task setting: Each subject was randomly assigned to one of two search interfaces (keyword or visual) • Controls for carry-over effect • Task setting: Each subject was requested to perform search tasks on three different scenarios • See slide deck appendix for scenarios • Incentives included a $100 cash prize for the highest level of accuracy • Subjects: Four sessions – 3 graduate sections, 1 undergraduate (seniors) section

28. The Search Tasks • Three randomized search scenarios • Controls for learning effect • Manipulated to return various result set sizes • Large “System Design” scenario • 20% of the data set (120 correct responses) • Medium “User Acceptance” scenario • 6-7% of the data set (40 correct responses) • Small “Risk Management” scenario • 1-2% of the data set (10 correct responses)

29. Demonstration of Search Interfaces

30. Summary of Results

31. Accuracy Hypothesis – 1.1 • Knowledge management systems that employ a visual tree-view hierarchy search interface will produce more accurate results than knowledge management systems that employ a text-based keyword search interface. Supported

32. Accuracy Hypothesis – 2.1 • As result set size increases retrieval accuracy will decrease. Supported

33. Troubles Measuring Experience • Bedard (1989) and Gregor & Benbasat (1999) discuss theoretical support for experience as a valid variable for both direct and moderating effects in knowledge-based research. • They also note the difficulties in finding a generally accepted definition and a method for operationalizing the concept.

34. Factor Analysis of Experience Items                           Rotated Component Matrix(a) Extraction Method: Principal Component Analysis.  Rotation Method: Varimax with Kaiser Normalization. a  Rotation converged in 4 iterations. • Content Experience (loaded on by SDExpert, UAExpert, RMExpert) • Educational Experience (loaded on by ISDegree, ISProf, EducationLevel, GradorUndergrad) • Search Experience (loaded on by SearchBig, SearchSml)

35. Accuracy Hypothesis – 3.1 • Subjects with more (educational) experience will have higher retrieval accuracy than subjects with less (educational) experience. Supported

36. Top 10 Accuracy Results* * Mean Accuracy Rate = 24%

37. Time Hypothesis – 1.2 • Users of knowledge management systems that employ visual tree-view retrieval interfaces will spend more time searching than users of knowledge management systems that employ text-based keyword retrieval interfaces. Supported

38. Time Hypothesis – 2.2 • As result set size increases time will increase. Note: Directional for the visual search interface only, the keyword interface remained flat across the scenarios. Directional

39. Time Hypothesis – 3.2 • Subjects with more (educational) experience will perform searches faster than subjects with less (educational) experience. Not Supported

40. Work Effort Hypothesis – 1.3 • Knowledge management systems that employ visual tree-view hierarchy search interfaces will use less work effort than knowledge management systems that employ text-based keyword search interfaces. Not Supported

41. Work Effort Hypothesis – 2.3 • As result set size increases work effort will increase. Not Supported

42. Work Effort Hypothesis – 3.3 • Subjects with more (educational) experience will use less effort than subjects with less (educational) experience. Not Supported

43. Satisfaction Hypothesis – 1.4 • Knowledge management systems that employ visual tree-view hierarchy search interfaces will have a higher degree of satisfaction than knowledge management systems that employ text-based keyword search interfaces. Not Supported

44. Satisfaction Hypothesis – 2.4 • As result set size increases satisfaction will decrease. Not Supported

45. Satisfaction Hypothesis – 3.4 • Subjects with more (educational) experience will be more satisfied than subjects with less (educational) experience. Not Supported

46. Summary of Hypotheses Results (Main Effects)

47. Summary of Hypotheses Results (Interaction Effects)

48. Revised Model (Figure 17, p.123)

49. Repeated Measures Analysis

50. Findings and Implications • Cognitive loading does have a significant effect on retrieval effectiveness with respect to accuracy and work effort. • 50+% Accuracy gains for approximately one minute more of your time. • Satisfaction between the two search mechanisms was virtually identical. • Work effort turned out to be significantly greater in the visual interface, opposite of hypothesized direction. • Retrieval accuracy can be improved for a limited cost using current technology.