Useful Decision Support : What is it – and Why is it so hard to create? Arizona Association for Institutional Research

1. Useful Decision Support: What is it – and Why is it so hard to create? Arizona Association for Institutional Research Annual Meeting March 2007 Richard D. Howard University of Minnesota rdhoward@umn.edu

2. Overview • What is Useful Decision Support (Actionable Knowledge)- How does it relate to decision making and decision support? • Information Support Circle – Converting data to useful information to inform campus planning and decision making. • Barriers to Effective Decision Support – Why are the data “wrong” and What needs to be done to “fix” them?

3. Useful Decision Support Actionable Knowledge “… is any knowledge that can be put into a design that the human mind can use in a causal manner. "(http://www.hi.is/~joner/eaps/y3_33875.htm) ACTION

4. Creating Actionable Knowledge isDecision Support Primary role is to reduce the risk to the decision maker CERTAINTY

5. Decision Support Focus High Persuade Prescribe Seek agreement of values Describe action needed Cause-effect known Prepare Pray Procrastinate if possible Identify alternative scenarios Low High Desirability of outcome agreement

6. Three Questions – Three Levels of Knowledge What did you find?-- Technical Knowledge What does it mean?-- Issues Knowledge So what?-- Contextual Knowledge All three types of knowledge must be present to create the most useful decision support .

7. Rational Decision Making Process Decision Knowledge Intelligence (Actionable Knowledge) Information Assessment Data

8. How is decision support created and what limits its effectiveness?

9. Information Support Circle Custodian Steward USER Decision Maker Identify and Measure Concepts Use and Influence Knowledge Collect and Store Data Quality Decision Making Deliver and Report Information Restructure and Analyze Facts PRODUCER Broker

10. Identification and Measurement ModelingIdentifying ConceptsSelecting Measures • Defines the area of concern and need. • ExcitementWhat could be done with knowledge?What events would be evident?What process leads up to these events? • ExplorationWhat are the key components in the process?How do they tie together?What is known about causality?What is assumed about the situation? • ClarificationWhat are the key questions?What essential elements of information exist?Alternative ways to measure elements.Costs and benefits of data alternatives. CollectingCoding andStoring UsingInfluencing andDecision Making RestructuringAnalyzing andIntegrating GeneralizingDelivering andReporting sacs2295

11. Identification and Measurement Disease • Belief Bulimia: Semi-random gorging and purging of data from data bases and random changes in beliefs about what is important with no direction. • Symptoms: • > Constricted belief structure without linkage to reality. • > Random interactions of users and technicians with frowns. • > Knee-jerk inclusion of data for specific problems. • > No goals set for major activities. • > No sequence of when various things are needed.

12. ModelingIdentifying ConceptsSelecting Measures CollectingCoding andStoring UsingInfluencing andDecision Making RestructuringAnalyzing andIntegrating GeneralizingDelivering andReporting Capture and Storage • Storage of data requires focus and friends. • Standardization: Identify critical and key elements and codesDefine and document elements and codesMeasure and verify quality and integrityEstablish on-going process • Key elements require: Standard coded representation over data sourcesA standard long nameA standard short nameA standard abbreviation • Administrative University Data Base elements (AUDB): Relevant to planning, management, operating or auditingRequired for use by more than one unitIncluded in official administrative report or surveyUsed to derive an element for one or more criteria above

13. Capture and Storage Disease • Data Dyslexia: Inability to recall or recognize the meaning of the data, not knowing where they came from, often confusing one element for another. • Symptoms: • > Random capture of data as they becomes available. • > Creative coding based on unwritten rules and what works. • > Using one variable for a specific purpose until later. • > Definition depends on who coded the variable. • > Process writes over data when new measure is available.

14. Data • “Facts” that are meaningless until put into a context, either with other data or in the context of a decision. • The sources of the these “facts” are typically the operating systems that drive the academic and administrative/support processes of the campus. • Their restructuring and analysis result in the creation of information which should be used to inform planning and decision making.

15. Census Data Constitutes a source of consistent data to support reporting, institutional effectiveness, program reviews, and ad hoc studies • Student Related Data – Same point in time during the academic term • Faculty and Staff Data – Same calendar date for each academic term • Financial Data – Beginning year budget and end of year expenditures • Facilities – Typically once a year

16. Institutional Administrative Data Management Infrastructure Personnel Financial Student Facilities Operational Systems Standardizing Recodes/Edits Data Description/ Dictionary User Support IADB Security Service Management Information Users

17. Restructure and Analysis ModelingIdentifying ConceptsSelecting Measures • Translate from the input resources to outcome concerns. • Reduce the complexity of the data and focus on specific concern. • Use various types of analyses: Description analysis Translate issues into targets and ranges Consider dispersion and associationComparison analysis Alternative when lack absolute standard Can be based on either internal or externalTrend analysis Depends on expectancy of causality Includes events in other situationsModeling analysis Combination of advanced techniques to predict Use leading indicators, multiple measures, and likelihoods. UsingInfluencing andDecision Making CollectingCoding andStoring RestructuringAnalyzing andIntegrating GeneralizingDelivering andReporting

18. Restructure and AnalyzeDisease Dimensional Dementia: Results are uninterpretable due to irrational combinations of data using methodology based on the available software. Symptoms: > Forgetting the context in which the data were collected. > Summarizing over data collected on various samples. > Using most impressive statistics available. > Cases left over when data bases are integrated. > Major analyses done on PC with no documentation.

19. Delivery and Reporting ModelingIdentifying ConceptsSelecting Measures • Focus on needs of the customer. • New technologies should: Maintain batch access to data bases. Provide processing environment and analyses. Support retrieval, analyses, and interpretation of internal and external data, based on relevant frames. Maintain historical types of data. Comply with external requirements of cross- analyses and integration of data from various sources. Develop storage and retrieval ability for documents. • Delivery includes written and verbal reports. • Reporting includes explaining and generalizing. CollectingCoding andStoring UsingInfluencing andDecision Making RestructuringAnalyzing andIntegrating GeneralizingDelivering andReporting

20. Delivery and Reporting Disease Myopic Megalomania: Self-centered, short-sighted delivery of information based on the whims of the deliverer and independent of the user needs. • Symptoms: > A firmly held belief of technical superiority. > Emphasis on the media and method rather than message. > Disregard of user desires or suggestions of data clerks. > Massive use of extreme-to ids in reports. > Constant purchases of individual software by users.

21. Use and Influence ModelingConceptsSelecting Measures • The key is institutional effectiveness. • Effective DS requires evidence of use. • Users must be supported as active learners. • DS products must be considered in decisions. • The timing of the decision cycle should be shared. • DS must be used in shaping future decisions. • DS should be part of the planning and assessment. • Information will be only as useful as the weakest point. • Cooperation is required for continuous improvement. • Influence comes from reducing the core uncertainty of users. • Cooperation and sharing is critical for quality. CollectingCoding andStoring UsingInfluencing andDecision Making RestructuringAnalyzing andIntegrating GeneralizingDelivering andReporting

22. Use and Influence Disease • Creative Carcinoma: Creating and using facts as needed with First-Liar's Rule, where the fact continues to be quoted until it is a festering sore. • Symptoms: > Junior staff frequently provide complicated definitions. > Executives believe they are invincible. > The lack of good data is blamed for poor decisions. > All decisions are last second to avoid disasters. > Organizational structures are not changed to reflect reality.

23. Information Support Circle Custodian Steward USER Decision Maker Identify and Measure Concepts Use and Influence Knowledge Collect and Store Data Quality Decision Making Deliver and Report Information Restructure and Analyze Facts PRODUCER Broker

24. Two Major Properties 1) Dependency- information created by the process will be only as good as the weakest step in the process. 2) Cooperation- all three roles must function for the good of the institution, none can function in self interest.

25. Some Thoughts about Barriers to Effective Decision Support?

26. Some Institutional Limitations (potential) Political – lack of trust that the data and analyses are reliable and appropriate Resources – lack of skills, access to institutional data, time, access to peers Leadership – inappropriate location in the institutional administrative structureand limited access to decision makers Institutional Culture – inability/unwillingness to act within the context of strategic goals and assessment information

27. The Balancing A Time Quality ct

28. Reality “There are two equally effective ways of keeping a board in the dark. One is to provide them with too little information. The other, ironically, is to provide them with too much.” “Building Better Boards,” by David A. Nadler, Harvard Business Review, May 2004, p.109

29. Institutional Belief: Data-informed processes are better • Different views on most issues can be informed with data. • Most needed data are available. • One can create a user capability and a comparison group. • One can obtain appropriate measures and metrics. • Tools are already available. • If you monitor the process, you can improve the process.

30. What Barriers Limit Effective Decision Support at Your Institution? From the Information Support Audit, check those characteristics that are present at your institution. People, Processes and Managing Data. (2004) McLaughlin, Howard, et. al. Association for Institutional Research

31. Most Effective Decision Support Decision Maker’s Context: Structure & Processes & Values Requires Contextual Intelligence Answers “So What?” More Effective Decision Support Organizational Context: Structure & Processes Requires Issues Intelligence Answers “What Does It Mean?” Least Effective Decision Support Data Context: Information from Analysis Requires Technical Intelligence Answers “What Did You Find?” Data in No Qualitative Context Data in Matching Qualitative Context Low Decision-Maker ProximityHigh Decision-Maker Proximity

32. Barriers to Effective DA • Developers set in their ways. • Systems/data tied to turf battles. • Willing to make do with old technology. • Unwilling to see needs outside operations. • Already busy taking care of "here and now". • Waiting for technology to solve the problem. McLaughlin & McLaughlin, 1989

33. An Irish Prayer May those who love us, love us;And those that don't love us,May God turn their hearts. And if He doesn't turn their heartsMay he turn their ankles,So we'll know them by their limp. AIR Newsletter, Sept 14, 1992