Action Analytics: Setting the National Agenda

1. Action Analytics: Setting the National Agenda Michael J. Offerman Capella University Linda L. Baer Bill and Melinda Gates Foundation October 13, 2010

2. “By 2020, America will once again have the highest proportion of college graduates in the world” President Barack Obama, February 24, 2009

3. White House Summit on Community Colleges The summit brought together community colleges, business, philanthropy, federal and state policy leaders, and students to discuss how community colleges can help meet the job training and education needs of the nation’s evolving workforce.

4. White House Summit on Community Colleges • Pathways to Baccalaureate • Increasing Community College completion • Affordability: Financial Aid to Community College Students • Community Colleges in the 21st Century • Importance of Community Colleges to Veterans and Military Families • Industry-Community College Partnerships

5. Skills for America's Future will bring employers and union leaders together with community college leaders from across the country in order improve workforce development and meet the competitive needs of business. Skills for America's Future is an initiative of the Aspen Institute, in partnership with the Pritzker Traubert Family Foundation. http://www.aspeninstitute.org/policy-work/economic-opportunities/skills-for-americas-future

6. A Stronger Nation Through Higher Education • Bill & Melinda Gates Foundation Post Secondary Success • Double the number of low-income adults who earn postsecondary degrees or credentials by age 26 by 2025 • $440 million to degree completion, curricular focus, maximizing technology enabled curriculum, analytics for student success • Lumina's big goal is to have 60 percent of Americans hold high-quality, two- or four-year college degrees and credentials by 2025 • 16 million more graduates than we will at current rates. • National Association of System Heads – Access to Success A2S focus on cutting the achievement gap in half by 2015

7. Presentation Plan: • Overview of Action Analytics • Define and describe • What it requires • National Action Analytics Agenda • Connecting the dots • Selected examples • Minnesota State Colleges and Universities • Capella University • Minnesota Action Analytics • Signals

9. Predictive Modeling • Predictive modeling is the process by which a model is created or chosen to try to best predict the probability of an outcome.[1] In many cases the model is chosen on the basis of detection theory to try to guess the probability of a signal given a set amount of input data. http://en.wikipedia.org/wiki/Predictive_modelling

10. Definitions of Action Analytics Academic analytics combines large data sets, statistical techniques and predictive modeling to produce “actionable intelligence.” Campbell, DeBlois, and Oblinger. 2007. EDUCAUSE review July/August. Action Analytics extends these approaches to include all of the facets of individual and institutional performance – academic, administrative, strategic planning, and work/career – and more. Action analytics is aboutpursuing the improvement and optimization of institutional performance along all dimensions. Norris, Donald, Linda Baer, Joan Leonard, Lou Pugliese, Paul Lefrere, “Action Analytics: Measuring and Improving Performance That Matters,” EDUCAUSE Review, Jan/Feb 2008.

11. Action Analytics Basics Rapid conversion of large data sets to actionable information that is pushed to advisors, faculty, staff, administrators and learners to drive learner success and institutional productivity • Learner Success = learning outcomes, persistence and completion • Drive learner success up while driving costs down

12. Action Analytics Involves • Tools, practices, mindset change • Continuous quality improvement • Accountability and transparency • Reimagining academic assumptions, policies and practices • Beyond a culture of evidence to a culture of data-driven improvement

13. Why Action Analytics Now? • Changes in external environment • Accountability • Learning outcomes assessment for accreditation • Economic reality • Reduced resources • Consumer demand for greater value • Data opportunities • Vast amount of data in online & blended delivery formats • Opportunities to open up to “non-power users” • Major federal, higher education association and foundation commitment

14. Timeframes for Action • 2008-2009 Campus leaders have been “staunching the flow,” responding to reduced funding with stop-gap actions • 2010-2012 Cutbacks will continue, although stimulus funding is reducing the pain, giving campuses the opportunity to come up with strategies for improving performance and value and establishing plans for financial sustainability in “the new normal” • 2013-2020 State revenues and funding will not rebound strongly, and there will be many competitors for funding; institutions will need to be strategically realigning to the new normal • 2020 Institutions will need to have achieved a new financial sustainability, with significant changes in practices

15. We believe • That various constituents expect us to do more with less and to achieve excellence • Action analytics can help us to do both • We need to learn from one another and quickly improve our utilization of data and analytics • There is a need for a national higher education agenda to drive action analytics

16. Assessment, Accountability, Accreditation and Analytics

17. Connecting and Aligning the Dots • Assessment • Accountability • Accreditation • Analytics • Concrete Measures • Concrete Targets • Concrete Assurances • Concrete Practices

18. Setting the National Action Analytics Agenda • Driving a national agenda for action analytics as a way to encourage higher education improvement • Two invitational symposia • Creation of content “observatory” and communities of practice • www.Edu1world.com

19. Applying analytics • What student outcome data are you using now? • How are you using the data? • With whom do you need to be sharing data? • Do you know the key “lose points” for students? • What tools are you using?

20. Applying analytics: • Do you understand what students experience? • What is the cost of student failure? • What is the cost of customized student success interventions? • What would you need to do to improve student success?

21. Strategic Intelligence for Higher Education What’s the best that can happen? What will happen next? What if these trends continue? Why is this happening? What actions are needed? Where exactly is the problem? How many, how often, where?

22. Examples of Action Analytics • Minnesota State Colleges and Universities accountability dashboard (MNSCU) • Capella University learning outcomes assessment systems and predictive modeling • Minnesota Action Analytics • Purdue University’s SIGNALS program

23. Minnesota State Colleges and Universities System and Campus Level Model

24. Strategic Directions • Increase access, opportunity and success • Ensure high-quality programs and services through a commitment to academic excellence and accountability • Provide programs and services to enhance the global economic competitiveness of the state, regions and its people • Innovate to meet current and future educational needs • Ensure the long term viability of public higher education in Minnesota

25. Assessment  Accountability Actions • Measures and Metrics (AQIP, Baldrige) • Targets and Performance Expectations • Percent Change in Enrollment • Net Tuition and Fees as % Median Income • Licensure Exams Pass Rate • Persistence and Completion Rate • High Quality Learning • Student Engagement • Partnerships • Related Employment of Graduates • Innovation • Facilities Condition Index

27. Performance Thresholds • Board or Chancellor goal or target • Board: Facilities condition index will improve • External reference standard • Licensure exam pass rates should approach state average • Expected value • Regression model predicts related employment rate based on program mix • Historical performance • Persistence rate should stay within historical range

29. Details and Documentation • View Trend: link to trend display • Explore Further: Link to drill-down analytic tool • Review Definition: Link to definition and documentation on data source and performance threshold method • Comments: Link to comments on performance and action plan if measure “needs attention”.

30. Making Accountability Systemic • Measures align with goals from system’s strategic plan • Dashboard reports system and institutional performance • Analytic tools enable institutions to drill-down on measures • Board holds chancellor accountable for system performance and progress • Chancellor holds vice chancellors and presidents accountable for performance and progress

31. Capella University Learning Outcomes Assessment Model

32. Recognition of Capella University Learning Outcomes Model 32 • 2010 Council for Higher Education Accreditation (CHEA) Award for Outstanding Institutional Practice in Student Learning Outcomes • 2009 IMS Global Learning Consortium Global Learning Impact Platinum Award • 2009 WCET Outstanding Work (WOW) Award • 2009 Interactive Media Awards (IMA) Best in Class Award

33. Capella University Model 33 • Learning Outcomes Assessment Plans (LOAP) for all bachelor’s and master’s degrees • Plan backwards from program level learning outcomes, developed in collaboration with industry and professional associations (employers) • Build curriculum maps from beginning to end • Embed formative and summative assessments—Fully Embedded Assessment Model (FEAM) • (Assess at concept, competency and outcome levels) • Rubrics to achieve consistency across faculty (positive retention impact)

34. Outcomes, Competencies, and Criteria Outcomes are the expected results that Capella has for its learners —what a learner should know, be able to do, and understand at the end of a course or program. Competencies are bundles of skills needed to demonstrate the outcomes. Criteria are measurable descriptions of learner performance against competencies.

35. Outcomes, Competencies, and Criteria Outcome: Communicate effectively. Competency: Communicate ideas effectively in writing. Competency: Speak persuasively. Competency: Collaborate with team members. Criterion:Lead the audience to visualize how your proposal will benefit them. Criterion:Get the audience’s attention. Criterion: Establish credibility. Unique Criterion Unique Criterion

36. Impact 36 • Identify and measure learning outcomes at program as well as core level. • Measure continuous quality improvement • Enhanced engagement with students and faculty • Strong rubrics and metrics • ALERTS • First Friday • Second Friday • Push information to advisors, faculty, learners, other staff • Share externally

37. Transparency 37 • Once infrastructure in place and data is generated: power in opening the system • Data available to students, faculty, leaders and others—ALERTS system • Open up externally • Transparency by Design—consumer information and accountability • http://CollegeChoicesforAdults.org • Context • Learning outcomes (both core and program level) • Engagement • Common alumni survey questions • Next: employer feedback

38. Minnesota Analytics Partnership • Minnesota State Colleges and Universities • Capella University • University of Minnesota • St. Scholastica

40. Signals Project • To increase the success of their students Purdue University has implemented an intervention system called Signals. • It combines predictive modeling and the data mining of their Blackboard Vista system to identify students who are academically at risk. • Risk ratings are assigned, integrated with Blackboard and available on the student’s Blackboard homepage. • Signals also provides intervention e-mails from instructors and communicates available campus resources to assist the student.http://www.itap.purdue.edu/tlt/signals/signals_final/index.htm

42. Selected Resources • Campbell, John P. Peter P. DeBlois, and Diana G. Oblinger, “Academic Analytics: A New Tool for a New Era,” EDUCAUSE Review, July/August 2007. • Davenport, Thomas and Jeanne Harris. Competing on Analytics. Harvard Business School Press. 2006 • Ewell, P. T. (2007). Making the Grade: How Boards Can Ensure Academic Quality. Washington, D.C.: Association of Governing Boards of Universities and Colleges • Ewell, P. T. & Jones, D. P. (2006). State Level Accountability for Higher Education: On the Edge of a Transformation. In N. B. Shulock (Ed.), Practitioners on Making Accountability Work for the Public (pp. 9-16). New Directions for Higher Education. 135, Fall 2006. • McLaughlin G. W. & McLaughlin, J. S. (2007). The Information Mosaic: Strategic Decision Making for Universities and Colleges. Washington, D.C.: Association of Governing Boards of Universities and Colleges. • Norris, Donald, Linda Baer, Joan Leonard, Lou Pugliese, Paul Lefrere, “Action Analytics: Measuring and Improving Performance That Matters,” EDUCAUSE Review, Jan/Feb 2008. • Norris, Donald, Linda Baer, Joan Leonard, Lou Pugliese, Paul Lefrere, “Framing Action Analytics and Putting Them to Work,” EDUCAUSE Review, Jan/Feb 2008, electronic article.

43. THANK YOU