Audience Question Cards: Please include e-mail address & questions for follow up

1. “Davenport’s Excellence System Assessment Pilot: Measurable Competencies for Improved Teaching and Learning, Student Certification, and Workplace Success” Wynalda TLI Presentation August, 2014 Dr. Irene Bembenista: Vice Provost for Assessment & Graduate Studies Dr. Wayne Sneath: Program Director Experiential Learning Mike Carey: Department Chair, Marketing

2. Audience Question Cards: Please include e-mail address & questions for follow up

3. Session Objectives • To explain the purpose and design of the Excellence System Assessment Pilot • To explain the proposal for certification of students in the Excellence System • To present data on the assessment of all nine Excellence System competencies in courses from two years of the pilot • To generate discussion on the implications of this ongoing assessment on improved teaching and learning and workplace success

4. The DU Excellence System History • The Mission of Davenport University is “to prepare individuals and organizations to excel in the knowledge-driven environment of the 21st Century.” • To that end, the Davenport University Excellence System was createdin partnership with employers who helped to define and reinforce the necessity of these competencies. • The Excellence System consists of nine professional competencies necessary for graduates to engage in life long-learning and succeed in their chosen profession.

5. The Provost’s Charge Committee Formed in January, 2012 “The committee shall use the DU Excellence System to develop a methodology and a process that will validate students graduating from DU shall have achieved the skills necessary for each of the DU principles of excellence”

6. Committee Proposal for Excellence System Certification Summary: 1. Assessment of proficiency in all 9 Excellence System competencies in key Foundations of Excellence courses & provision of feedback via the development of an electronic portfolio system 2. Assessment of proficiency in all 9 Excellence System outcomes in mid- level and capstone-level courses in each program via the portfolio system 3. Determination of minimum cumulative GPA in all assessed courses

7. What is Proposed as Certification in the Excellence System? Minimum score at the Mastery Level in each Excellence System Competency via analysis of assignments from capstone and/or other program courses (score TBD via pilot) + Minimum cumulative GPA in all assessed courses in the Foundations + Program Level (GPA TBD) = Certification

8. Year I: Goals of the Pilot 1. Determine a baseline of performance in all 9 Excellence System competencies by examining required assessment assignments from three Foundations of Excellence courses: COMM 120: Presentation Techniques RAA: Final Informative or Persuasive Speech Competencies Assessed: Ethical Reasoning and Action Information & Tech Proficiency Professional Communication ENGL 311: Professional Writing RAA: Group Project Competencies Assessed: Critical & Creative Thinking Analysis & Problem Solving Written Communication Leadership & Teamwork SOSC 201: Diversity in Society RAA: Experiential Learning Paper Competencies Assessed: Global & Intercultural Competence Civic and Social Responsibility

9. Year 1: Goals of the Pilot Determine performance in all 9 Excellence System competencies at mid-level and capstone-level courses in programs: Marketing was the pilot program with assessment in: MKTG 213: Advertising Foundations MKTG 441: Strategic Marketing (Capstone) 3. Create common rubrics for all 9 Excellence System competencies utilizing cross-college faculty teams 4. Report data to the Provost with recommendations by May 1, 2013, including a recommendation for development of an electronic portfolio system through which chosen assignmentscan be deposited and assessed by course-level faculty

10. Some Important Questions to be Answered via the Pilot • What are the appropriate Foundations of Excellence, Foundations of Business, Technology, or Health, and/or Program courses in which to assess the competencies? • Is a review of course RAAs via a common rubric the best means of assessment? Should we recommend nationally normed exams or create our own? • How can we effectively train all instructors in these courses to perform consistent assessment of competencies and report this data? • What are the best means to assess competenciesnot • easily evaluated by a team via an artifact (technology • proficiency, teamwork, professional communication?) • What role should ongoing assessment teams play for the Excellence System? Are they cross-college or only by Program? Should they be a “quality check” on course-level instructor assessment scores?

11. Process for Collection & Assessment of RAAs • Faculty in chosen sections require students to post RAAs or RAAs are downloaded from existing course shells • A sample of all RAAs submitted were downloaded & identifying information was removed • 4 person, cross-college faculty assessment teams were formed and trained in inter-rater reliability and sample scoring; teams read samples to calibrate before scoring independently • Teams assessed 6 of 9 competencies • Course faculty assessed “Observable” competencies of Information & Technology Proficiency, Leadership & Teamwork, and Professional Communication

12. E-Portfolio System Vetting Committee Purposes: Research options for an e-portfolio system which would assist with ongoing Excellence System assessment Recommend a system with proposed costs and implementation timeline to the Provost as part of the pilot Portfolio system should serve multiple purposes: ES Assessment & Student Formative Feedback Faculty Rank & Promotion Dossiers Student Career Development Prior Learning Assessment Portfolios

13. Irene Year One Pilot Results for: Global & Intercultural Competence

14. Global & Intercultural CompetenceGraduates understand that working and succeeding in an inclusive, multicultural, and international world involves complex issues present in diverse environments 1. Evaluates and demonstrates culturally-appropriate behavior in diverse domestic and global contexts2. Interprets issues and events from diverse domestic and global perspectives3. Demonstrates an understanding of the dynamics of power, privilege and inequality and appropriate interventions/responses to bias in diverse domestic and global contexts 4. Demonstrates awareness of personal culture and biases and how these influence interactions/relationships in diverse domestic and global contexts5. Demonstrates an ability to interact successfully with people from diverse domestic and global perspectives6. Understands and examines differing value systems to determine appropriate courses of action and decision making in the diverse work environment and the domestic and global community

15. -- Introductory Level—SOSC 201’s Experiential Learning Paper-- Reinforced Level—MKTG 213’s Group Advertising Plan-- Mastery Level—MKTG 441’s Group Strategic Marketing Plan RAAs Mapped to GIC

16. -- Only 32% of RAAs required a 3rd reader, thus 68% agreement-- AACU’s standard—reliable at 70% or greater when two readers are within one level of scoring agreement on a rubric-- GIC was one of the highest reliabilities of the 9 Excellence System competencies assessed in Year One Inter-Rater Reliability for GIC

17. -- Mean score was 6.7 of 18 rubric points (6 outcomes x 3)-- 94% of students were proficient in this competency in SOSC 201 -- 27% of these scored above the Introductory Level 2012-13 Results: GIC in SOSC 201

18. -- 94% proficiency at Introductory Level does NOT demonstrate the global aspect of this competency-- GIC Rubric at Introductory Level assesses either domestic OR global competence-- GIC Rubric at the Mastery Level assesses both domestic AND globalcompetence Implication I

19. -- Students scored lowest in GIC Competency Outcome 6:“Understands and examines differing value systems” Implication II

20. Aggregate Results of All 9 Excellence System Competencies

21. Percentage of Pilot Students Proficient vs. Not-Proficient in all Excellence System Competencies: Year One Pilot Data

22. Mike Year One Pilot: Marketing Program Data

23. Percentage of MKTG441 Students Proficient v. Not Proficient by Excellence System Competency Year One Pilot: MKTG Program

24. Marketing Program Excellence System Certification Analysis for MKTG 441

25. Correlation Analysis ofMKTG Program GPA to RAA Assessment Performance

26. Irene Year Two PilotPlanning & Results

27. Conclusion to Year One Pilot Average rubric scores compiled and report data was provided to the Provost with recommendations in June, 2013 Provost and Deans requested a second year of comparative data Work began to plan Year Two of the pilot: 2013-14

28. Planning Year Two of the Excellence System Pilot • Score a sample of papers from Fall 2013 courses and compare assessment teams’ scores to those already completed by course faculty to determine reliability of instructor scores as a means to do ongoing assessment • Continue to refine RAA assignments and Excellence System rubrics • Use data to determine baseline proficiency scores for certification at the Program Level • Inform curriculum mapping and construction of new RAA assignments • All PROGRAMS in the colleges charged to identify courses to assess RAAs for all levels of Excellence System proficiency—I, R, and M

29. 2013-14 Pilot: Phase II Pilot Courses: 12 courses, 80+ sections, 1100+ studentsin Fall 2013 & Winter 2014 CONTINUED ASSESSMENT OF THE FOLLOWING COURSES: COMM 120: Professional Communication ENGL 311: Leadership & Teamwork, Written Communication SOSC 201: Global & Intercultural Competence, Civic & Social Responsibility ADDITIONAL COURSES ADDED FOR ASSESSMENT IN YEAR TWO: HLTH 101: Critical & Creative Thinking, Analysis & Problem Solving FRSM 100: Critical & Creative Thinking, Analysis & Problem Solving BUSN 210: Ethical Reasoning & Action CISP 112: Information & Technology Proficiency NEW Marketing Program Courses: MKTG 310, 412, 421, 430, 441

30. Year Two Pilot Results: General Total Unique Students Assessed: 2013:N=1076 Total Unique Students Assessed: 2014: N=654

31. Excellence System Competency Means 2013 v. 2014

32. Excellence System Proficiency Comparison 2013 v. 2014

33. Mike Marketing Program: Year Two Pilot Results

34. 2014 Marketing Program Courses & Sample Sizes GIC (MKTG 421: N=26) CSR (MKTG 310: N=0) ERA (MKTG 310: N=17) CCT (MKTG 430 N=40) APS (MKTG 412: N=46) LT (MKTG 441: N=33) ITP (MKTG 412: N=46) WC (MKTG 421: N=26) PC (MKTG 441: N=33) Total Unique Students Assessed: 2014:N=162

35. Marketing Program Competency Means 2013 v. 2014

36. Marketing Program Proficiency Comparison 2013 v. 2014

37. MKTG Program Lessons Learned • Need to Determine the Right: • Course aligned with the Excellence System competency • Assignment (RAA) with the Excellence System competency • Team v. Individual assignments • Rubric language • Training for faculty on scoring using rubrics • Variance in Team v. Faculty scoring in Year Two • Remember: • Not a lot of additional time in reading assignments: most RAAs are read once for a course grade, but Excellence System rubric scores do NOT impact students’ course grades • Not all 9 Excellence System competencies are assessed in each course; assessment is spread out across Foundations and Program-Level courses

38. MKTG Program Next Steps

39. Wayne Current Work & Overall Project Next Steps

40. Curriculum Mapping, CRIRs, and ES Data Collection Summer 2014 Work Excellence System competencies mapped at I,R, and M levels for all academic programs across the colleges CRIR revisions to identity RAAs to be assessed against ES common rubrics Development of an Excellence System assessment website: www.davenport.edu/academics/assessment/excellence-system 4. Development of Google Drive score sheet system for 9 Excellence System competencies to be used by course faculty to report data

41. Project Next Steps Determination on final criteria for Excellence System Certification Begin assessment in all sections of all identified Excellence Systems assessment courses across all academic programs in Fall 2014 E-Portfolio pilot with College of Urban Education cohort (Fall 2014) Broader portfolio pilot sample (Winter 2015) Audience Questions and Comments