E-Outcomes Assessment Project: Technology Linking Assessment & Learning

1. E-Outcomes Assessment Project: Technology Linking Assessment & Learning Luvai Motiwalla, Steven Tello & Kathryn Carter College of Management UNIVERSITY OF MASSACHUSETTS LOWELL Presentation at The NERCOMP ConferenceMar 20th 2007, Worcester, MA.

2. Agenda • Assessment Challenge • Project Overview • Program-Level Assessment Process • Systems Integration • Progress to Date • Future Directions

3. The Assessment Challenge • Increased calls for accountability in higher education • The Spelling Report, Secretary of Educationcalls for “evidence-based” assessment measures • Regional and Professional Accreditation agencies • Widespread interest in higher-educationinstitutions to define and assess student learning objectives • No resolution on how to define, measure, and achieve quality learning • Different approaches for outcomes assessment • student learning within courses and academic programs • student learning within specific fields or majors • Pre and Post learning assessments, surveys, etc.

4. The Assessment Challenge • What is Assessment?Many different perspectives • Administrator, Faculty, Student, Agency • “Assessment is the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development.”(Palomba & Banta, 1999)

5. The Assessment Challenge Problems with Assessment • Tedious • Costly • Untimely • Painful • Useless • A labor intensive, paper-based process driven by accreditation agency visits & timelines

6. A Possible Solution… Design and develop an electronic outcomes assessment process • which captures the course or student activities (e.g., assignments, discussions, or exams) within a learning management systems (LMS) • categorize it relative to the college or university’s stated learning goals or outcomes.

7. Project Overview • Two-year, grant-funded project • UMass Lowell & Davis Educational Foundation • Use LMS to organize and report student progress toward program-level learning outcomes • Interdisciplinary focus • 6 Business Core Courses, 6 General Education • Two operational foci • Faculty Development • Systems Development

8. Project Overview - Goals • Improve the learning environment for students by increasing student access to content, instructors and peers. • Improve content and assessment consistency across multiple disciplines. • Facilitate student development of academic e-portfolios. • Reduce overall administrative costs through online paperless assessment, reporting, tracking.

9. Assessment Process - Learning Outcomes I II III IV V VI VII VIII Breadth of Knowledge UniversityGeneral EducationOutcomes Clear Communication InformationLiteracy Ethics Self-Direction/ Collaboration Critical Thinking Diversity InformationCommunication Technologies COMLearning Outcomes Quantitative & Qualitative Skills Ethics & Values Communication Skills Decision-making&Problem solving Diversity Impact Team Leadership & Membership GlobalImpact

10. Assessment Process – Self Reporting Current Process Manual, Paper-based, Self-report

11. Assessment Process: New Approach Proposed Process Automated, Menu-driven, Drill down Assessments Discussion Topic A Assignment 1 Exam 1 & 3 Assignment 2 Group Tool Usage Group Project Excel Assignment Case Assignment Courses CoM Learning Outcomes Course A Course B Course C Course N COMMUNICATION SKILLS QUANTITATIVE SKILLS TEAM PARTICIPATION IT USAGE ETHICS DIVERSITY IMPACT GLOBAL IMPACTS

12. Assessment Process: Future Reporting Possibilities Digital Dashboard

13. Systems Integration • Use LMS (WebCT Vista) as Assessment Collection Tool • Organize & present course assessment activities • Present outcomes to students • Collect assessment artifacts • System limitations require external database • To map course activities to program-level outcomes • To organize & report progress toward outcomes

14. Assessment Tool Chat Tool CourseContent Files DiscussionTool Grade Book Syllabus Tool Assignment Tool Groups Systems Integration • Use the activity tools in WebCT Vista • Train faculty in specific use of these tools

15. Systems Integration • Use PowerSight to report off Grade book • Train faculty to use grade book

16. Systems Integration • EOutcomes Database • My SQL, Perl, Web Interface • Export CSV from WebCT, Import to EOutcomes • Faculty map outcomes to specific course activities in EOutcomes Database • Administrators then report on progress toward outcomes through EOutcomes

17. EOutcomes System V1 Systems Integration

18. Learning Outcomes Menu Systems Integration

19. Progress to Date – Year 1 • Conducted Summer Training Workshop • Seven courses offered in Fall 2006 / Spring 2007 • Developed a prototype EOutcomes database and application

20. Goal 1: Improve learning environment by increasing student access… • 7 Courses offered Fall2006, 385 students • Each Web-Enhanced to varying degrees.

21. Goal 1: Improve learning environment by increasing student access…

22. Goal 2: Improve content and assessment consistency across multiple disciplines • EOutcomes Training Institute, June 2006 (June, 2007 as well) • Learning Outcomes • Assessment Matrix in Excel • Assessment Tools in WebCT • Reporting Tools in WebCT • Respondus Exam Tool • EOutcomes Database • Tablet PC Demo • Training Materials online • http://comwebserver.uml.edu/eoutcomes/ • EOutcomes WebCT Course

23. Goal 3: Facilitate student development of web-based academic portfolios. • ePortfolios provide ability to link sample student work to activity to learning outcomes • Very limited success due to WebCT issues and manual nature of effort • ePortfolios often focused on student need to build a portfolio, rather than administrative need to measure achievement • Continue to review possible tools within WebCT and external • Possibly EOutcomes Database

24. Goal 4: Reduce overall administrative costs • Possible savings include: • Supply savings – less scantron sheets and paper for exams • Proctor savings – less time spent scanning, reviewing and reporting exam grades (less room for error) • Faculty time savings when: • grading exams, many are auto-scored • grading quizzes, many are auto-scored • filling out assessment spreadsheets • collecting student work samples • making copies • Savings are balanced by initial development effort on part of faculty, and maintenance

25. Emerging Research Questions • Does the use of a LMS in a web-enhanced course increase student awareness of program-level learning outcomes? • Do students perceive a benefit in the use of a LMS for course-level assessment activities? • Do faculty perceive a benefit in the use of a LMS for course-level assessment activities? • Does the use of a LMS facilitate the organization and reporting of program-level assessment data for accreditation purposes? • Does the LMS in a web-enhanced course have any positive impact on students’ involvement in a course or satisfaction with the course or their learning?

26. Next Steps • Develop EOutcomes Reporting System tools • Incorporate six more courses in 2007-2008 academic year • Conduct training institutes in Summer 2007 • Measure Project Achievements • Integrate Research Agenda

27. Thank you • Prof. Luvai MotiwallaLuvai_Motiwalla@uml.edu • Prof. Steven TelloSteven_Tello@uml.edu • EOutcomes Websitehttp://comwebserver.uml.edu/eoutcomes/