Program Outcome Assessment: Case Study from Three Engineering Programs

1. Program Outcome Assessment: Case Study from Three Engineering Programs Presented by Toni L. Doolen, Professor School of Mechanical, Industrial & Manufacturing Engineering Oregon State University 09/21/2011

2. Abstract This presentation will introduce a framework for a continuous improvement process for programs driven by program outcome assessment. A case study illustrating the application of this framework to programs in the College of Engineering will be presented. Program outcome data collected from different sources (courses, students, and alumni) is being used to provide feedback to evaluate whether or not changes are needed in programs. A set of processes has been implemented to create an annual review cycle with a focus on continuous improvement of programs. Specific examples and templates for program outcome assessment and reporting will be provided.

3. Agenda • Overview of the School of Mechanical, Industrial, and Manufacturing Engineering (MIME) • Assessment and Evaluation in MIME • Program Educational Objectives • Student Outcomes • Course Learning Outcomes • Continuous Improvement process

4. MIME • Demographics • ~35 full-time faculty members and instructors • ~185 graduate students in three programs (industrial engineering, material science mechanical engineering) • ~1,375 undergraduates in five programs (energy engineering management, general, industrial, manufacturing, and mechanical) • ~ $7.3 million in research funding in 2010

5. Need for Assessment and Evaluation • Undergraduate programs accredited under ABET • Many employers require accreditation • Students must graduate from accredited engineering program to sit for PE (Professional Engineering exam)

6. Three Levels of Assessment and Evaluation

7. Assessment Data collected from Multiple Stakeholders • Students • Alumni • Industrial Advisory Board Members • Employers (local, regional, national, international)

8. Levels of Assessment in MIME

9. Program Educational Objectives Program educational objectives are broad statements that describe what graduates are expected to attain within a few years of graduation. Program educational objectives are based on the needs of the program’s constituencies. Program constituencies: students, faculty, employers/graduate schools, alumni, state of Oregon, …

10. PEOs A & E Process • Defined for programs • Our graduates will be systems thinkers. • Our graduates will be global collaborators. • Our graduates will be innovative designers and problem solvers. • Assessed and evaluated using alumni (3 – 5 years after graduation) survey data, employer survey data, and IAB input. • Responsibility for A&E and CI process is MIME Head/AH, ABET coordinator, and undergraduate program committees

11. MIME Program Educational Objectives

12. PEO Assessment Data from Alumni • Assessment of MIME PEOs is completed using an annual Alumni Survey that is administered to alumni 2 – 5 years after graduation. • Items include specific questions about PEOs. Rated on a 7-point scale (1=not at all; 7=extremely). Respondents also rate importance of each PEO. • Survey provides an assessment of PEO achievement directly and also uses aggregated Student Outcome achievement ratings provided by alumni for a second measure. • Other demographic details are collected related to employment and general satisfaction with educational experiences.

14. PEO Evaluation • PEOs and assessment results reviewed with IAB annually. • PEO data analysis and recommendations prepared by ABET coordinator. Analysis and recommendations reviewed with UPCs on an annual basis and actions for improvement identified.

15. Linkage of PEO’s to Student Outcomes

17. Student Outcomes Student outcomes describe what students are expected to know and be able to do by the time of graduation. These relate to the skills, knowledge, and behaviors that students acquire as they progress through the program

18. Student Outcomes A&E Process • General and Program-specific student outcomes defined e.g. • Assessed and evaluated using exit surveys completed by students in the quarter they graduate and through Course Learning Outcome mapping • Responsibility for A&E and CI process is ABET coordinator and UPC

19. Student Outcomes Examples • General outcomes for all programs • Ability to design and conduct experiments, as well as to analyze and interpret data • Ability to design a system, component, or process to meet desired needs • Ability to function on multi-disciplinary teams • Program-specific outcomes • Ability to design, develop, implement and improve integrated systems that include people, materials, information, equipment, and people • Ability to work professionally in mechanical systems area including the design & realization of such systems

20. Student Outcome Assessment via Exit Survey • Assessment of student outcomes is completed using an annual Exit Survey that is administered to graduates in the term that they graduate, i.e. administered all 4 quarters. • Items cover Student Outcomes that are rated on a 7-point scale (1=not at all; 7=extremely). • Survey provides a direct assessment of program specific students outcomes in addition to a-k. • Other demographic details are collected related to coop experiences, number of interviews, employment status, and general satisfaction with their educational experiences (labs, faculty, advising, facilities, classes, etc.)

24. Student Outcome Evaluation Process • Student Outcome data analysis and recommendations are prepared by ABET coordinator. Analysis and recommendations reviewed with UPCs on an annual basis and actions for improvement identified.

26. CLOs Course-level Assessment and Evaluation

27. Course Learning Outcomes Course learning outcomes are statements that describe what students are expected to know and/or be able to do at the completion of a specific course.

28. CLOs • 3 – 7 LO’s typical for a course • Examples of LO’s • Be able to identify and describe (in writing) the most significant challenges faced by engineering managers in organizations in today's global environment. • Be able to identify and facilitate solutions to ethical dilemmas faced by engineering managers

29. Student Outcome Assessment via Course-Level Assessment and Evaluation • Faculty must also identify two other direct measures to evaluate the achievement of Course Learning Outcomes and ultimately of the linked Program Outcome. Specific assignments, exams, project reports, and grading rubrics are used by faculty for direct measures as appropriate to a given course. • Faculty members are responsible for collecting the necessary data for assessment and for completing appropriate analyses of the data to determine whether or not Program Outcomes have been achieved.   • Faculty must also provide a narrative summary of their evaluation. This process is completed annually for all required courses (or every other year for courses taught on an every other year basis). Faculty members are responsible for reporting any significant issues in meeting Program Outcomes to the UPC. Evidence of this process is provided through the course notebooks.

30. CLO Assessment Plan Example

31. Evaluation of Learning Outcomes

32. CLO Evaluation Example Expectation is for at least three distinct measures for each learning outcome, one of these measures can be student evaluation of CLO achievement for SET

33. Continuous Improvement Process

34. Continuous Improvement Cycle for all A&E Processes

35. PEO CI Process Examples • Act • Do

36. Study

37. Plan PEO CI Process Examples

38. Plan SO CI Process • Do • Study • Act • Study

39. Plan CI Process: Plan In response to the ABET reviewer’s concerns on thermal design and the persistent low achievement rating of Outcome c, the ME ABET coordinator as well as the UPC began discussions on possibility of enhancing the design experience for students in Fall 2009. A recommendation from the coordinator and the UPC on this matter to the ME faculty is tentatively scheduled for Fall 2010.

40. CI Process: Do • Do Considering the low achievement rating on Outcome c and d, ME 382 was revised in its Fall 2009 offering to include a greater emphasis on designing real world products and forming diverse teams (using personality and skills as metrics). This change in project focus, while retaining the same design methodology topics, is expected to increase student awareness of the relevance of design to the mechanical engineering profession through is concrete experience with real mechanical systems (covering all domains of mechanical systems). Any potential impacts from these revisions will be first captured in the 2011 exit surveys (since ME382 is a junior-level course). Since Fall 2009, ME382 has been made a required course for MfgE and an elective for IE students. This ensures that the teams are more multi-disciplinary. Any potential impacts from these revisions will be first captured in the 2011 exit surveys.

41. CI Process: Study and Act • Study • Act

42. CLO CI Process