Program Assessment Process for Continuous Quality Improvement (CQI)

1. Program Assessment Process for Continuous Quality Improvement (CQI) Prepared by: Abul Fazal M. Arif Member, ABET Steering Committee November 16, 2008.

2. Contents • ABET General Criteria, 2008-2009 Cycle • Assessment Process – key components • Tasks to be Done • Closing the Loop • Action Plan Program Assessment Process

3. ABET GENERAL CRITERIA “Renumbered” Criteria:(2008-2009 Accreditation Cycle) Criterion 1. Students Criterion 2. Program Educational Objectives Criterion 3. Program Outcomes Criterion 4. Continuous Improvement Criterion 5. Curriculum Criterion 6. Faculty Criterion 7. Facilities Criterion 8. Support Criterion 9. Program Criteria Program Assessment Process

4. Assessment Process for Quality Assurance Criterion 1. Students Criterion 2. Program Educational Objectives Criterion 3. Program Outcomes Criterion 4. Continuous Improvement Criterion 5. Curriculum Criterion 6. Faculty Criterion 7. Facilities Criterion 8. Support Criterion 9. Program Criteria Assessment is an ongoing process of setting high expectations for student learning, measuring progress toward established learning outcomes, and providing a basis for reflection, discussion and feedback to improve University academic programs. It is a systematic and cyclic process that makes expectations and standards explicit and public. Program Assessment Process

5. Program Educational Objectives Performance Criteria Program Learning Outcomes Closing the Loop Action Taken / To be Taken Curriculum Mapping Assessment Methods Evaluation Key Components of Assessment Process Program Assessment Process

6. Educational objectives are statements that describe the expected accomplishments of graduates during the first few years after graduation—usually 3-5 years. Program Educational Objectives Performance Criteria Program Learning Outcomes Closing the Loop Action Taken / To be Taken Curriculum Mapping Assessment Methods Evaluation Program Assessment Process

7. Program Educational Objectives Performance Criteria Program Learning Outcomes Closing the Loop Action Taken / To be Taken Curriculum Mapping These are statements related to skills, knowledge and behaviors that describe what students are expected to know and/or be able to do by the time of graduation. • If students have achieved these outcomes, it is anticipated that they will be able to achieve the educational objectives after graduation. Assessment Methods Evaluation Assessment Process for Quality Assurance Program Assessment Process

8. Assessment Process (cont.) • At the heart of assessment are the Program Objectives and Outcomes. • Purpose: To assess the extent a program is achieving its objectives and outcomes. • To be conducted continuously at different stages  continuous improvement. • The assessment process itself should be evaluated and refined in light of emerging insights. • All accrediting bodies, including ABET, require evidence of continuous quality improvement of both,Process and Program. Program Assessment Process

9. Tasks to be done Task 1: Review Program Educational Objectives (PEO) Task 2: Review Program Learning Outcomes Task 3: Align Learning outcomes with PEO Task 4: Present to Constituents Task 5: Get Approval & Publish Task 6: Mapping Course Outcomes to Program Outcomes Task 7: Develop Performance Indicators Task 8: Identify Assessment Tools/Methods Task 9: Develop Assessment Plan Task 10: Data Collection and Evaluation Task 11: Closing the Loop Program Assessment Process

10. TASK 1: Review Program Educational Objectives (PEO) • Review process will be initiated by Department ABET Committee or PAC • Guidelines for review • PEO should be consistent with proper definition. • PEO should be aligned with mission of university/College/Department. • Numbers of PEO should be manageable. • PEO should be measurable. Program Assessment Process

11. STEP 1: Revise Program Educational ObjectivesExamples of PEO • Prepare graduates who will be successful in mechanical engineering-related careers and other diverse career paths • Prepare graduates to be leaders • Prepare graduates to be good citizens of the society • Prepare graduates to pursue advanced degree, if so desired • Prepare graduates to be life long learners • Prepare graduates to be entrepreneurs Program Assessment Process

12. TASK 1: Review Program Educational ObjectivesExamples of PEO - Combined • Prepare graduates who will have successful careers, and become leaders, in industry and the public sector • Prepare graduates who will appropriately apply acquired knowledge, work well in team, effectively communicate ideas and technical information, and continue to gain knowledge • Prepare graduates who will continue professional development and will pursue continuing education opportunities relevant to their careers Note: For more examples of PEO, refer to Aggarwal’s workshop notes. Program Assessment Process

13. TASK 2: Review Program Learning Outcomes Task 2.1: Engineering programs must demonstrate that their students attain the following (a) to (k) outcomes: (a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability (d) an ability to function on multidisciplinary teams Program Assessment Process

14. Task 2: Review Program Learning Outcomes (cont.) (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively (h) the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context (i) a recognition of the need for, and an ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice Program Assessment Process

15. Task 2: Review Program Learning Outcomes (cont.) • Task 2.2: Check if any additional outcomes that may be articulated by the program criteria. • Check Criterion 9 on Program Criteria from ABET 2008-2009,(For Mechanical Engineering) Mechanical Engineering program criteria include the requirement that the program must demonstrate that graduates have the ability to: apply principles of engineering, basic science, and mathematics (including multivariate calculus and differential equations) to model, analyze, design, and realize physical systems, components or processes; and work professionally in both thermal and mechanical systems areas. Program Assessment Process

16. TASK 3: Align Learning outcomes with Program Objectives. • Program outcomes must foster attainment of program educational objectives. • Link outcomes with PEO • All Learning Outcomes may or may not link to all/any PEO Program Assessment Process

17. TASK 4: Present to Constituents Main Stakeholders/Constituents • Faculty through Department Council • Employers through Industrial Advisory Committee • Students through various Professional Groups or Student Societies etc. Others: • Administration • Alumni • Other Graduate Schools • Recruiters • Employers Note: Mostly faculty define Program Learning Outcomes Program Assessment Process

18. TASK 5: Get Approval & Publish Get Approval • Final form of the Program Educational Objectives and Learning Outcomes should be approved by the Department Council • Keep record in “ABET File” Publish at: • Department Website • General Catalog • Bulletin Boards etc Keep minutes of meeting of constituent’s involvement and Council approval as evidence in record. Program Assessment Process

19. Program Educational Objectives Performance Criteria Program Learning Outcomes Closing the Loop Action Taken / To be Taken Curriculum Mapping Curriculum Mapping is the alignment of the program learning outcomes with the curriculum and, therefore, it needs development of performance indicators based on the curriculum. Assessment Methods Evaluation Program Assessment Process

20. TASK 6: Mapping of Course Learning Outcomes to Program Learning Outcomes • Include only Core (required) courses • Select courses for Outcome Assessment Program Educational Objectives Performance Criteria Program Learning Outcomes Closing the Loop Action Taken / To be Taken Curriculum Mapping Assessment Methods Evaluation Program Assessment Process

21. Mapping: Course Learning Outcomes and Program Outcomes Program Assessment Process

22. Mapping: Core Courses & Program Outcomes Program Assessment Process

23. Program Educational Objectives Performance Criteria Program Learning Outcomes Closing the Loop Action Taken / To be Taken Curriculum Mapping TASK 7: Develop Performance Indicators for each Program Outcome Assessment Methods Evaluation Program Assessment Process

24. Performance Criteria (indicators, PIs) are defined as the specific, measurable statements identifying the specific knowledge, skills, attitudes and/or behavior students must demonstrate as indicators of achieving the outcomes. Program Educational Objectives Performance Criteria Program Learning Outcomes Closing the Loop Action Taken / To be Taken Curriculum Mapping Assessment Methods Evaluation Program Assessment Process

25. Examples of Performance Indicators • Outcome (a): Ability to apply knowledge of mathematics……… • Applies mathematics principles to obtain solutions. • Uses principles of sciences and engineering in solving engineering problems. • Combines scientific and engineering principles to formulate models of processes and systems. • Combines mathematics principles to formulate models of processes and systems. Program Assessment Process

26. Examples of Performance Indicators • Outcome (a): Ability to apply knowledge of mathematics……… 1. Applies mathematics principles to obtain solutions. 2. Uses principles of sciences and engineering in solving engineering problems. 3. Combines scientific and engineering principles to formulate models of processes and systems. 4. Combines mathematics principles to formulate models of processes and systems. Generally, it is recommended to limit performance criteria to three-four for each outcome. Note: For more examplesof PI, refer to Sub-Committee Report. Program Assessment Process

27. Assessment is one or more processes that identify, collect, and prepare data to evaluate the achievement of program outcomes and program educational objectives. Program Educational Objectives Performance Criteria Program Learning Outcomes Closing the Loop Action Taken / To be Taken Curriculum Mapping Assessment Methods Evaluation Program Assessment Process

28. TASK 8: Identify Assessment Tools/Methods. • Assessment tools are of two types: • Direct measures are based on facts (it also includes opinion based on facts). • Indirect measures are based on opinion only. • Multi-source or multi-method approach is generally recommended for data collection and assessment. • Indirect measures are necessary but not sufficient. They should be complemented with at least one direct measure. • Sometimes it may not be possible to use direct methods to measure some of the outcomes (for example team work). In those cases, we can rely on indirect methods. Program Assessment Process

29. Task 8: Identify Assessment Tools (cont.)Examples of Direct measures • Student course portfolios and senior capstone course projects using overall score of learning outcome rubric • External review or performance appraisal through direct observation on internship • Locally developed exams. • Oral exams • Standardized exams such as FE, GRE, GMAT • Special student achievements (awards, student publications and presentations at various meetings) • External examiners • Graduate schools Program Assessment Process

30. Learning Outcome Rubric Rubrics are developed from performance indicators and consist of sets of categories that define the process towards meeting the goal. Program Assessment Process

31. Example of Rubric on Teamwork Program Assessment Process

32. Task 8: Identify Assessment Tools (cont.)Examples of Indirect measures • Questionnaires and written surveys (Alumni and employer surveys) • Senior exit surveys and interviews • Student course evaluations • Grades ** • Focus groups conducted by trained moderator • Students internship evaluations (if based on opinion) ** Refer to Sub-Committee Report or Aggarwal Workshop Notes. Program Assessment Process

33. TASK 9: Development Assessment Plan • Selection of Courses • Data Collection Cycle • Identify Data To be Collected Program Assessment Process

34. Mapping: Core Courses & Program Outcomes Program Assessment Process

35. Selection of Courses for Outcome Assessment Program Assessment Process

36. Selected Courses for Outcome Assessment Program Assessment Process

37. Data Collection Cycle Program Assessment Process

38. Data Collection Cycle Learning Outcomes are measured on a continuous basis (3 years cycle is acceptable). You do not have to measure every student for every outcome. For PEOs, typically data is collected every three years from all constituents for every objective. Program Assessment Process

39. Data to be collected For Program Learning Outcomes • Students Course Evaluation using rubric • Student Course Portfolios • Grades • Performance Appraisal • Senior exit surveys and interviews • Locally developed exams For Program Educational Objectives • Alumni Survey • Employers Survey • Industrial Advisory Board’s Feedback • Placement Data Program Assessment Process

40. TASK 10: Collection of Data & Evaluation Program Assessment Process

41. Evaluation is one or more processes for interpreting the data and evidence accumulated through assessment practices. Evaluation determines the extent to which program outcomes or program educational objectives are being achieved, and results in decisions and actions to improve the program. Program Educational Objectives Performance Criteria Program Learning Outcomes Closing the Loop Action Taken / To be Taken Curriculum Mapping Assessment Methods Evaluation Program Assessment Process

42. Example of Evaluation • Evaluation using EXCEL file Program Assessment Process

43. TASK 11: Closing the LOOP Program Assessment Process

44. The feedback process is critical to creating and maintaining a systematic quality assurance system. When successfully implemented, all elements of the quality assurance process interact with one another. Program Educational Objectives Performance Criteria Program Learning Outcomes Closing the Loop Action Taken / To be Taken Curriculum Mapping Assessment Methods Evaluation Program Assessment Process

45. Program Educational Objectives EXAMPLE of Assessment Plan, Report & Rubrics Performance Criteria Program Learning Outcomes Closing the Loop Action Taken / To be Taken Curriculum Mapping Assessment Methods Evaluation Program Assessment Process

46. ABET Accreditation: key Dates Program Assessment Process

47. Tasks to be done Task 1: Review Program Educational Objectives (PEO) Task 2: Review Program Learning Outcomes Task 3: Align Learning outcomes with PEO Task 4: Present to Constituents Task 5: Get Approval & Publish Task 6: Mapping Course Outcomes to Program Outcomes Task 7: Develop Performance Indicators Task 8: Identify Assessment Tools/Methods Task 9: Develop Assessment Plan Task 10: Data Collection and Evaluation Task 11: Closing the Loop Program Assessment Process

48. Task 3: Draft Self-Study Reports ReadyBy: Academic Departments Submission date: 1-4-2009 Program Assessment Process

49. Sources of Information • http://www.abet.org/forms.shtml • Aggarwal’s workshop # 1 lectures • Aggarwal’s workshop # 2 lectures • ABET Sub-Committee Report Program Assessment Process

50. THANK YOU Program Assessment Process