Assessment, Evaluation & Continuous Improvement The Experience of King Abdulaziz University

1. Assessment, Evaluation & Continuous ImprovementThe Experience of King Abdulaziz University Ali M. Al-Bahi Professor of Aeronautical Engineering Director of the Academic Accreditation Unit Jubail Industrial College - Oct. 23, 2011

2. والصلاة والسلام على أشرف الأنبياء والمرسلين سيدنا محمد وعلى آله وصحبه أجمعين

3. Contents • Who are we? • ABET TAC Accreditation • Assessment and Evaluation of Program Educational Objectives • Assessment and Evaluation of Student Outcomes • Conclusions JIC Presentation

4. Who Are We? JIC Presentation

5. King Abdulaziz University Profile • KAU is Saudi Government financed institution. • Founded in 1967 in Jeddah, 70 Km west of Makah holy city • 18 Faculties/Colleges & 9 Deanships • 110+ Departments offering a diversity of specializations. • 4100+ Faculty members & 2100+ Administrative and support personnel • 95000+ Students. • 30 Scientific Chairs • 6 Centers of Excellence • 16 Scientific Research Centers JIC Presentation

6. College of Engineering Profile • Founded in 1975 G • 2700+ Students • 7000+ Alumni to date • 9 Departments offering 12 programs. • 240+ Faculty members • 100+ Administrative and support personnel JIC Presentation

7. ABET Substantial Equivalency • The 12 Programs started introducing themselves to the world of accreditation and quality education in 2002 through ABET SUBSTANTIAL EQUIVALENCY. • Although this was not A FULL ACCREDITATION, it represented a huge step forward. • This was the only available international evaluation at that time and allowed some deviations from in-land accreditation: “Program is comparable in educational outcomes but may differ in format or method of delivery” Nevertheless the wheel of quality started to rotate JIC Presentation

8. ABET Full Accreditation In 2003, once the successful ABET visit to evaluate the 12 engineering programs was completed, The College started to prepare for the next evaluation visit in 2008 which coincided with: • the introduction of the new outcome-based Engineering Criteria EC 2000, • the phase-out of the substantial equivalency in favor of full accreditation. EC 2000 represented a major paradigm shift influenced by pressures from industry and global competition. It addressed the effectiveness of engineering programs by focusing on assessment & evaluation process that assures the achievement of educational objectives & outcomes. In August 2008 the 12 KAU Engineering Programs obtained the full ABET Accreditation under EC 2000 JIC Presentation

9. Academic Accreditation Unit (AAU) • Established in May 2005 • 4 faculty members (working part time) • Initial mission: Qualify the education system in the Faculty of Engineering to Meet ABET EC 2000 Standards. AAU: Quality Assurance In KAU Engineering Education JIC Presentation

10. ABET TAC Accreditation JIC Presentation

11. ABET • The Accreditation Board of Engineering and Technology , actually known as ABET Inc. • The recognized accreditation entity for college and university programs in applied science, computing, engineering, and technology in the US. • A federation of 30 professional and technical societies in these fields. • For over 77 years, ABET has provided leadership and quality assurance in higher education. • Currently ABET accredits some 2,800 programs in more than 600 colleges and universities inside the US and worldwide. JIC Presentation

12. Elements of ABET Criteria Definitions General Criteria: apply to all programs accredited by an ABET commission. Each program accredited by an ABET commission must satisfy every Criterion in the General Criteria for that commission. Program Criteria: provide discipline-specific accreditation criteria. Programs must show that they satisfy all of the specific Program Criteria implied by the program title. Any overlapping requirements need be satisfied only once. JIC Presentation

13. ABET-TAC Definitions Program Educational Objectives: • Broad statements that describe what graduates are expected to attain within a few years of graduation. Program educational objectives are based on the needs of the program’s constituencies. Student Outcomes: • Describe what students are expected to know and be able to do by the time of graduation. These relate to the knowledge, skills, and behaviors that students acquire as they progress through the program.  JIC Presentation

14. ABET-TAC Definitions (Continued) • Assessment : one or more processes that identify, collect, and prepare data to evaluate the attainment of student outcomes and program educational objectives. Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the outcome or objective being measured. Appropriate sampling methods may be used as part of an assessment process. •  Evaluation: one or more processes for interpreting the data and evidence accumulated through assessment processes. Evaluation determines the extent to which student outcomes and program educational objectives are being attained. Evaluation results in decisions and actions regarding program improvement JIC Presentation

15. ABET-TAC Continuous Improvement Criterion 4 The program must regularly use appropriate, documented processes for assessing and evaluating the extent to which both the program educational objectives and the student outcomes are being attained. The results of these evaluations must be systematically utilized as input for the continuous improvement of the program. Other available information may also be used to assist in the continuous improvement of the program. JIC Presentation

16. Assessment & Evaluation of Program Educational Objectives (PEOs) JIC Presentation

17. Establishment of PEOs Broad statements that describe what graduates are expected to attain within a few years of graduation. They should be consistent with the mission of the institution, the needs of the program’s various constituencies, and ABET criteria. They should be documented (website, publications, posters, etc.) They should be established with the full involvement of all program constituencies. Program constituencies include Alumni, Employers, Students/ parents, faculty, institution management. JIC Presentation

18. Establishment of PEOs -Continued JIC Presentation

19. Assessment of PEOs • Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the objective being measured. • Assessment of PEOs should be carried outside the institution - since you need to evaluate the achievement of the graduates. • You should target your graduates (alumni) and their employers (and not the students or the faculty). • You should assess and evaluate: • Adequacy of PEOs to the needs of your constituencies. • Achievement of PEOs by your graduates. JIC Presentation

20. JIC Presentation

21. Adequacy and Achievement of PEOs • PEOs Adequacy cycle (3 year cycle, twice during an ABET accreditation cycle). It may result in redefinition of PEOs. • PEOs achievement cycle (normally carried in the same time as the adequacy cycle). It may result in modifying the student outcomes or enhancing of their implementations. JIC Presentation

22. Direct Assessment of PEOs • Facts Finding Questionnaire (Versus opinion-based surveys) • Employer survey (every four years) • Alumni survey (every three years) • Archival Data • Placement records (mean time to find a job) • Admissions to and success in graduate programs (if applicable) • Inputs from the Industrial Advisory Board JIC Presentation

23. Direct Assessment of PEOs - Continued Define for each Program Education Objective a set of: • Indicators. • Assessment tools. • Performance criteria for each assessment tool. • Time line for assessment and evaluation. JIC Presentation

24. Assessment and Achievement of Student Outcomes (SOs) JIC Presentation

25. Establishing Student Outcomes • ABET is an outcome-based approach. • Your Student Outcomes should be documented. • They should foster the attainment of your PEOs. • There should be a documented process to review and revise your SOs. • The students should demonstrate achieving SOs before graduation. JIC Presentation

26. JIC Presentation

27. Outcome Elements and Attributes • Development of each outcome into Outcome Elements:different abilities specified in a single outcome that would generally require different assessment measures. • For each outcome element define outcome attributes or indicators: measurable actions that explicitly demonstrate mastery of the abilities specified in an outcome element, at each level of learning. JIC Presentation

28. Design Backward Intended Learning Outcomes of the Lesson Intended Learning Outcomes of the Unit Intended Learning Outcomes of the Course Student Outcomes of the Program Deliver Forward Infusion of Student Outcomes into Courses JIC Presentation

29. Assessment of Student Outcomes • Direct Measures • Course assessment (every semester) • Internship evaluations. • Indirect Measures • Exit interviews • Focus groups (every year) • Exit survey (every semester) • Yearly evaluations JIC Presentation

30. Assessment and Grading Both intend to identify what the students have learned. Grades don’t identify which specific student learning outcomes and at what levels students have learned. Some course grades include additional student behaviors that are not related to student learning outcomes . For example, a grade of B in a mathematics course indicates a student learned an acceptable level of math, but not specific math knowledge or skills or what areas need improvement. Grades need to be clearly linked and aligned to learning outcomes and/or rubrics to be sufficient as direct evidence for assessment purposes. JIC Presentation

31. Key Courses Approach • Assessment of SOs is the task of the program. • The program is required to collect SOs assessment data supported by convincing evidences to demonstrate that the students achieved all SOs. • Each course is responsible to introduce some of these outcomes, enhance the students abilities to achieve them, and collect convincing evidences transferable to the program committee responsible for evaluation and continuous improvement actions. • It is sufficient that each course will do this for only a limited number (2 or 3) of the SOs (Known as critical outcomes, the course is a key course for these SOs) if and only if compulsory courses together cover all outcomes with reasonable redundancy. JIC Presentation 30

32. Course-level assessment of key outcomes There are 3 Alternatives • Rubric evaluation of 3 samples of previously graded student work that covers the major attributes/indicators of each key outcome. • The easiest but the least convincing approach • Rubric-based assessment of a specially designed assignments that covers the major attributes/indicators of a key outcome. • A standard approach • Rubric-based assessment of a reflective Student Portfolio. • A recommended approach - JIC Presentation

33. What is a rubric • A rubric is a descriptive scoring guide composed of • Things you are looking for in an assignment • Guidelines for evaluating each of those things. • A rubric is some kind of scorecard that breaks down a written or demonstrated assignment into manageable, observable pieces. • A rubric is a descriptive scoring/grading tool that is generally used for subjective assignments. JIC Presentation

34. Workshop Assessment Rubric JIC Presentation

35. Conclusions • To get accredited we were required to demonstrate to ABET evaluators that: • WE KNOW WHAT WE DO. • WE KNOW WHY WE DO IT. • WE KNOW HOW TO DO IT. • WE DO IT WELL. • WE CAN PROVE IT. • WE RECEIVE INPUT AND FEEDBCAK. • WE HAVE A PROCESS TO MAKE CONTINUOUS IMPROVEMENTS. • To get accredited you are required to have: • WELL LAID-OUT PROCESS, and • INVOLVEMENT OF THE ENTIRE FACULTY IN THE PROCESS. JIC Presentation

36. وآخر دعوانا أن الحمد لله رب العالمين