Learning Outcomes LOs

1. Learning OutcomesLOs Dr. Gregory J. Maffet NCAAA Consultant Dr. Naser M. Sarhan NCAAA Consultant KFUPM 29-30 January 2014

2. Workshop’s Learning Outcomes (LOs) Attendees at the end of this workshop will be able to: 1. Comprehendthe nature and role of Program and Course learning outcomes in instruction. 2. Alignan understanding relationships between Teaching Methods, Assessments Methods and LOs, 3. Write learning outcomes using the correct format

3. Workshop’s LOs • Summarizethe role of learning outcomes in instruction and assessment. • Why we are writing learning outcomes? • The role of learning outcomes in assessment? (including Rubrics and KPIs) 5. Usethe five domains of learning specified in National Qualification Framework; including the selection of verbs that map to measurable instructional objectives, learning outcomes and assessment.

4. Workshop’s Los • Construct learning outcomes from learning objectives, in order to develop learning outcomes for Programs and Courses. • Mapping learning outcomes

5. 1st Day Sessions

6. Example from your SSR SSRP – Profile, p.12 3. Key Performance Indicators The following KPI’s are adopted to monitor the achievement in accomplishing the Program objectives. • Graduation with a GPA acceptable by Industry • satisfaction at alumni/employer surveys • To build up an instrument park, corresponding to most world class universities • Field training of students must be continuously updated with latest techniques. • Industry-standard computer modeling and interpretation packages must be included in the curriculum • Average time for procuring equipment and instruments.

7. Example from your SSR SSRP. p.9

8. Example from your SSR SSRP – Standard 4, p.36 The learning outcomes taken together become the overall strategy for the degree plan. The department is currently looking into the possibility of setting a standard end-of-Program examination (exit exam) that will assess specific learning outcomes. Verification of learning outcomes comes in the form of feedback from potential employers from work placement SSRP – Standard 3, p.34 Periodically, feedback is sought from alumni and employers. In many occasions, the feedback is used for Program quality improvement almost immediately. The grades of graduating students are reviewed as a way to measure performance.

9. Example from your SSR • The assessment of Program outcomes is done on continuous basis. SSRP, p.22 • Since the Program outcomes are intrinsically related to Program objectives, achievement of Program outcomes is an essential prerequisite of student qualification at graduation. .... Level of achievement of the Program outcomes is periodically measured to examine the extent to which they are met.SSRP, p.24

10. Example from your SSR The intended student learning outcomes are periodically evaluated through various means like public presentations, exams, assignments, projects, etc. Appropriate Program evaluation mechanisms including graduating student surveys, employment outcome data, employer feedback and subsequent performance of graduates are used to provide evidence about the usefulness of intended learning outcomes and the extent to which they are achieved. SSRP – Standard 4., p.38

11. Example from your SSR • Students learning outcomes are evaluated based on HW, quizzes, and exams, and in some cases reports and presentations, are used to measure the student learning outcomes. SSRP, p.31 • Faculty members make sure that students are tested in a manner that allows them to assess the extent to which learning outcomes are met. They maintain a check on their testing procedures. SSRP, p.35.

12. KPIs for Profile Data, NCAAA KPIs

13. Key Performance Indicators: Why? Performance of a higher education institution & its Programs is complex (teaching, research, community…) KPIs summarize performance in key areas  scientifically, rationally, and meaningfully for different stakeholders: Faculty & Staff Students External agencies (NCAAA, employers,…)

14. What are KPIs??? … KPIs = Key Performance Indicators …a measure of performance or achievement …a Key Success Indicator (KSI) …a measure of results and efficiency Quantifiable performance measures used to define success and measure progress toward the achievement of goals. (maybe qualitative via rubrics)

15. KEYis fundamentally important to gain advantage; a make-or-break component for success. Performance when outcomes can be clearly measured, quantified, and easily influenced by the institution or Program. Indicator provides leading information on future performance. (ie., when the gas gauge is on “E” then the leading information tells us that the car’s future performance will be to stop; so the action plan is to get gas immediately) KPI

16. NCAAA Learning & Teaching KPIs 5. Ratio of students to teaching staff (Based on full time equivalents) 6. Students overall rating on the quality of their courses.(Average rating of students on a five point scale on overall evaluation of courses.) 7. Proportion of teaching staff with verified doctoral qualifications. 8. Percentage of students entering Programs who successfully complete first year. 9. Proportion of students entering undergraduate Programs who complete those Programs in minimum time. 10. Proportion of students entering post graduate Programs who complete those Programs in specified time. 11. Proportion of graduates from undergraduate Programs who within six months of graduation are:employed, enrolled in further study, not seeking employment or further study

18. Standard 4 Teaching and LearningKPI for Ratio of Students to Teaching Staff

19. Internal Benchmark We looked previously at the internal KPIs We now think consistently about benchmarks for the same areas “year by year”  to establish data trends and trend-analysis What are the strengths and weaknesses of using the internal average as a benchmark? (e.g. improve to 90% or improve by 10%)

20. Quantitative KPI  Ratio of students to teaching staff Actual Benchmark  25 to 1 (current) Target Benchmark  20 to 1 (goal) Internal Benchmark  24 to 1 (past, 2012) External Benchmark  26 to 1 (KSA) (International) 16 to 1 Interpret this statistical data? (Meaning) Analyze the results? (SWOT? / other)

21. Standard 4 Teaching and LearningKPI for Ratio of Students to Teaching Staff

22. Standard 4 Teaching and LearningKPIs For Percentage Of Students’ Completion Rates

23. Standard 4 Teaching and LearningStudent Satisfaction Survey

24. Standard 4 Teaching and LearningFaculty Satisfaction Survey Complete KPI Template

25. KPI Trend Report KPI  Percentage of students entering undergraduate Programs who complete those Programs in minimum time; 4 years. (NCAAA KPI) Analysis: discussion and evaluation: strengths, recommendations, predictions

26. Profile Trend Report Enrollment Trends and Predictions (Prediction/Actual) Based on previous years, what is predicted? Analysis: Descriptions and Evaluation (What? and Why?) Strengths, Recommendations, & Predictions

27. Profile Trend Report Enrollment Trends and Predictions (Prediction/Actual) Make predications & analyze data Analysis: Descriptions and Evaluation (What? and Why?) Strengths, Recommendations, & Predictions

28. Quantitative KPI  Ratio of students to teaching staff Actual Benchmark  25 to 1 (current) Target Benchmark  20 to 1 (goal) Internal Benchmark  24 to 1 (past) External Benchmark  26 to 1 (KSA) (International) 16 to 1 Interpret this statistical data? (Meaning) Analyze the results? (SWOT? / other)

29. Program Specifications Program KPI and Assessment Table NOTE The following definitions are provided to guide the completion of the above table for Program KPI and Assessment. KPI refers to the key performance indicators the Programs used in the SSRP and are approved by the institution (if applicable at this time). This includes both the NCAAA suggested KPIs chosen and all additional KPIs determined by the Program (including 50% of the NCAAA suggested KPIs and all others). Target Benchmark refers to the anticipated or desired outcome (goal or aim) for each KPI. Actual Benchmark refers to the actual outcome determined when the KPI is measured or calculated. Internal Benchmarks refer to comparable benchmarks (actual benchmarks) from inside the Program (like data results from previous years or data results from other departments within the same college). External Benchmarks refer to comparable benchmarks (actual benchmarks) from similar Programs that are outside the Program (like from similar Programs that are national or international). KPI Analysis refers to a comparison and contrast of the benchmarks to determine strengths and recommendations for improvement. New Target Benchmark refers to the establishment of a new anticipated or desired outcome for the KPI that is based on the KPI analysis.

31. University Comparison Matrix Analysis (Strengths and Recommendations)

32. Introduction • The Design phase of a typical curriculum development process (Analysis, Design, Development, Implementation, Evaluation) is largely concerned with developing clear learning objectives and learning outcomes. • It is important to recognize during this phase that there is a direct relationship between objectives, learning outcomes, teaching strategy / methods & students assessments (Rubrics – KPIs with benchmarking and analysis).

33. Theory Into Practice 5 Questions for Instructional Design • What do you want the student to be able to do? (Outcome) • What does the student need to know in order to do this well? (Curriculum) • What activity will facilitate the learning? (Pedagogy—learning & teaching) • How will the student demonstrate the learning? (Assessment) • How will the teacher know the student has done this well? (Criteria)

34. Learning Outcome are: • Learning outcomes: describe whatlearners are supposed to know, understand, or are able to do at the end of the Program or course. • LOs are based upon: • the needs of the learner (individual & personal). • the needs of society (mission statements). • what the learner should know about a particular subject in order to perform successfully (career and personal lifestyle).

35. LOs are: • What a STUDENTshould learn as a result of a period of specified and supported study. • The ACHIEVEMENTSof the learner rather then theintentions of the teacher. • Focus is directly on Student Performance.

36. LOs are: Formal statements that articulate: • Whatstudents know and are able to do after instruction • Whystudents need to do this -- relevancy

37. LOs are: Are concerned with the learning of the student: STUDENT… PERFORMANCE and ACHIEVEMENT • What the student CAN DO • What the student KNOWS AND CAN DO • What the student UNDERSTANDS OR • COMPREHENDS and CAN DO Must be measurable or observable

38. Objectives vs. LOs • The distinction between learning outcomes and learning objectives is not universally recognized. • Many instructors may find that the term “learning outcomes”describes what they have already understoodby the term “learning objectives.” What is the difference?

39. For NCAAA the difference between course LOs and objectives… • Learningobjectives arestatements of what the teacherintends for the students to learn and are generally part of a teacher-centered approach[are Mission, traditional, teacher or content driven]. • Learningoutcomes arestatements of what the student will KNOW and be able to DO or demonstrateas a result of their learning and are part of a student-centered approach.

40. Objectives vs. LOs • Learning objectives, for example, may outline the material the INSTRUCTOR intendsto cover in the course / Program or the disciplinary questions the class will address. Known as IN-PUTS. • By contrast, learning outcomesfocus on what the STUDENTS know, comprehendand realistically are able to do… [skill performance] by the end of an assignment, activity, class, or course [achievement]. Known as OUT-PUTS.

41. Objectives vs. LOs • learning outcomes, mean focusing on the application and integration of the course content from the perspective of the student. • learning outcomes can more explicitly and directly addressexpectations for student learning.

42. Objectives ------- Outcomes Assessments Measurable Observable What else overlaps?

43. Writing Objectives & LOs Learning objectivescan be written as teacher or curriculum centered content or they can be re-written as student-centered learning outcomes. The teacher will... OR the student will…. Both learning objectives and outcomes must be measurable or observable. One assessment is for teachingand another assessment is for a student’s learning.

44. Examples • Example of a Learning Objective: Students will be taught the basic principles of database searching. [teacher will teach basic principles … ] • Example of a Learning Outcome: Students will be able to apply the principles of database searching in a review of literature. [student will KNOW and APPLY…]

45. Objectives or Outcomes? Which Dentist do you want working on your teeth? Student A: The teacher will instruct the student to know how to successfully drill cavities and repair teeth… Student B:The student earns 100% on the exam for drilling cavities and repairing teeth…. Student C:The student knows how and successfully drills out cavities and repairs teeth… Student D: The teacher successfully taught the student to drill out cavities and repair teeth.

46. Objectives or Outcomes? Which Pharmacist do you want filling your meds? Student A: The teacher will instruct the students to know how to successfully fill medical prescriptions… Student B:The student earns 100% on the exam for filling medical prescriptions…. Student C:The student knows how and successfully fills medical prescriptions… Student D: The teacher successfully taught the student to fill medical prescriptions….

47. Exercise • Please work as group in writing three learning objectives for your Program. • Now, re-write these objectives as LOs • Be prepared to share them and analyze the difference – they will be collected in order to be used latter.

48. NCAAA 10 Minute Break Session 2 Importance & Benefits of LOs Where do LOs come from? Characteristics of LOs LO Process and Levels NQF Learning Domains & LO – Verbs

49. The Importance of LOs 1. LOs build evidence for accountability, accreditation, and for continuous improvement. • Showevidence of how well students learn. • Useevidence for continuous improvement and strategic plans.

50. The Importance of LOs • Know what you are doing… • Know why you are doing it… • Know what students are learning as a result; (key for assessment). • Make improvement changes based on results (research based improvements)