Outcome Assessment: A Seven-Year Study

1. Outcome Assessment: A Seven-Year Study Professor David G. Meyer School of Electrical and Computer Engineering

2. Introduction • A proven method for satisfying the Accreditation Board for Engineering and Technology (ABET) “Criterion 3” requirements is the formulation of outcomes specific to “core” courses in a curriculum, which are tied to the program outcomes. • This presentation summarizes a 7-year study on formulating and assessing course-specific outcomes

3. Basic Questions Addressed • Formulation of content-specific learning outcomes that can be consistently and quantitatively assessed • Formulation of effective outcome assessment instruments along with mechanisms to determine outcome demonstration thresholds • Formulation of grading strategies that incorporate outcome demonstration thresholds yet produce results consistent with prior (accepted) grading practices (i.e., course GPA in range of approximately 2.5 – 2.8 on a 4.0 scale) • Formulation of outcome remediation strategies that are both fair and efficient

4. Edict from the ABET Visitors “We want to see evidence of failing grades assigned for cases in which students [who would otherwise be passing but] failed to successfully demonstrate one or more course outcomes.” Interpretation: We can’t just say that, because a student earned a total score above the “course passing threshold”, that he/she has successfully demonstrated all the course outcomes. Also, application of outcome assessment should produce some form of meaningful and measurable difference.

5. Major Issues • Balance • fairness to students • providing students with a sufficient number of opportunities to demonstrate course outcomes • ensuring that students’ own work is used as the basis for outcome demonstration success • overhead for instructor • keeping the incremental workload associated with outcome assessment and tracking to a minimum • keeping the outcome assessment process “contained” within a given term/semester

6. Major Issues • Evaluation Instruments • exams (whole or question subsets) • quizzes (written/oral) • homework assignments • labs • papers/projects/presentations • Passing Thresholds • static (what absolute value?) • dynamic (what algorithm?) • consistent with grading cutoff (A-B-C-D) thresholds

7. Major Issues • Assigning course grades consistent withproven prior practice, yet reflecting meaningful application of outcome assessmentthresholds • Achieving consistency • semester-to-semester • professor-to-professor • course-to-course • school-to-school • institution-to-institution

8. Sample Course-Specific Outcomes ECE 270 – Introduction to Digital System Design • an ability to analyze static and dynamic behavior of digital circuits • an ability to represent Boolean functions in standard forms, to map and minimize them, and to implement them as combinational logic circuits • an ability to use a hardware description language to specify combinational logic circuits, including various “building blocks” such as decoders, multiplexers, encoders, and tri-state buffers • an ability to design and implement arithmetic logic circuits • an ability to analyze, design, and implement sequential circuits and use a hardware description language to specify them • an ability to design and implement a simple computer based on combinational and sequential building blocks KEY: Modularization of course material

9. Sample Outcome Assessment Exam Primary Assessment of Outcome 2 for CN: 0000-X X 1. Most minimization methods are based on a generalization of… 2. The fact that any combinational logic circuit can be realized using only... 3. The complement of this function is represented by the ON SET... 4. The dual of this function is represented by the ON SET... 5. This expression is an example of... 6. The XOR property listed below that is NOT true is... 7. A NOR gate is logically equivalent to... 8. A circuit consisting of _ is logically equivalent to... 9. This circuit exhibits the following type of hazard... 10. The number of prime implicants is... 11. The number of essential prime implicants is... 12. The number of non-essential prime implicants is... X 13. The cost of a minimal sum of products realization of this function is... X 14. The cost of a minimal product of sums realization of this function is... X 15. The fewest number of 2-input NAND gates needed to realize this function is... X 16. The fewest number of 2-input NOR gates needed to realize this function is... 17. The fewest number of 2-input OD NAND gates needed to realize this function is... 18. The number of pull-up resistors required is... 19. The simplest realization of the function depicted by this circuit is... X 20. The minimum SoP function realized by this circuit is... X 21. The ON-SET of the function realized by this circuit is... X 22. The minimum PoS function realized by this circuit is... X 23. Realization of this same function using only 2-input NOR gates would require... X 24. Steady-state analysis of the function realized by this circuit predicts... 25. The dynamic behavior of the circuit does NOT match the steady-state analysis... Number of correct answers: 15 (out of 25 questions)

10. Sample Outcome Demonstration Report ECE 270 Course Outcome Report for: 0000-X, enrolled in Lab Division 5 Outcome 1: An ability to analyze static and dynamic behavior of digital circuits Outcome 1: Primary Assessment Score = 9/ 25 = 36.0% Passing Threshold = 52.0% Outcome 1: Sorry - this outcome was not demonstrated Outcome 2: An ability to represent Boolean functions in standard forms... Outcome 2: Primary Assessment Score = 15/ 25 = 60.0% Passing Threshold = 45.0% Outcome 2: Congratulations - this outcome was successfully demonstrated Outcome 3: An ability to use a hardware description language to specify combinational... Outcome 3: Primary Assessment Score = 15/ 25 = 60.0% Passing Threshold = 50.0% Outcome 3: Congratulations - this outcome was successfully demonstrated Outcome 4: An ability to design and implement arithmetic logic circuits Outcome 4: Primary Assessment Score = 12/ 25 = 48.0% Passing Threshold = 54.0% Outcome 4: Sorry - this outcome was not demonstrated Outcome 5: An ability to analyze, design, and implement sequential circuits... Outcome 5: Primary Assessment Score = 12/ 25 = 48.0% Passing Threshold = 52.0% Outcome 5: Sorry - this outcome was not demonstrated Outcome 6: An ability to design and implement a simple computer Outcome 6: Primary Assessment Score = 15/ 25 = 60.0% Passing Threshold = 46.0% Outcome 6: Congratulations - this outcome was successfully demonstrated You will need to retake the outcome(s) you have missed on the Final

11. Outcome Demonstration Summary Report ECE 270 PRIMARY OUTCOME ASSESSMENT REPORT Outcome 1 -- Avg Score: 67.4% Threshold: 52.0% Took: 205/206 = 99.5% Passed: 179/205 = 87.3% Failed: 26/205 = 12.7% Outcome 2 -- Avg Score: 59.0% Threshold: 45.0% Took: 203/206 = 98.5% Passed: 167/203 = 82.3% Failed: 36/203 = 17.7% Outcome 3 -- Avg Score: 64.9% Threshold: 50.0% Took: 203/206 = 98.5% Passed: 175/203 = 86.2% Failed: 28/203 = 13.8% Outcome 4 -- Avg Score: 67.6% Threshold: 54.0% Took: 203/206 = 98.5% Passed: 171/203 = 84.2% Failed: 32/203 = 15.8% Outcome 5 -- Avg Score: 66.2% Threshold: 52.0% Took: 200/206 = 97.1% Passed: 171/200 = 85.5% Failed: 29/200 = 14.5% Outcome 6 -- Avg Score: 64.0% Threshold: 46.0% Took: 200/206 = 97.1% Passed: 169/200 = 84.5% Failed: 31/200 = 15.5% Summary of Outcome Demonstration: Number of students who demonstrated 0 outcome(s) on the primary assessments: 2 Number of students who demonstrated 1 outcome(s) on the primary assessments: 5 Number of students who demonstrated 2 outcome(s) on the primary assessments: 9 Number of students who demonstrated 3 outcome(s) on the primary assessments: 12 Number of students who demonstrated 4 outcome(s) on the primary assessments: 22 Number of students who demonstrated 5 outcome(s) on the primary assessments: 51 Number of students who demonstrated 6 outcome(s) on the primary assessments: 105

12. Refinements • Early (discarded) attempts • fixed passing threshold (60%) on weighted sum of (selected) lab, homework, and exam scores • fixed passing threshold (60%) on primary assessment, remediation homework, and final assessment • Main problems • basically everyone “passed” – strategy did not prove to be effective • impossible to ensure that students’ own work was being evaluated on labs and (remediation) homework

13. Refinements • Restricting outcome assessment to exams only • fixed passing threshold (60%) on primary and final assessments • use of “conditional failure” grade for those who would otherwise be passing, with 3rd attempt for remediation of failed outcome(s) the following semester • Main problems • fixed passing threshold became a significant factor: hard (“impossible”) to write exams that produce a predictable mean/distribution • lack of containment within term (“overhead leakage”) • grade inflation due to allowing final assessment to be used as replacement score for all N outcomes

14. Refinements • Incorporation of dynamic thresholds • usedon each primary and final assessment, calculated as mean – standard deviation, with bounded range of 40% to 60% (choice based on empirical study of data sets in multiple courses over multiple terms) • must pass at least 50% of N (total) outcomes on the primary assessments to qualify for a passing grade • final assessment can be used for remediation and/or improvement (replacement) of score on up to N/2 outcomes • self-contained (no “overhead leakage” to following semester)

15. Refinements • Controlling grade inflation • added limited opportunity for grade improvement on final assessment • effective outcome score = weighted average of primary and final assessments (60-40) • number of outcomes re-taken limited to half • must have passing grade to qualify for final • grade inflation associated with grade improvement on final reduced by about an order of magnitude

16. Refinements • Reducing outcome-induced failures • students with grades as high as “B” were still failing course due to failure to demonstrate all course outcomes on two attempts (primary + final) • award grade of “I” or “E” if otherwise have passing grade after final but still need to satisfy one or more outcomes • remediation exam(s) taken subsequent semester of enrollment at scheduled exam times with rest of class • if fail this third remediation attempt on any given outcome, receive failing grade and re-take course • some “overhead leakage” reintroduced, but minimal

17. Results Trial 4 – Static thresholds, N replacement, 3 tries Trial 5 – Dynamic thresholds, N/2 replacement Trial 6 – Dynamic thresholds, no replacement Trial 7 – Dynamic thresholds, weighted average

18. Results Trial 4 – Static thresholds, N replacement, 3 tries Trial 5 – Dynamic thresholds, N/2 replacement Trial 6 – Dynamic thresholds, no replacement Trial 7 – Dynamic thresholds, weighted average

19. Results Desired range, based on proven prior practice Trial 4 – Static thresholds, N replacement, 3 tries Trial 5 – Dynamic thresholds, N/2 replacement Trial 6 – Dynamic thresholds, no replacement Trial 7 – Dynamic thresholds, weighted average

20. Lessons Learned – A “Delicate Balance” • Assigning course grades consistent with proven prior practice • Providing incentives for students to successfully demonstrate outcomes, yet not causing grade inflation and/or excessive instructor workload • Establishing reasonable, meaningful thresholds for outcome demonstration success that are decoupled from “exam difficulty” • Determining a fair level of pass/fail “filtering” based on outcome demonstration success

21. Some Remaining Questions • What is an appropriate amount of outcome-based “filtering” for various kinds of courses? • currently averaging about 10-15% in lower division courses • is this too much, too little, or about right? • What is an appropriate balance between number of attempts allowed to demonstrate an outcome, and the incremental workload associated with managing this effort? • currently offer three attempts • is this too much, too little, or about right?

22. Summary and Conclusions • What may appear to make the most sense “on paper” may not be the “best” strategy when all factors are considered • Finding an “optimal” strategy is non-trivial • Different kinds of courses (e.g., “content-intensive” vs. “design-intensive”) require different outcome assessment instruments, thresholds, and remediation strategies

23. Senior Design Outcomes and Assessment • An ability to apply knowledge obtained in earlier coursework and to obtain new knowledge necessary to design and test a system, component, or process to meet desired needs – design component report • An understanding of the engineering design process – individual lab notebook • An ability to function on a multidisciplinary team – project specific success criteria (functionality) • An awareness of professional and ethical responsibility – professional component report • An ability to communicate effectively, in both oral and written form – formal design review, final written report, and final presentation

24. Outcome Tracking Results forDigital Systems Senior Design 2 Tablet PCs per team TCSP sessions initiated HTML notebooks initiated 1 Tablet PC per team Outcome 1 – design component report Outcome 2 – individual lab notebook Outcome 3 – project specific success criteria Outcome 4 – professional component report Outcome 5 – technical communication skills

25. Summary and Conclusions • Have the “basic questions” been answered? • Formulation of content-specific learning outcomes that can be consistently and quantitatively assessed YES • Formulation of effective outcome assessment instruments along with mechanisms to determine outcome demonstration thresholds YES • Formulation of grading strategies that incorporate outcome demonstration thresholds yet produce results consistent with prior (accepted) grading practices (i.e., course GPA in range of approximately 2.5 – 2.8 on a 4.0 scale) YES • Formulation of outcome remediation strategies that are both fair and reasonably efficient YES

26. Discussion Questions Should outcome assessment strategies be more widely deployed across the ECE curriculum? What are some successful assessment methods used in industry that could translate to the academic environment?