Outcomes Assessment in Computer Science

1. Outcomes Assessment in Computer Science Robert Lingard Assessment Coordinator August 27, 2009

2. Outline • The Importance of Assessment • Problems with Assessment • Goals of an Assessment Process • An Approach to Assessment • Applying the Software Engineering Paradigm • The Annual Assessment Process at CSUN • Conclusions

3. Importance of Assessment • Assessment is required by accrediting boards (e.g., ABET). • Those with vested interests (financial and otherwise) in education are demanding accountability. • Without assessment it is difficult to know what changes to make to improve learning.

4. Problems with Assessment • Pressure for assessment can cause a “rush to assessment,” and produce meaningless results. • Pressure to “close the loop” can cause premature decisions regarding program changes. • There is a tendency to only assess high performance areas in order to “showcase” a program. • Results are often not validated. • The effects of changes are often not assessed.

5. Goals of an Assessment Process • To facilitate the direct assessment of student learning • To measure student retention and the ability to apply what they have learned • To provide mechanisms to ensure the continuity of the process • To be an efficient and natural extension of normal operations • To satisfy the requirements of ABET and the University for assessment

6. Approach to Assessment • “The first and only goal [in education is to] teach for long-term retention and transfer.” – Diane Halpern • The best assessments of student learning are based on direct measures of achievement. • How well students have learned is best assessed by looking at their ability to apply what they learned earlier to new situations.

7. Requirements for a Complete Assessment Process • It must be comprehensive – must cover the full range of learning outcomes • It must include multiple judgments – multiple sources of evidence must be used • It must include multiple dimensions – different facets of student performance must be included • It must collect direct evidence – direct measures of student attainment must be used

8. The Software Engineering Paradigm • Understand the problem • Plan a solution • Carry out the plan • Make sure the solution is correct

9. Understanding the Problem • Before beginning assessment, the first step is to decide what to assess – i.e., pick the most important things to assess. • Collecting information from faculty, students, alumni, and employers can give hints as to where learning problems exist. • Surveys, faculty meetings, and student interviews are ways of collecting this information.

10. Planning a Solution • Once a learning outcome is selected, a plan for conducting the assessment must be developed. • It must be determined how the assessment will be done – i.e., embedded questions in exams, a standard instrument, etc. • How the results will be evaluated must also be determined – e.g., rubrics must be developed. • It must also be determined who will do the various tasks required and when they will be done.

11. Carrying Out the Plan • The steps of the plan must be carried out, and the plan must be monitored to ensure successful completion of the assessment. • Someone must be designated as the lead, and this person has the responsibility for monitoring the plan.

12. Making Sure the Solution is Correct • Assessment results obtained must be validated – this step is often omitted. • One way to validate results is to make comparisons among several independent assessments using different approaches. • Failure to validate the results can result in program changes that may not be appropriate. • After program changes are made the effects of the changes must be assessed.

13. Reliability and Validity Reliable & valid Valid but not reliable Reliable but not valid

14. Questions to Ask about an Assessment • Was the instrument we used reliable? • Did we measure what we intended to measure? • If we used a sample, was the sample size large enough to be confident in the results? • If we are comparing the results of two assessments, are the differences significant?

15. Iteration • Like software engineering this process is iterative. • Once program changes have been made, they must be assessed to determine whether the desired result has been accomplished. • Analysis of this reassessment might indicate the need for further changes.

17. The Annual Assessment Process at CSUN • Department forms Assessment Committee The program is divided into seven areas and a coordinator is chosen for each area. These coordinators constitute the Assessment Committee. • Assessment Committee recommends outcomes/objectives to be assessed The Assessment considers results from previous assessments, informal assessments, and the length of time since particular outcomes have been assessed to determine the current set to be assessed. • Department approves assessment goals and assessment plans are prepared After department approval the Assessment Committee decides on an assessment approach and develops a schedule of activities.

18. The Assessment Committee Consists of the Program Area Coordinators

19. The Annual Assessment Process at CSUN • Department forms Assessment Committee The program is divided into seven areas and a coordinator is chosen for each area. These coordinators constitute the Assessment Committee. • Assessment Committee recommends outcomes/objectives to be assessed The Assessment considers results from previous assessments, informal assessments, and the length of time since particular outcomes have been assessed to determine the current set to be assessed. • Department approves assessment goals and assessment plans are prepared After department approval the Assessment Committee decides on an assessment approach and develops a schedule of activities.

20. Student Learning Outcomes • Demonstrate an understanding of algorithms and data structures. • Demonstrate an understanding of computer organization and architecture. • Demonstrate an understanding of programming language concepts and knowledge of a variety of programming language paradigms. • Demonstrate proficiency in using a high-level computer language. • Demonstrate an ability to apply mathematical skills appropriate to the computer science discipline. • Demonstrate an awareness of the evolution and dynamic nature of the foundational core of computer science. • Demonstrate proficiency in collecting, analyzing, and interpreting data and information. • Demonstrate a problem solving ability. • Demonstrate an understanding of emerging technologies and a working knowledge of currently available software tools. • Demonstrate an understanding of the principles and practices for software design and development. • Be able to apply the principles and practices for software design and development to real world problems. • Be able to effectively communicate orally. • Be able to effectively communicate in written form. • Be able to work effectively on a team. • Demonstrate knowledge of the social impact of computing. • Demonstrate an understanding of the professional and ethical considerations of computing. • Demonstrate the knowledge and capabilities necessary for pursuing a professional career or graduate studies. • Recognize the need for, and show an ability for, continuing professional development.

21. Program Educational Objectives • A few years after graduation, graduates of the computer science program will: • Be able to apply the principles of computer science, mathematics, and scientific investigation to solve real world problems appropriate to the discipline • Be able to apply current industry accepted computing practices and new and emerging technologies to analyze, design, implement, and verify high quality computer-based solutions to real world problems. • Exhibit teamwork and effective communication skills. • Be able to positively and appropriately apply knowledge of societal impacts of computing technologies in the course of career related activities. • Be successfully employed or accepted into a graduate program, and demonstrate a pursuit of lifelong learning.

22. The Annual Assessment Process at CSUN • Department forms Assessment Committee The program is divided into seven areas and a coordinator is chosen for each area. These coordinators constitute the Assessment Committee. • Assessment Committee recommends outcomes/objectives to be assessed The Assessment considers results from previous assessments, informal assessments, and the length of time since particular outcomes have been assessed to determine the current set to be assessed. • Department approves assessment goals and assessment plans are prepared After department approval the Assessment Committee decides on an assessment approach and develops a schedule of activities.

23. Sample Assessment PlanLearning Outcome: Be able to effectively communicate in written form • Tool(s) to be Used in the Assessment: • The Writing Proficiency Exam (WPE) • Term Papers written in Comp 450 • Assessment Activities to be Performed, When and by Whom: • Establish minimum and average WPE score values considered to be acceptable for each version of the WPE. (E.g., 3% better than the average scores over the past 5 years.) – May 2005, by the Software Engineering Course Group. • Collect the WPE scores for all Computer Science students over the last 5 years (noting which version was taken). – September 2005, obtained from Institutional Research by the Software Engineering Course Group. • Statistically analyze the WPE scores relative to past scores of similar students. – October 2005, by the Software Engineering Course Group. • Create a set of criteria (rubric) for assessing the quality of student written work, including the setting of standards for acceptability. – November 2005, by the Software Engineering Course Group. • Gather the term papers written by all students currently taking Comp 450. – December 2005, obtained from the Comp 450 instructors by the Software Engineering Course Group. • Select a sample of the term papers of graduating students and assess them according to the established criteria. – January 2006, by a committee selected by the department for the assessment of written communication. • Prepare an assessment summary report for the department including recommendations for program improvement. – February 2006, by the Assessment Committee.

24. The Annual Assessment Process at CSUN • Department approves assessment plans and assessments are conducted Program Area coordinators ensure that the planned assessment activities are completed and that a final report is prepared • Assessment are analyzed and program changes are recommended The Assessment Committee analyzes the results of the completed assessments and makes recommendations for program changes • Department reviews recommendations and makes program changes that are determined appropriate At a department meeting the recommendations of the Assessment Committee are discussed and proposed program changes are made only if there is approval from the department as a whole.

25. Sample RubricLearning Outcome: Be able to effectively communicate in written form

26. The Annual Assessment Process at CSUN • Department approves assessment plans and assessments are conducted Program Area coordinators ensure that the planned assessment activities are completed and that a final report is prepared • Assessment are analyzed and program changes are recommended The Assessment Committee analyzes the results of the completed assessments and makes recommendations for program changes • Department reviews recommendations and makes program changes that are determined appropriate At a department meeting the recommendations of the Assessment Committee are discussed and proposed program changes are made only if there is approval from the department as a whole.

27. Sample Assessment Results Learning Outcome: Demonstrate an understanding of the principles and practices for software design and development

28. The Annual Assessment Process at CSUN • Department approves assessment plans and assessments are conducted Program Area coordinators ensure that the planned assessment activities are completed and that a final report is prepared • Assessment are analyzed and program changes are recommended The Assessment Committee analyzes the results of the completed assessments and makes recommendations for program changes • Department reviews recommendations and makes program changes that are determined appropriate At a department meeting the recommendations of the Assessment Committee are discussed and proposed program changes are made only if there is approval from the department as a whole.

29. Sample Program ImprovementsLearning Outcome: Demonstrate an understanding of the principles and practices for software design and development • The following recommendations for program improvement were made based on the results of the assessment: • Modify the course objectives for introductory computer science courses to include an introduction to software engineering concepts. • STATUS: This was approved by the department and the members of the Software Engineering Course Group will provide information to the Fundamental Concepts Course Group to assist them in appropriately revising the course objectives for the introductory courses. • Modify the course objectives for elective courses with software engineering projects to include the reinforcement of software engineering concepts. • STATUS: This was approved by the department and the members of the Software Engineering Course Group will provide information to those teaching elective courses to assist them in appropriately revising the course objectives to reinforce software engineering concepts. • Add a senior software engineering design project as a require for graduation. • STATUS: After extensive discussions by the department regarding ways of implementing such a requirement, a committee was formed to develop a recommended approach and report back to the department for approval.

30. Conclusions • The process has been used successfully for program assessment. • It has been embedded into the normal operations of the department and become accepted by faculty. • Since all faculty are involved, no one has become overburdened. • It has provided an effective means for directly assessing student learning with a focus on retention and transfer. • More attention is needed in the areas of validating results and ensuring the reliability of the assessment instruments used.