Undergradute Curriculum Assessment/Evaluation Report

1. Undergradute Curriculum Assessment/Evaluation Report Ted Baker CS Faculty Workshop 18 August 2009

2. Overview • Training for instructors • Review of student exit survey responses • Accreditation status summary • Things we need to do Not the most logical order, but the most critical items first.

3. Training for Instructors • Go to www.cs.fsu.edu/ugcc for • approved course objectives; include them. • SMALC outcomes that you must assess (if any) • Rubric you must use • Action plan from prior year • end-of term reports from prior instructors • Go to DOF website for most recent • Attendance statement • ADA statement • Honor code statement • Send syllabus to DUGS at start of semester • Preferably, in time for review before start of classes • We will soon (hopefully) have a web-based system for uploading the syllabi.

4. Other Beginning-of-Term Matters • Confer with instructor from prior term, if possible. Get advice, materials. • Find out who is teaching the course for distance/PC campus, and communicate with them. • If you want to change textbooks, propose the change to the UGCC (but try for stability). • Beware of university and departmental requirement for final examination in all UG courses, given during final exam week. • If you have a TA, be sure to supervise, observe, mentor, and evaluate.

5. Issues from Exit Survey • See http://www.cs.fsu.edu/ugcc/smalcs09/surveys.html • Today, see spreadsheet.

6. Issues raised by student advisory panel • See http://www.cs.fsu.edu/ugcc/minutes/ugsac_02_10_09.html • Discrete math sequence is not working • Some math instructors are not teaching the courses well • Students are not motivated, because they don’t see the applications • We are not using the material enough in our courses soon enough, so student forget what they did learn • Significant repetition between and within Discrete Math courses, COP4530, and COP4531 • Many students seem to be discourage dfrom major by the DM sequence • Have CS faculty teach discrete math • Integrate DM II into CS courses where it is used

7. More • COP4531 tries to rush through too many algorithms, without adequately developing the underlying design techniques • Courses that are logically connected, as prerequisites, need to be offered in contiguous semesters, so that students don’t forget what is needed from the preceding course • Overlapping course pairs not well coordinated and not consistent: sometimes duplication, and sometimes gaps. Need more instructor coordination, more consistency. • COP3014-COP3330 • COP3330-COP4530 • COP4530-COP4531

8. More • Problems with instructors who are new to a course. Suggest less turnover of instructors in a course. • Excessive use of Powerpoint makes lectures move to fast, too hard to follow. Suggest more hand-writing to engage students. • Need more exposure to event-driven programming • Electives are a problem. Department should poll students to see what electives they want. • Need more room for electives in the curriculum (less required courses). • Department seems to put more priority on graduate education than undergraduate. • Too much overlap between CDA3101 and CDA3100.

9. More • COP4710 requires students to use Java. This ends up being duplicative of the Java course. • Intro to Unix and Unix Tools courses sometimes overlap a lot, and vary a lot in content from term to term. • We are not making good use of recitation meetings. They should not just be another lecture, perhaps by the TA. They should be used to interact with students, with a focused activity. • COT4420 notes need updating. • More use of IDEs (along with command line) in courses beyond the introductory level. • COP3014 might attract more students if there were more interesting (e.g., GUI) assignments in it.

10. Accreditation Status • Interim report submitted in June, now being reviewed by ABET/CAC representative. • You can read the report if you go to http://www.cs.fsu.edu/ugcc/interim.pdf. • Username/password = smalcs/smalcs • If this report is approved (in July 2010), we will be approved for another 4 years. • Otherwise, we would probably be asked to do another interim report in June 2011.

11. Issues in the Report • Coverage of social and ethical issues related to computing. • Adequacy of elective offerings. • Sufficiency of university resources. • Outcomes and assessment process.

12. Ethical & Social Issues • Added material to DB and SE courses • Created new course, CIS4250 • This addresses the ABET complaint, but • DB and SE course treatment is now duplicative • It can’t be phased out until grandfathered students graduate, • and we find a way to handle distance students. • Distance students can’t do oral debates. • Oral debates can’t be made optional for CIS4250, since it is an “oral competency” course for SACS accreditation purposes. • One idea: • Create another ethics course, related to computing (maybe only 2 hrs, maybe only DL, maybe only taught by adjuncts) • Students who take that course must also take SPC2600 to cover oral competency requirement.

13. Elective Offerings • This is partly a student perception problem. • Report shows that we actually are offering more electives than the institution of the chair of the team that made the criticism. • Remaining issues: • Many are not practically available to most students, due to tough prerequisites (dual-listed graduate courses, other electives as prerequisite, senior-level courses as pre-requisite) • Distance students have fewer options • What we are doing: • Offer popular ones, without daunting pre-requisites, more often. • Things we might do: • Allow students to take some electives out of CS? • Work with other Florida universities to share distance course offerings?

14. Adequacy of Resources • ABET reviewers probably thought they were helping the program, by putting pressure on the FSU administration to replace lost faculty lines. • Our interim report explains effects of current economic distress on the University, and that • we are lucky to have suffered no layoffs, and to have been allowed to replace Ken Baldauf and Chrissy. • I have explained this also, in person, to the ABET reviewer. • We can hope that ABET will appreciate putting further pressure on FSU will not help.

15. Assessment • We have expanded our assessment process. • Tried to combine ABET requirements with SACS/SMALC requirements. • Doubled number of SMALC outcomes • to cover min. set of ABET/CAC-required outcomes • Added end-of-term instructor course reports • to provide more useful information for program improvements • Added focus-group meeting with undergraduate students • Added more formal review and reporting process • including this report, to the workshop

16. Remaining Problems • Faculty education. • Everyone needs to understand the process and the goal. • Therefore, this talk and more to come. • Increased workload (“death by assessment”) • Therefore, proposals to come, to reduce load. • Fundamental fallacy in ABET model • We can still make a good faith effort.

17. “Death by Assessment” • Gathering the assessment data is a lot of work. • This is distracting us from other essential duties, including the evaluation and planning that are supposed to be driven by the assessment process. • Faculty members still do not all understand what is expected. • The DUGS is overwhelmed by the workload. • The DUGS has already spent more than a man-month at this, dunning for data, meeting with people to explain what is needed, extracting files from e-mails and Web pages, editing, renaming to a standard system, copying into directories, reformatting into .pdf, renaming and reformatting them, creating web links to the files, etc. Moreover, thhis manual work is very error-prone. • Much of the supporting data files still need to be populated on our web site, to which the interim report refers. • The DUGS has been meeting with the webmaster since late fall 2008 to develop an on-line tool that would automate most of this, but the process seems to have stayed on the back burner.

18. Assessement Processes • State Mandated Academic Learning Compacts • We specify some specific minimal achievements (“student educational outcomes”, and “program outcome”) for our graduating students, which we must assess and report every year. • ABET • Starts similarly to SMALC, but goes further. We must demonstrate that we use the data from assessments as the basis for program improvements.

19. ABET Model • Engineering method should be applied to the process of education. • We define a set of desired outcomes for our educational program. • We gather data on how well those outcomes are being achieved. • We review and evaluate the data, which drives our plans to improve our program. • Not just the instructional process, but also the meta-processes (outcome statements, assessment methods, evaluation process, etc.)

20. Fallacy in the Model • Assumes we have a repeatable process. • So that we can vary one parameter, and assess how this has improved/hurt the program. • This seems to be called “regression-discontinuity design” by experimental design theorists. • We have different people teaching courses, sometimes in the same term. • Students coming into courses also have had different experiences in preceding courses. • There are so many differences in courses and students that we can’t isolate the effects of any specific change with reliability.

21. What can we do? • Try to make the process more repeatable? • Not rotate instructors so frequently. • Communicate more between successive instructors, reusing text, slides, etc. • Take a more holistic view of the program assessment, recognizing that regression-discontinuity experimental method is not feasible in this environment. • Turn in syllabus before the term, and have it reviewed for consistency.

22. How to reduce workload • Increase priority of on-line data collection. • Collect SMALC data only once a year. • Reduce set of courses in which SMALC data is collected, to a few capstone courses. • Collect instructor reports in every course, and rely more on them for directing program improvements. • Extend survey to gather more feedback from alumni. • Continue • Do thorough review of assessment data less frequently, rotating through courses and outcomes on a multi-year cycle. • Which ones to start with? The math/theory sequence?

23. Remaining Instructor Responsibilities • SMALC data collection, in some courses. • End-of-term report, in every course. • Reviewing SMALC and other data, and making specific recommendations to DUGS and/or UGCC.

24. Other UG Curriculum Issues • New BA degree requirements • Relaxing prerequisites, to make courses more accessible

25. New BA Requirements • Tougher than BS; requires 12 more courses than BS, in humanities. • 2 students graduated with BA in 2007, none in 2008 & 2009. • ABET considers this a separate degree; charges double fees, sends extra accreditation visitor; wants examples of student transcripts to prove program is vital. • We can’t provide example transcripts. • We decide to drop ABET accreditation of the BA degree. • This opens up possibilities • To attract more students to CS • By eliminating for the BA some requirements imposed solely to satisfy ABET/CAC

26. Relaxing Prerequisites • CS curriculum discourages cross-disciplinary studies, due to very deep prerequisite structure. • This also causes scheduling problems for students, and limits their choices of electives. • It is not clear that we actually depend on all these prerequisites. • It is time to review all prerequisites, to see if they are essential.

27. Some changes, already • Java for non-majors • dropped C++ prerequisite • C++ for non-majors • Reduced math pre-requisite • COP4610 • Reduced CDA3101 to co-requisite or instructor permission • Security courses • COP4530 replaced by CDA3100 • Can we go further?