Evaluation & Assessment in the Undergraduate Curriculum

1. Evaluation & Assessmentin the Undergraduate Curriculum Kris Stewart NPACI Education Center on Computational Science and Engineering San Diego State University http://www.edcenter.sdsu.edu/ Professor, Computer Science http://www.stewart.cs.sdsu.edu/ Assessment as Collaboration

2. What is Computational Science? Computational Science Teamwork and Collaboration Science Discipline Physics, Chemistry, Biology, etc. Computer Science Hardware/Software Applied Mathematics Numerical Analysis, Modeling, Simulation

3. What is Computational Science?Recognized as separate discipline E.g. Society for Industrial and Applied Mathematics http://www.siam.org/cse/ and Osman Yassar, IEEE Computing in Science & Engineering http://www.cps.brockport.edu Should CSE be a separate major?(Computational Science [no, too general] or Computational Physics [yes, natural extension]) Should it be a separate department?(How to support interdisciplinarity?)

4. Kris’ Faculty Background(Kris Stewart, Director, San Diego State University, California State University System) • Numerical Analyst* led to • Supercomputing and Undergraduate Education (SUE**) led to • Supercomputing Teacher Enhancement Program (STEP***) led to • Education Center on Computational Science & Engineering (EC/CSE) *MS/CS SDSU 1979, JPL 1981, PhD UNM 1987, SDSU 1986**UCES (DoEnergy 1994) *** Smithsonian Research Collection (1996)

5. UCES Paradigm(thanks Tom Marchioro and the “crew”, 1994?) Previous exposure to “assessment”

6. 1998/99 Assessment by LEAD Assessment as a Collaboration • Background • Workshop in Wisconsin April 1997 to learn about assessment and make it real • NPACI starts 1 October97 • EC/CSE requested assessment for 1998 • Preparation for SDSU Campus Visit by Baine & Julie • Discussions at SC98- SuperComputing Fall98 Orlando • Email to SDSU faculty gather attitudes

7. Grand Challenges for HPC (Stewart & Zaslavsky, SC98) • Faculty system of rewards does not encourage teaching innovations • Lack of awareness of HPC technologies already used in research or teaching for different fields • Faculty & students unaware of benefits and accomplishments of HPC • HPC technologies considered too complex/inaccessible for undergraduate instruction • Sequential HPC-related curricula is absent • Curricula using very large data sets not widely available • Adjust to different learning styles when material is complex • Variety of platforms/software leads to fragmented curricula • School administration/support staff not ready for HPC • Specs of computers and networks below user expectations • We had been thinking about this (based on April 97 LEAD Workshop in WI)

8. Building the Community of Faculty • These challenges are people-centric, not technology-centric and of interest to the community • Systemic Change requires understanding the system and working within it • Empower teachers (find the time), recognition (from chair/dean), support (student assistants)

9. Faculty skepticism: Convincing evidence that computer-based tools enhance teaching process? Knowledge of modern computational methods and availability? Incentive from department and insufficient tech support? LEAD Interviews with V.P., Deans, Chairs Faculty Fellows program identified as a target OUTCOMES: our local infrastructure at SDSU took us more seriously survey instruments refined SWB value recognition Interviews on SDSU CampusLEAD applying info from email surveys (Spr 99)

10. Assessment not just requirement Rather, found to be • vital tool to assist in clarifying student and faculty needs • improve prioritization skills • validation of focus on human factors to integrate HPC (modeling & visualization) into undergrad curriculum

11. California Education Infrastructure • K12 Education (standards based, performance based) • Community Colleges (Freshman/Sophomore) • Vocational • University preparation • California State University System (23 campuses) • University of California (9 campuses, Merced in 2004) • Independents (Stanford U., CalTech, U. Southern California)

12. California Education Infrastructure(testbed for change) IMPAC http://www.cal-impac.org/Intersegmental Major Preparation Articulated Curriculum ProjectCommunity Colleges and Four Year Universities Charter Schools (Preuss School) preuss.ucsd.eduDr. Rozeanne Steckler and Dr. Mike Bailey“Fostering Scientific Curiosity in All Children”San Diego Supercomputer Center

13. Involving University FacultyInfrastructure for Change • NPACI/SDSU Faculty FellowsLocal Support from College Deans and Department Chairs (participation buy-in and faculty recognition) • SDSU Academic Advisors (across disciplines) • Professional Meeting (SC2001, SIAM, ACM, SIGCSE, your suggestions?)

14. Grand Challenges for HPC (Stewart & Zaslavsky, SC98) • Faculty system of rewards does not encourage teaching innovations • Lack of awareness of HPC technologies already used in research or teaching for different fields • Faculty & students unaware of benefits and accomplishments of HPC • HPC technologies considered too complex/inaccessible for undergraduate instruction • Sequential HPC-related curricula is absent • Curricula using very large data sets not widely available • Adjust to different learning styles when material is complex • Variety of platforms/software leads to fragmented curricula • School administration/support staff not ready for HPC • Specs of computers and networks below user expectations

15. Building the Community of Faculty • These challenges are people-centric, not technology-centric and of interest to the EPSCOR community • Systemic Change requires understanding the system and working within it • Empower teachers (find the time), recognition (from chair/dean), support (student assistants)

16. Faculty Fellows Fall 01 Synergyamong themselves and with their chairs and deans People, Time, Support, Recognition …

17. NSF/EHRNational Science Foundation/Education and Human Resources Directoratewww.ehr.nsf.gov/EHR/RED/EVAL/handbook/handbook.htm LEADAssessment and Evaluation1998 Formative for the EC/CSEhttp://www.cae.wisc.edu/~lead/pages/produts/eot-paci.pdf

18. Faculty Workshops Introduce Computational Science Host Biology Workbench (BioQuest) Support at NPACI AHM02 Extend Faculty Fellows EC/CSE involve CSU

19. Contact Us Kris Stewart, Director http://www.edcenter.sdsu.edu stewart@sdsu.edu Jeff Sale jsale@edcenter.sdsu.edu Kirsten Barber barber@edcenter.sdsu.edu

20. The Ed Center Project and its Evaluation • Goal: incorporate computational science techniques in undergraduate curricula • Differences from STEP: target audience, institutional arrangements • Methods and Strategies: • Faculty Fellows • Workshops, presentations, in-house projects, trying out new approaches in own teaching, tools development • Computational Science Olympics (curriculum from the bottom-up!!) • Surveys, continuous assessment of faculty involvement and learning outcomes • Evaluated by the LEAD Center in 1998-99

21. Introducing the EC/CSE The mission of the Ed Center on Computational Science and Engineering? www.edcenter.sdsu.edu Who are the people involved? www.edcenter.sdsu.edu/staff Some of our projects: www.edcenter.sdsu.edu/projects/ Some of our activities: www.edcenter.sdsu.edu/news/ Some resources:www.edcenter.sdsu.edu/repository

22. Undergraduate Faculty: A Tough Target Group • Obstacles: lack of time, tenure and review considerations, lack of awareness about available technologies • Undergraduate faculty: • ¾ have used WWW often or sometimes (1997), but not in the classroom (only 18% - 1998) • The gap between those NEVER using computers in the classroom, and those using them OFTEN, is the largest for untenured faculty, increasing towards tenure review • Only 12% of surveyed faculty saw themselves as having a use for HPC applications in courses (higher for Sciences and Engineering) • 11% of faculty have students working with computer models OFTEN

23. Using computers in the classroom versus number of years as a faculty member (1997 Faculty Survey)

24. Students Using Computers in the Classroom (1997 Faculty Survey) 1. 1 OFTEN 2. 2 SOMETIMES 3. 3 RARELY 4. 4 NEVER 5. 9 DK/REF (Missing values) Primarily graduate faculty Primarily undergraduate faculty College of Arts & Letters College of Sciences

25. Strategies for Building Faculty Community • Reliance on most enthusiastic and technically advanced instructors who are already using computing and modeling in classes • The Faculty Fellows program: • Stakeholders: • College Deans - Specific support • Faculty - Compensation, and acknowledgement, of the value of the faculty members contribution • Benefits • College • Department (Faculty Fellows as discipline-specific spokespersons for EC/CSE) • Faculty (as individuals) • Ed Center on Computational Science and Engineering • Building a special infrastructure for curriculum transformation: human, institutional, technical – is a requirement for successful introduction of advanced techniques (since they are more demanding on faculty time and efforts) The problems and strategies are not that much different from STEP!!

26. Faculty Fellows during 1998-2000 • Faculty Fellows representing four departments from four colleges: Geological Sciences, Geography, Computer Engineering, Business Information Systems • Bi-weekly meetings at the Ed Center • Faculty Fellows as “ambassadors” of computational science • Partnership with LEAD for evaluation during 1998-99 • Follow-on Activities (Susan Millar, LEAD) • CATS (Classroom Assessment Techniques) • FLAG (Field-tested Learning Assessment Guide) • SALG (Student Assessment of Learning Gains)

27. More Strategies • Trying the new approaches in our own teaching: • Teaching the Supercomputer class with group-based problem-solving environments • Real-time distance teaching with Web-based collaborative software (featured as Microsoft Case Study in Higher Ed.) • Development of Sociology Workbench, a free on-line survey data analysis system that can be used for evaluation of student outcomes and other surveys

28. Channels for Influencing Pre-Service Teacher Preparation • Use of advanced computing modules in general ed courses (e.g. Geol 303 “Natural Hazards”) • Cooperation with College of Education faculty and students, esp. in Education Technology: focus on experimentation with new technologies in classroom setting • Computational Science Olympics: supporting the “bottom-up” development of computational science curricula • Providing on-line assessment technologies: • Sociology Workbench: http://edcenter.sdsu.edu

29. SWB Convenient Tool to Learn from Student Survey Data • Online tool for “standard public data sets” or your own data set http://edcenter.sdsu.edu • Small Sample, therefore only useful as feedback for the instructor • Can be used with “forms” interface directly into SWB format, as in June ‘99 CSU Faculty Workshop

30. SWB as Analysis ToolView Student Comments (text)

31. SWB as Analysis ToolIsolate on Specific Survey Response

32. SWB as Analysis ToolExplain the Response on Learning with “doing more”

33. SWB as Analysis ToolExplain learning with “active participation”

34. Lessons Learned • Institutional support required for program to be sustainable • Individual reform-ready faculty is focus for support • Infrastructure: • Build a Synergistic Environment (across disciplines) for Faculty • Continuous monitoring through interviews, surveys, discussions

35. What’s Next • The approaches we described proved useful for two target audiences. We believe that the strategies and lessons learned can be extrapolated in a targeted effort to incorporate computational science technologies in pre-service teacher preparation • This may be a funding opportunity? • We will be happy to contribute our knowledge and share experiences…

36. References ASCP Workshop • User-Friendly Handbook for Project Evaluation: Science, Mathematics, Engineering and Technology Education, NSF 93-152www.ehr.nsf.gov/EHR/RED/EVAL/handbook/handbook.htm • Sid Karin: The Importance of Science Literacy in a Computing World (see enVision Science Magazine, V.15 No. 2 • K. Stewart & I. Zaslavsky, “Ten Grand Challenges”, Supercomputing ’98 Orlando FL SC98 Conference Paper • J. Foertsch & B. Alexander, “Integrating HPC into the Undergraduate Curriculum”, report by LEAD Center June 1999 http://www.cae.wisc.edu/~lead/pages/products/eot-paci.pdf

37. Evaluation & Assessment • U. Wisconsin LEAD for 1998/99 Ed Center evaluation • www.cae.wisc.edu/~lead/pages/products/eot-paci.pdf • Follow-on Activities (Susan Millar, LEAD)CATS (Classroom Assessment Techniques)FLAG (Field-tested Learning Assessment Guide)SALG (Student Assessment Learning Guide)

38. Evaluation and Assessment of Classroom PracticeWhere to start? • User-Friendly Handbook for Project Evaluation: Science, Mathematics, Engineering and Technology Education, NSF 93-152www.ehr.nsf.gov/EHR/RED/EVAL/handbook/handbook.htm • Student Surveys - Need a compatible tool for instructor to examine results with • Sociology WorkBench (SWB) developed by team of undergraduate computer science majors employed by the EC/CSE

39. Information Overload • How can teachers help out? • How can technology providers help out? • Ans: personalize the linkage.

40. Information Overlab Lab • Using the WWW extensively in the classroom, implies obligation to discuss Information Overload • http://www.stewart.cs.sdsu.edu/infolab/index.html Timeline of Technologyhttp://www.stewart.cs.sdsu.edu/infolab/timeline_tech.html

41. Computer Network Security • Campus-wide “buy-in” recognizing the use of high-speed networksthe responsibility for “secure network use” • SDSU Academic Senate adopts interim policy • UCSD Acceptable Use Policy useful start

42. A Little History - May 10, 1869 • What was the major event? • Http://memory.loc.gov • Thank you to the US Library of Congress for the on-line exhibit preserving our “Memory”

44. Promentory Point, UTAH

45. How does this relate to us today?

46. Technologies to Motivate Learning • Mr. Sid uses Wavelets for compression of Railroad Maps • Sense of History must be tied to personal, student- based, “stories” • University Faculty need encouragement this will “count” for professional recognition by their peers and administrators (Deans…)

47. ONLINE Resources for Undergrads • The Education, Outreach and Training (EOT) of the PACIs http://www.eot.org • feature several projects • Ed Center on CSE • Shodor Undergraduate Computational Physics www.orst.edu • Minority Institutions, a Directory • Student Biology Workbench (Bioinformatics) • Underrepresentation in Computer Science (CDC)

48. Learning Technologies from the PACIs • Data Intensive Computing Environment • Digital Video/eTEACH • Interaction Environments: Bays to Brains • Neuroscience: Telescience/Tomography • NISE College Level One Institute on • Learning Technology • Remote Collaborative Learning • www.eot.org

49. This is Only the Beginning... YOU ARE HERE TIME

50. What’s Next • Your comments and thoughts? • SIAM Computational Science & Engineering Conference – Sept. 2000 – Washington D.C. • SC2000 – November 2000 – Dallas TX