Dr. Mary Spencer Dr. Chris Lobban Dr. María Schefter Dr. Greg Witteman University of Guam

1. Evolution of Institutional Capacity to Support the Assessment-Change-Effectiveness Cycle:An Undergraduate Science Program Case Study Dr. Mary Spencer Dr. Chris Lobban Dr. María Schefter Dr. Greg Witteman University of Guam

2. Introduction • Pre-assessment of attendees • Synthesis of 5 years of student outcomes assessment at UOG (Dr. Spencer) • Evaluation and the RISE Program • Background and framework (Dr. Schefter) • Example from the classroom (Dr. Lobban) • Assessment and information technology (Dr. Witteman) • Interactive sharing and discussion • Wrap-up • Post-assessment

3. UOG’s NIH RISE Program • RISE program is a broad and flexible grant for student, faculty, and institutional development. • Long-term goal is more minority PhDs in biomedical research. • Short-term goal is to increase motivation and capacity for biomedical research.

4. UOG’s NIH RISE Program • Evaluation is required. • Goals/objectives for NIH must be in terms of measurable outcomes for students, faculty, or the institution. • Present program includes student apprenticeships in research labs and a science technology classroom, plus faculty development opportunities.

5. How reluctant scientists are getting involved in evaluation • Evaluation standards • Engaging scientists • Goals and measurable objectives • Student input • Closing the loop

6. The “big picture” • In the present paradigm of biology, life is organized into “levels,” – or systems – with each level having “emergent” properties not seen in the parts. • Analogous to an institution of higher ed.? • The “big picture” of our assessment efforts. • Assessment of RISE is multi- level as well as multidisciplinary.

7. Levels of Organization Organism (individual bat) Body system (skeletal system) Organ (leg bone) Part of an illustration in Lobban & Schefter (1997)

8. Courses and workshops • o Factual conceptual knowledge of course content • o Lab skills appropriate to course • o Specific training in (e.g.) computer skills • o Specialized science reading / writing skills (e.g., lab reports) • o Links between course objectives and program / institutional / gen. ed. outcomes Educational outcomes: “levels” for assessment Community University Majors Courses & workshops

9. Majors (Discipline-specific training / education) • o Factual / conceptual knowledge of the field • o Proficiency in using scientific literature • o Ability to perform appropriate data collection /analysis • o Apprenticeship experiences in research labs Educational outcomes: “levels” for assessment Community University Majors Courses & workshops

10.  University education • o Reading/Writing/Analytical skills (GRE) • o Critical Thinking skills • o Computer literacy • o Presentation skills • o General education outcomes Educational outcomes: “levels” for assessment Community University Majors Courses & workshops

11. Career /Community level • o Career success (as PhD researcher or other) • o # Scientific findings • o # of PhD researchers • o Community service – science ed., biota, • (endangered) species survey work Educational outcomes: “levels” for assessment Community University Majors Courses & workshops

12. NIH RISE Program Assessment—change—effectiveness cycle(s)? Community Change University Majors Courses & workshops Assessment

13. Grass roots • Learning objectives • 3 parts (Mager) • Observable behavior (esp. verb… Bloom) • Of what…? • Criteria, e.g., scoring rubric

14. COGNITIVE PROCESS DIMENSION Remember Understand Apply Analyze Evaluate Create KNOWLEDGE DIMENSION Factual knowledge Conceptual knowledge Procedural knowledge Metacognitive knowledge

15. Source: Anderson, L.W. & D.R. Krathwohl. 2001.

16. Grass roots • Strengths and Weaknesses • Student self ratings • Faculty ranking of skills by courses

18. Pooled results of detail questions

19. Faculty assessment of skills for courses: A. Students need this skill as a prerequisite. B. Students need basic skill and I help them with it. C. I teach students this. D. Could be helpful in the course but not necessary. Cross out if skill is not useful in your course Also please: Put a star by the number if you think students will need this skill in most graduate biomedical/behavioral programs

20. Introduction • Pre-assessment of attendees • Synthesis of 5 years of student outcomes assessment at UOG (Dr. Spencer) • Evaluation and the RISE Program • Background and framework (Dr. Schefter) • Example from the classroom (Dr. Lobban) • Assessment and information technology (Dr. Witteman) • Interactive sharing and discussion • Wrap-up • Post-assessment

21. Typical syllabus

22. Revised syllabus

23. Source: Anderson, L.W. & D.R. Krathwohl. 2001.

24.  To gain an understanding of Pacific Island environments and the ecological principles on which they operate: the ecosystems (reefs, forests, savanna, wetlands); the biological, physical, and chemical processes and interactions that regulate these systems; and the ways in which humans affect and are affected by the natural environment. • Your understanding will be tested through your skills in: • interpreting – e.g., changing classification diagrams into text or vice versa; reading graphs; • exemplifying – e.g., giving an example of … • classifying – e.g., being able to classify the trophic level of an animal from a food web diagram • summarizing – e.g., Be able to summarize the process by which Darwin arrived at his hypothesis of atoll formation. • inferring – e.g., draw a logical conclusion from presented information • comparing – e.g., determine how similar things are as a criterion for applying analogy; • explaining – e.g., explain the cause of drought during El Nino

25. Understand the scientific process Darwin used and how his hypothesis of atoll formation was tested. • Be able to summarize the process by which Darwin arrived at his hypothesis. (Do NOT state or explain his hypothesis.) • Be able to explain why Darwin’s model of atoll formation was a scientific hypothesis (i.e., not a belief/statement of faith, nor idle speculation); • Using Darwin’s hypothesis, be able to infer the relative ages of two oceanic islands given maps of them. • Be able to recall what was done to test Darwin’s hypothesis. Became…

26. Evaluation Plan to Determine Program Outcomes (NIGMS-MORE) • Describe formative evaluations--these are evaluations carried out during the course of implementing activities to assess its suitability for the need. • Describe summative evaluations--these evaluations are carried out at the end of the activity to assess the outcome. • Discuss the use of qualitative and quantitative data collection methods. • State when in the course of implementing the activity data will be collected. • State any plans to make a mid-course modification of activities if formative evaluations indicate a need to change. • Provide examples of questionnaires to be used to collect qualitative improvements such as perceptions of participants. • State how data will be analyzed and provide the types of statistical methods to be used, if any, to test the reliability of the data. • Identify who will collect and analyze the data and provide credentials of the person(s) selected for collection and analysis of data. • Source: NIGMS-MORE Division

27. Source: Anderson, L.W. & D.R. Krathwohl. 2001.

28. Introduction • Pre-assessment of attendees • Synthesis of 5 years of student outcomes assessment at UOG (Dr. Spencer) • Evaluation and the RISE Program • Background and framework (Dr. Schefter) • Example from the classroom(Dr. Lobban) • Assessment and information technology (Dr. Witteman) [Click to continue slide show or to download next ppt file] • Interactive sharing and discussion • Wrap-up • Post-assessment