Assessment in the Biology Department in AY 2007-8

1. Assessment in the Biology Department in AY 2007-8 Caroline Solomon Friday December 5

2. Biology Department Outcomes • The development of the biology department outcomes took several years (thanks to Dr. Ann Powell who led us in this effort) before finally being polished by Bette Landish

3. Biology Department Outcomes 1. Demonstrate knowledge of current theories of biology and concepts basic to biology, including: Levels of complexity (molecular/cellular through population/communities/ecosystems); biological principles and processes; connections among concepts across disciplines (physical sciences, mathematics, social sciences). 2. Demonstrate critical thinking skills through: Identifying appropriate questions, issues or problems to be explored; analyzing, interpreting and evaluating evidence; constructing well-supported, clearly articulated arguments and conclusions.

4. Biology department outcomes (con’t) 3. Demonstrate information literacy and research skills through effective use of: Sources of information in biology including published literature and scientific databases; computer applications to acquire and (continued) analyze experimental data and develop graphic models. 4. Demonstrate experimental research methods including: Scientific methods and instrumentation; safe and appropriate use of laboratory equipment; experimental design; data analysis; familiarity with professional standards in science.

5. Biology department outcomes (con’t) 5. Demonstrate effective communication with varied audiences and media by: Providing clear structure and transitions; demonstrating substantial understanding via research, credible sources and supporting evidence; demonstrating audience-appropriate purpose, agenda, language and style. 6. Demonstrate preparation for future career and educational goals by: awareness of personal competencies (strengths and weaknesses); ability to work in a team to accomplish a goal; understanding professional and ethical behavior.

6. Four year plan for assessment

7. Outcome #1: Biology content Demonstrate knowledge of current theories of biology and concepts basic to biology, including: Levels of complexity (molecular/cellular through population/communities/ecosystems); biological principles and processes; connections among concepts across disciplines (physical sciences, mathematics, social sciences). (1) Direct measurement: ETS Biology Field Test (2) Indirect measurements: (a) Revised student evaluations to reflect outcomes (b) Exit surveys from seniors (2007 & 2008)

8. Results from ETS Biology Field Test *areas that need to be worked on the most

9. Student distribution *Subscores are reported; (% compared to institutional mean scores that includes seniors from 345 domestic institutions)

10. Our next step… closing the loop for Outcome #1 Weaknesses: biochemistry, cell biology & botany/plant physiology How do we change our curriculum or courses to address these weaknesses? Department decided to: (1) emphasize botany and plant physiology more in second semester of introductory biology (BIO 108) and in ecology (BIO 241) (2) strongly encourage students to take cell biology or molecular biology; offer the course as early as Spring 2009

11. Outcome #5: Effective Communication Demonstrate effective communication with varied audiences and media by: Providing clear structure and transitions; demonstrating substantial understanding via research, credible sources and supporting evidence; demonstrating audience-appropriate purpose, agenda, language and style. (1) Direct measurement: Rubrics for lab reports, literature reviews & presentations (2) Indirect measurement: Exit surveys from seniors (2007 & 2008)

12. Assessment tools for this outcome • Writing a. Lab report rubric b. Literature review rubric • Presentations a. Class presentation rubric

13. Lab report rubrics • Two different lab rubrics were used during AY 2007-2008 • Why did we revise the lab report rubric for Spring 2008? • A lab report rubric workshop was held in December 2007; the scoring for three different student products were not consistent among department members

14. Lab report rubric used in Fall 2007

15. Revised lab report rubric

16. Lab Reports – Fall 2007 We identified these problem areas in December 2007; Intro & discussion involve critical thinking BIO 103; n= 48 BIO 108; n= 32 BIO 231/241; n = 17

17. Lab Reports – Spring 2008 BIO 104/GSR 150; n= 85 BIO 110; n= 33 BIO 211,221, Capstone; n =30 Note difference in scales from Fall 2007 (5 points) to Spring 2008 (4 points)

18. A closer look…..Title and Abstract Need to figure out why With the new lab report rubric – we can now see things in more detail

19. A closer look….Introduction Decline because SQ is given in earlier courses?

20. A closer look….Materials & Methods Students need more work in developing their OWN experiments Senior Exit survey: Did the program teach you how develop your own experiments? 2007 – 3.7; 2008 – 2.5 Did the program prepare you to use various laboratory instruments? 2007 – 4.2; 2008 – 3.6

21. A closer look….Results Students need more practice interpreting & creating their own graphs Senior Exit survey: Did the program teach you how to analyze, interpret and evaluate evidence? 2007 – 4.7; 2008 - 3.1 Did the program teach you how to develop good graphs and charts? 2007 – 4.2; 2008 – 2.4

22. A closer look….Discussion & Conclusion Still needs improve- ment Senior Exit survey: Did the program teach you how to support your arguments and conclusions? 2007 – 4.4; 2008 – 2.8

23. A closer look….Quality of Writing

24. A closer look….Citation & Bibliography Improvements were made in this category from fall to spring

25. Synthesis of results of writing assignments • Decrease in ability from freshman year to upper level courses: • To write a good abstract • Identification of scientific question • Areas that need improvement • Development of own experimental procedures (identification of controls) • Data analysis; graphing • Discussion & Conclusion (critical thinking)

26. Our next step… closing the loop for Outcome #5 • The discussion of the results from the lab report rubric produced a healthy discussion about comfort levels of different faculty members about Excel (data analysis & graphing) and defining levels of expectation of mastery in Excel were needed

27. Presentations • Assessment of presentations during Fall 2007 were compounded by the use of different rubrics by different professors, no recording of presentations • In January 2008, the department developed a presentation rubric that was used in Spring 2008

28. Presentations – Fall 2007 Note: For most categories n= 20, but some had n= 26 due to different rubrics; for better understanding of categories see journal club rubric (BIO 231, 241, 401)

29. Presentations – Spring 2008 n=8 n=3

30. Improvements that needed to be made in collection of data for presentations • Identification of weaknesses: • Content • Use of visual media • Bilingual communication (sign clarity) • Often people did not do all the categories • No recording of presentations; collection of evidence • A new protocol to record presentations for Fall 2008

31. Focus for AY 2008-2009

32. Four year plan for assessment

33. Outcome # 2: Critical Thinking Demonstrate critical thinking skills through: Identifying appropriate questions, issues or problems to be explored; analyzing, interpreting and evaluating evidence; constructing well-supported, clearly articulated arguments and conclusions. Suggestions for measurements: (1) Critical thinking components of writing & presentation rubrics (2) Inclusion of data analysis problems and conceptual problems in exams in major courses (following Kitchen et al. 2003) (3) Scoring on analytical problems on the ETS Field Test

34. Focus for Spring 2009 • Invite Virginia Anderson from Towson to present on how to assess critical thinking in biology (coming January 15th) • Develop assessment tools for critical thinking (outside of writing rubrics)

35. Outcome #6: Career Preparation (Ava) Demonstrate preparation for future career and educational goals by: awareness of personal competencies (strengths and weaknesses); ability to work in a team to accomplish a goal; understanding professional and ethical behavior. (1) Direct measurement: Alumni surveys (how many are employed in the field after graduation?) (2) Indirect measurements: (a) Internship portfolio (b) Exit surveys from seniors (2007, 2008 & 2009)

36. Reflections on assessment • At the first department meeting of the AY, we had three groups of data to help us guide us in making decisions about the future direction of the department: (1) Assessment data from the 2 outcomes (2) Preliminary data from internship portfolios (3) An analysis of Biology curriculums around the country at schools of a comparable size to Gallaudet

37. Even if we are starting to develop tools for the two student outcomes we want to measure in AY 2008-9, we are continuing to collect data on the two learning outcomes we started last year - it’s an ongoing process!