2009 GE and Assessment Conference February 27, 2009 Baltimore, Maryland

2. Assessment Develop learning outcomes Verify curriculum alignment Develop an assessment plan Collect assessment evidence Assess the evidence Close the loop

3. Let’s focus on each step. First, the learning outcomes.

4. AAC&U Suggestions: The Essential Learning Outcomes Knowledge of human cultures and the physical and natural world Critical and creative thinking Quantitative literacy Civic knowledge and engagement Ethical reasoning and action

5. Campus Examples

6. Knowledge of Human Cultures: Students can describe and analyze the capacity of race, ethnicity, class, gender, sexuality, disability, age, generation, and/or nationality to inspire, inform, and influence writers, artists and audiences (CSU Monterey Bay)

7. Quantitative Literacy Students can use tables, graphs, charts, and diagrams to explain concepts or ideas (Cabrillo CC)

8. Ethical Reasoning and Action Students can engage questions of ethics and recognize responsibilities to self, community, and society at large (U. of Delaware)

9. Alignment Is your curriculum systematically designed to engage all students in achieving your learning outcomes?

10. A Cohesive Curriculum Coherence Ongoing Practice of Learned Skills Systematically Created Opportunities to Develop Increasing Sophistication and Apply What Is Learned Synthesizing Experiences to Integrate Learning

11. We want to develop deep and lasting learning—not shallow, short-term learning.

12. GE Curriculum Map

13. The Assessment Plan: Meaningful Manageable Sustainable

14. Some Characteristics of Quality Assessment Valid Reliable Actionable Efficient and cost-effective Engage students Interest us

15. Common Direct Assessment Strategies Published Tests Locally-Developed Tests Embedded Assignments and Course Activities: Signature Assignments Portfolios Capstone Projects

16. Common Indirect Assessment Strategies Surveys Interviews Focus Groups

17. Assessing the Evidence Direct assessment often involves the application of rubrics. Raters should be normed/calibrated to achieve reliable results. Indirect assessment requires taking an honest look at what people are telling us.

18. Closing the Loop You can’t fatten a pig by weighing it.

19. A thermometer does not cure a fever.

20. Closing the loop requires collegiality and flexibility usually requires focusing on our curriculum (vs. my course) may require the support and collaboration of faculty, staff, administrators, and faculty and staff development professionals

21. Some Friendly Suggestions Focus on what is important. Don’t forget your adjunct faculty. Close the loop. Learn from your colleagues—both on-campus and external.

22. Without assistance, each of us will not only reinvent the wheel, we’ll also reinvent the flat tire.

23. =============================== We three are all addressing a different perspective on the issue of assessment techniques to advance student engagement and learning. I am a librarian – and as a profession we have been actively grappling with this issue for decades –some of you may be surprised to hear that – others here may be used to working with librarians to embed learning outcomes in courses and across the curriculum. Why are we so involved? The answer lies in the LEAP report itself – translated into this visual… =============================== We three are all addressing a different perspective on the issue of assessment techniques to advance student engagement and learning. I am a librarian – and as a profession we have been actively grappling with this issue for decades –some of you may be surprised to hear that – others here may be used to working with librarians to embed learning outcomes in courses and across the curriculum. Why are we so involved? The answer lies in the LEAP report itself – translated into this visual…

24. Our common goal is student learning - articulated best for me by the essential learning outcomes from Leap – the articulation of the knowledge, skills, and personal and social attributes we expect students to exhibit when they leave us. (e.g., inquiry and analysis; critical and creative thinking; written and oral communication; quantitative literacy; information literacy) But LEAP is also about how the foundation for student learning is engagement -- engaging students in their educational experience in three important ways. How they are inspired and engaged by faculty as instructors in and out of the classrooms, and through learning strategies and approaches developed by faculty to develop each student’s own passion and curiosity – teaching methods that are student centered – research based – rooted in real life. Teaching methods that require students to navigate within the world of information as critical explorers..regardless of the topic or the discipline... Ways to excite and motivate students through the content of the curriculum- engaging in the big questions – “both contemporary and enduring”. And then the importance of engaging students to take steps for their own development through a myriad of experiences available external to the curriculum – but intentionally linked to it as we design those learning experiences – together. This image is intentionally simplistic – but it reflects the inter-relationships of student learning and for me is a reminder of the overall more complex framework for learning that extends far beyond the content of the course.. Libraries success is measured in large part on their contribution to student learning – but how do we know what that is? Our success is inextricably linked to the content of the courses – and even more-so – how the learning is structured – - especially around assignment design. 7 years ago I wrote and managed two grant projects totalling $888,000 from the Mellon Foundation to restructure gateway and large impact courses at UC Berkeley – to ensure that research-based, inquiry learning was part of the experience. WHY the Library? Because we invest hundreds of millions of dollars in our libraries to collect or license materials – and if they are not used to advance student learning – then we may as well save the money…Our common goal is student learning - articulated best for me by the essential learning outcomes from Leap – the articulation of the knowledge, skills, and personal and social attributes we expect students to exhibit when they leave us. (e.g., inquiry and analysis; critical and creative thinking; written and oral communication; quantitative literacy; information literacy) But LEAP is also about how the foundation for student learning is engagement -- engaging students in their educational experience in three important ways. How they are inspired and engaged by faculty as instructors in and out of the classrooms, and through learning strategies and approaches developed by faculty to develop each student’s own passion and curiosity – teaching methods that are student centered – research based – rooted in real life. Teaching methods that require students to navigate within the world of information as critical explorers..regardless of the topic or the discipline... Ways to excite and motivate students through the content of the curriculum- engaging in the big questions – “both contemporary and enduring”. And then the importance of engaging students to take steps for their own development through a myriad of experiences available external to the curriculum – but intentionally linked to it as we design those learning experiences – together. This image is intentionally simplistic – but it reflects the inter-relationships of student learning and for me is a reminder of the overall more complex framework for learning that extends far beyond the content of the course.. Libraries success is measured in large part on their contribution to student learning – but how do we know what that is? Our success is inextricably linked to the content of the courses – and even more-so – how the learning is structured – - especially around assignment design. 7 years ago I wrote and managed two grant projects totalling $888,000 from the Mellon Foundation to restructure gateway and large impact courses at UC Berkeley – to ensure that research-based, inquiry learning was part of the experience. WHY the Library? Because we invest hundreds of millions of dollars in our libraries to collect or license materials – and if they are not used to advance student learning – then we may as well save the money…

25. Beginning in school, and continuing at successively higher levels across their college studies, students should prepare for twenty-first-century challenges by gaining: Knowledge Of Human Cultures and The Physical and Natural World Intellectual And Practical Skills Personal And Social Responsibility Integrative Learning through the application of knowledge, skills, and responsibilities to new settings and complex problems College Learning for the new global century identifies Essential Learning Outcomes – Knowledge.. Skills personal and social responsibility and it underscores the importance of creating an integrated learning experience.. I am going to focus on the intellectual and practical skills – expecting that the implications transfer to other essential learning outcomes such as those that are part of social and personal responsibility College Learning for the new global century identifies Essential Learning Outcomes – Knowledge.. Skills personal and social responsibility and it underscores the importance of creating an integrated learning experience.. I am going to focus on the intellectual and practical skills – expecting that the implications transfer to other essential learning outcomes such as those that are part of social and personal responsibility

26. These are the I and p skills in LEAP are… and it’s important not to lose track of the recommendation that… My own area of expertise for almost 20 years is that set of learning outcomes known as information literacy -- what they are – how to create learning environments where they can be developed – more dependent on how we teach than what we teach – and how we ultimately assess student learning of those outcomes. There is a great deal of overlap between info lit and these other I and p skills – oftentimes if you were to ask me to apply one of these labels on a specific outcome – I would be hard pressed to do so.. These are the I and p skills in LEAP are… and it’s important not to lose track of the recommendation that… My own area of expertise for almost 20 years is that set of learning outcomes known as information literacy -- what they are – how to create learning environments where they can be developed – more dependent on how we teach than what we teach – and how we ultimately assess student learning of those outcomes. There is a great deal of overlap between info lit and these other I and p skills – oftentimes if you were to ask me to apply one of these labels on a specific outcome – I would be hard pressed to do so..

27. Consider for a moment outcomes in these three areas… and since I don’t have clickers – I am going low tech and going to ask for show of hands – I am going to flash learning outcomes from sources in one of these three areas… and I want to know if you think it is A – info lit B – critical thinking Or c – communication Consider for a moment outcomes in these three areas… and since I don’t have clickers – I am going low tech and going to ask for show of hands – I am going to flash learning outcomes from sources in one of these three areas… and I want to know if you think it is A – info lit B – critical thinking Or c – communication

28. Is this a or b or c Hands????Is this a or b or c Hands????

30. Is this a or b or c Hands???? Is this a or b or c Hands????

33. These are a few of the 122 learning outcomes that are part of info lit competency standards for higher education… And as you can see – there is substantial inclusion of critical thinking, problem solving, and communication in information literacy.These are a few of the 122 learning outcomes that are part of info lit competency standards for higher education… And as you can see – there is substantial inclusion of critical thinking, problem solving, and communication in information literacy.

34. It doesn’t matter what we call the learning outcomes – or how we cluster them.. Whether they are the I and p skills from LEAP It doesn’t matter what we call the learning outcomes – or how we cluster them.. Whether they are the I and p skills from LEAP

35. Or what are referred to as integrated skills at some institutions – like Washington StateOr what are referred to as integrated skills at some institutions – like Washington State

36. Or a phrase I just heard recently from some Australian colleagues – they have an office for the development of these skills – in and out of curriculum – called academic skills… Or a phrase I just heard recently from some Australian colleagues – they have an office for the development of these skills – in and out of curriculum – called academic skills…

37. Or what is sometimes called this – particularly by the literacy communities… NFIL - with over 200 member organizations - just held an international colloquium in Alexandria - cosponsored by UNESCO and IFLA -- Alexandria Proclomation.. Information Literacy lies at the core of lifelong learning. It empowers people in all walks of life to seek, evaluate, use and create information effectively to achieve their personal, social, occupational and educational goals. It is a basic human right in a digital world and promotes social inclusion of all nations. OECD The knowledge-based economy is characterised by the need for continuous learning of both codified information and the competencies to use this information. As access to information becomes easier and less expensive, the skills and competencies relating to the selection and efficient use of information become more crucial... Capabilities for selecting relevant and discarding irrelevant information, recognising patterns in information, interpreting and decoding information as well as learning new and forgetting old skills are in increasing demand. Or what is sometimes called this – particularly by the literacy communities… NFIL - with over 200 member organizations - just held an international colloquium in Alexandria - cosponsored by UNESCO and IFLA -- Alexandria Proclomation.. Information Literacy lies at the core of lifelong learning. It empowers people in all walks of life to seek, evaluate, use and create information effectively to achieve their personal, social, occupational and educational goals. It is a basic human right in a digital world and promotes social inclusion of all nations. OECD The knowledge-based economy is characterised by the need for continuous learning of both codified information and the competencies to use this information. As access to information becomes easier and less expensive, the skills and competencies relating to the selection and efficient use of information become more crucial... Capabilities for selecting relevant and discarding irrelevant information, recognising patterns in information, interpreting and decoding information as well as learning new and forgetting old skills are in increasing demand.

38. These skills and abilities have some common elementsThese skills and abilities have some common elements

39. It’s irrelevent what you call them – and I would argue that its equally irrelevent how you label them…The first and most important step is to define specific learning outcomes for your students – and the devil is in the details – We all want students who are critical thinkers – but what precisely do we mean by that – what do the outcomes specifically look like – in measurable ways… and how do we explicitly communicate our expectations to our students so that they know what is expected Here are some examples Oftentimes the process of articulating the learning outcomes for the institution is the most important part of the assessment process – because the conversations should include he various stakeholders – faculty – student life – librarians – students – and the articulation about the specific learning e.g. information literacy standards provide a framework – yes there are five standards and 122 learning outcomes – but it is very important for each institution to have a process of articulating those learning outcomes for your own students… It’s irrelevent what you call them – and I would argue that its equally irrelevent how you label them…The first and most important step is to define specific learning outcomes for your students – and the devil is in the details – We all want students who are critical thinkers – but what precisely do we mean by that – what do the outcomes specifically look like – in measurable ways… and how do we explicitly communicate our expectations to our students so that they know what is expected Here are some examples Oftentimes the process of articulating the learning outcomes for the institution is the most important part of the assessment process – because the conversations should include he various stakeholders – faculty – student life – librarians – students – and the articulation about the specific learning e.g. information literacy standards provide a framework – yes there are five standards and 122 learning outcomes – but it is very important for each institution to have a process of articulating those learning outcomes for your own students…

40. A learning outcomes approach means basing program and curriculum design, content, delivery, and assessment on an explicit identification of the integrated knowledge, skills, and values needed by both students and society. It focuses on what students should be able to do and not what knowledge they possess. A learning outcomes approach means basing program and curriculum design, content, delivery, and assessment on an explicit identification of the integrated knowledge, skills, and values needed by both students and society. It focuses on what students should be able to do and not what knowledge they possess.

41. But here is where we have to pay greater attention… in a learning outcomes approach - But here is where we have to pay greater attention… in a learning outcomes approach -

42. As a matter of fact – the assessments are so firmly embedded in the curriculum that they are often indistinguishable from instruction -they are designed to provide the student with a genuine rather than a contrived learning experience Recommendation 13 of College Learning Students should know from the time they enter college that they will be expected to complete milestone and culminating projects— “authentic performances”—to demonstrate both their progress in relation to the essential outcomes and their ability to use the learning outcomes in the context of their chosen fields. Here are some authentic assessments at a course level Checklist - steps followed… Categorizing Grid - print tools to identify articles in disciplines electronic databases to find articles… Journals - reflection on process portfolios - cumulative process - samples of work over time -can include work - self assessment - progress report conferences - ask specific questions to focus on process and provide feedback Rubrics - scaled set of criteria that clearly defines what acceptable and unacceptable performance looks like As a matter of fact – the assessments are so firmly embedded in the curriculum that they are often indistinguishable from instruction -they are designed to provide the student with a genuine rather than a contrived learning experience Recommendation 13 of College Learning Students should know from the time they enter college that they will be expected to complete milestone and culminating projects— “authentic performances”—to demonstrate both their progress in relation to the essential outcomes and their ability to use the learning outcomes in the context of their chosen fields. Here are some authentic assessments at a course level Checklist - steps followed… Categorizing Grid - print tools to identify articles in disciplines electronic databases to find articles… Journals - reflection on process portfolios - cumulative process - samples of work over time -can include work - self assessment - progress report conferences - ask specific questions to focus on process and provide feedback Rubrics - scaled set of criteria that clearly defines what acceptable and unacceptable performance looks like

43. Let’s go back to a couple learning outcomes for information literacy and think about responsibility for the learning and for the authentic assessment The outcomes included in info lit competency standards for higher ed range from simple to complex on Bloom’s taxonomy of cognitive processes..Let’s go back to a couple learning outcomes for information literacy and think about responsibility for the learning and for the authentic assessment The outcomes included in info lit competency standards for higher ed range from simple to complex on Bloom’s taxonomy of cognitive processes..

48. And since I reference Iskills as a standardized assessment tool – I thought I would share just a bit more information about it. According to my analysis – there is overlap in some important areas with the CLA– but some even more important areas of uniqueness. According to the common scoring rubric for CLA as compared to the detailed outcomes for ISkills , CLA does not cover outcomes defined in Iskills as defining and accessing information – specifically articulate a need for information that defines a hypothesis or problem in operational terms, develop and apply a systematic strategy for ethically and legally finding, retrieving, and sorting information from a variety of relevant resources, representing a wide range of perspectives, acknowledging sources appropriately And Iskills – although it includes several areas of communication – does not address the craft of writing – punctuation, syntax, sentence structure etc.. Further analysis would be important for any institution that decides to use these tools. And since I reference Iskills as a standardized assessment tool – I thought I would share just a bit more information about it. According to my analysis – there is overlap in some important areas with the CLA– but some even more important areas of uniqueness. According to the common scoring rubric for CLA as compared to the detailed outcomes for ISkills , CLA does not cover outcomes defined in Iskills as defining and accessing information – specifically articulate a need for information that defines a hypothesis or problem in operational terms, develop and apply a systematic strategy for ethically and legally finding, retrieving, and sorting information from a variety of relevant resources, representing a wide range of perspectives, acknowledging sources appropriately And Iskills – although it includes several areas of communication – does not address the craft of writing – punctuation, syntax, sentence structure etc.. Further analysis would be important for any institution that decides to use these tools.

50. Why do I – a librarian – care – or even know about assessment? What can I add to a conversation where others may have spent a career focusing on it – some of you in this audience. The answer lies in the LEAP report itself – translated into this visual… Why do I – a librarian – care – or even know about assessment? What can I add to a conversation where others may have spent a career focusing on it – some of you in this audience. The answer lies in the LEAP report itself – translated into this visual…

51. Goals: Personal and Social Responsibility, Intercultural competence Inventory Existing programs that inform these goals Existing data which give insight into these outcomes Decide what new information you need Multiple Outcomes=Multiple Measures Value-added (longitudinal) Milestones (periodic assessment)

52. Pluralistic Orientation: End of First Year (N= 29,796 students; Alpha reliability =.85) Tolerance of others with different beliefs .82 Ability to work cooperatively with diverse people .81 Openness to having my own views challenged .80 Ability to discuss/negotiate controversial issues .77 Ability to see world from someone else’s perspective .76 Now on entering student surveys, end of first year, college senior surveys, ten year followup (CIRP)

53. College Experience Effects on Pluralistic Orientation: End of First Year, CIRP

54. Social Agency Values (N= 27,288 students; Alpha reliability .76) Influencing social values .84 Helping to promote racial understanding .76 Influencing the political structure .75 Helping others who are in difficulty .68

55. College Experiences Predicting Social Agency Among College Seniors

56. Using Survey Data and Standardized Tests How are some survey measures related to cognitive outcomes or moral reasoning? Which do we use? Study in classrooms: Introductory Courses or Psychology Pool Administer standard tests to random sample, split samples, use sophisticated software to analyze Study in Intro to Psychology, random assignment of 289 students to test relationship between cognitive tests and outcomes on our survey Moral reasoning was administered in three classrooms to compare introductory classes that had similar goals but were different in content. URM composition was 15%.

57. CIRP Items and Cognitive Measures

58. University Classroom StudyMoral Development

59. Predisposition, Diversity Content, and Pedagogy Effects On Moral Reasoning

60. What We Learned There are multiple ways to measure the common goal of preparing college students to participate in a diverse democracy including: Multiple measures of cognitive outcomes Changes in interests in social issues (e.g. causes of poverty) Moral reasoning tests Leadership behaviors (in college and post college) Cultural awareness Colleges can learn more by linking local data with these common measures

61. Multi-Campus Surveys:Assessing Campus Climate for Diversity

62. Resources Higher Education Research Institute, UCLA www.gseis.ucla.edu/heri News New national research Registration for survey instruments Publications, research reports Tool kits for assessment Interactive forum online: Ask questions and get advice from others using data on college students