Core Assessment: Attributes of a Successful Program at James Madison University

1. Core Assessment: Attributes of a Successful Program at James Madison University Dr. Dena Pastor Center for Assessment and Research Studies TAIR Professional Development Workshop September 2004

2. James Madison University (JMU) • Location: Harrisonburg, VA • Students: • ~15,000 undergraduate • ~700 graduate • State-funded university operated by its own board of visitors

3. JMU Infrastructure for Assessment • Center for Assessment and Research Studies (CARS) • Supports JMU assessment endeavors • General Education • Programs/Majors • Student Affairs • Alumni • 9full-time faculty members & 3 administrative staff • Faculty split their time evenly between assessment duties & teaching/research • Expertise in IRT, SEM, Validity, Computer-Based Testing, HLM, Examinee Motivation, Performance Assessment, Student Development • Offers both M.A. and Ph.D. programs in Assessment & Measurement • 10-15 students

4. JMU Infrastructure for General Education • Dean of General Education • Cluster Coordinators • Faculty given release time for this task • Steering Committee • Broad faculty representation • Positions allocated to departments to teach General Education and major courses • Each cluster undergoes a review process every 5 years

5. General Education General Education at JMU • 41 credit hours of courses organized into 5 clusters Cluster 1: 9 credits Skills for the 21st Century Critical Thinking, Written & Oral Communication, Information Literacy Cluster 2: 9 credits Arts & Humanities Culture, Philosophy, Fine Arts & Literature. Cluster 3: 10 credits The Natural World Quantitative & Scientific Reasoning Cluster 4: 7 credits Social & Cultural Processes Global & American: History, Govt., Economics, Anthropology Cluster 5: 6 credits Individuals in the Human Community Wellness, Psychology & Sociology

6. Wellness: Choose one Sociocultural: Choose one General Education Course Offerings Example from Cluster 5’s Wellness & Sociocultural Domains

7. Use Information Assessment Steps Maintain Information Analyze Information Collect Information Select/Design Instruments Establish Objectives

8. Cluster Learning Objectives • Created by cluster faculty with assistance from CARS • Start by asking: • “What should a student know and be able to do as a result of completing their coursework in this cluster?” • What the student knows & is able to do should NOT be tied to specific courses taken within the cluster.

9. Example of Cluster Learning Objectives Example from Cluster 5’s Wellness & Sociocultural Domains After completing Cluster Five students should be able to: • Understand the dimensions of wellness, the various factors affecting each dimension, and how dimensions are interrelated. • Recognize an individual's level of health and wellness and understand how these levels impact quality of life. • Discriminate between ethical and non-ethical practices in the social/behavioral sciences.

10. Match Learning Goals With Coursework • Have instructors rate the extent to which the material necessary to achieve each of the goals is emphasized in their course • This will help ensure that students are able to achieve goals regardless of which combination of courses they take A student taking GHTH100 & GPSYC101 should have the same chance of achieving goals as student who takes GKIN100 & GSOCI240

11. Assessment of Learning Objectives • Faculty within each cluster decide how to assess the extent to which students are mastering the cluster’s learning objectives • Most clusters create a multiple-choice test from scratch, with continual improvements made on the instrument

12. Assessment Tools Almost all assessment tools are multiple-choice tests created by JMU faculty.

13. Matching Assessment Tools to Learning Objectives Example from Cluster 5’s Sociocultural Domain Assessment

14. Administration of Assessment Tools • Two institution-wide Assessment Days • Fall (August): Incoming freshmen tested at orientation • Spring (February): Students with 45-70 credits (sophomore/junior) tested • All undergraduate classes are cancelled on this day • All students are required to participate • course registration blocked for those who do not participate • Students are assigned to testing rooms by last 4-digits of their id #s where a particular series of instruments are administered

15. Assessment Day Data Collection Scheme:Repeated Measures COHORT 1 Students in each cohort are tested twice on the same instrument – once as incoming freshmen and again in the second semester of the sophomore year. COHORT 2 COHORT 3

16. Compare pre-test scores to post-test scores (same students) of Cluster “Completers” . Compare Incoming Freshmen with Sophomore Students (different students). Examine the Effect of Number of Courses. Correlate Scores and Course Grades. With this data, we can… Students completing their cluster courses should score higher on the post-test as sophomores than they did on the pre-test as incoming freshmen. Sophomores should outperform incoming freshmen. Students who have completed a larger # of courses within a cluster should outperform students who have completed a smaller # of courses. There should be a positive correlation between test scores and the grades received by students in the cluster courses.

17. 1. Compare pre-test scores to post-test scores (same students) of Cluster “Completers” . Example from Cluster 4’s American Experience Test

18. 2. Compare Incoming Freshmen with Sophomore Students (different students). Example from Cluster 3’s Natural World Assessment

19. 3. Examine the Effect of # of Courses. Example from Cluster 3’s Natural World Assessment

20. 4. Correlate Scores and Course Grades. Example from Cluster 4’s Global Experience Test

21. Assessment Information Not Collected on Assessment Day These are both Computer-Based Tests administered throughout the school-year at a computer lab on campus. All students must exceed the passing score set by the faculty within their first year. A random sample of ~400 students taking the Gen. Ed. writing course are asked to submit a portfolio of their best work which is rated according to a 4 point rubric by the faculty. The GCOM test is a Computer-Based Test embedded within a course: This test serves as the final exam in all sections of the Gen. Ed. communication course.

22. Other Assessment Information Previously Collected on Assessment Day

23. How does General Education Assessment Benefit JMU Faculty? • Collaboration among faculty in different disciplines • Opportunity to communicate with one another • Feel ownership over assessment tools, more invested in the results • Data may help support resources they would like to acquire • Gain skills that they can take back into the classroom • Opportunities for scholarship

24. How does General Education Assessment Benefit JMU Students? • Receive instruction that is more tailored to their needs • Receive instruction that is more consistent across sections • Held accountable for knowledge/skills (ISST, Tech Level tests, GCOM final exam)

25. How does General Education Assessment Benefit the University? • Enhanced public relations and visibility • Can communicate information to parents, possible applicants, stakeholders and say “this is what a students will know and be able to do as a result of completing the Gen Ed courses at JMU” • Enhanced grant funding opportunities • Marketable assessment tools can be a source of income

26. JMU’s Marketable Assessment Tools • Computer-Based Multiple-Choice Tests • ILT: Information Literacy Test (60 items) • QR: Quantitative Reasoning Test (45 items) • SR: Scientific Reasoning Test (80 items) • Marketable Tests Being Developed • Oral Communication • Critical Thinking

27. What are Other Institutions Doing? • Approaches to Gen Ed assessment (Palomba & Banta, 1999) : • Individual Course-Based Approach • Multi-Course Approach • Noncourse-Based Approach

28. Individual Course-Based Approach • Instructor submits a document showing how goals are being addressed in their particular course • May include syllabus, quizzes, tests, assignments, activities, examples of students’ work • May be judged by other faculty for the extent to which course is covering goals

29. Individual Course-Based Approach • College of William and Mary is taking this approach • Also using faculty and student surveys • Different Gen Ed “clusters” assessed in different years • Office of Assessment oversees process http://www.wm.edu/wmoa/index.htm

30. Individual Course-Based Approach • Advantages • Maximizes faculty involvement • Assessment results may help identify changes that need to be made in a particular course • May ensure that goals are being addressed in the classroom • Disadvantages • Can be labor intensive for faculty • Doesn’t necessarily involve faculty collaboration • Faculty may only choose information that makes them “look good” • Not necessarily measuring whether or not student has mastered the goal – just measuring whether or not student was exposed to material that would allow them to master the goal Some information from Palomba & Banta, 1999

31. Multi-Course Approach • Faculty from different disciplines join together to decide upon assessment for their “cluster” • Winthrop University takes this approach • Collect assessment data via course-embedded assessments: common essays, tests, assignments across courses that are graded according to a common rubric • Also administers the National Survey of Student Engagement (NSSE) and the College BASE to freshmen and seniors (repeated measures) • Office of Assessment oversees process Some information from Palomba & Banta, 1999 http://www.winthrop.edu/assessment/gened.htm

32. Multi-Course Approach • Advantages • Encourages faculty collaboration • Focuses on student experience in a “cluster”, not in a particular course • Shows evidence of student learning • Disadvantages • When course-embedded, dictates what assignments/tests, etc. must be administered in a particular course, which may be a turn-off to some instructors

33. Noncourse-Based Approach • Campus-wide assessment activities focused on individual student or groups of students, not a particular course • Assessments administered outside classroom to all students or a sample or students • Assessment tools may be locally-developed (like JMU) or standardized; may include surveys

34. Noncourse-Based Approach • Advantages • Assessment removed from classroom, instructors are given more freedom • Faculty collaboration encouraged • Focuses on students’ experience in a “cluster” • Disadvantages • Logistics of collecting data • Student motivation to take assessment seriously • Mechanisms need to be in place:1) to ensure that students are exposed to materials within courses that will enable them to meet objectives; 2) to match instrument to objectives

35. Assessment Information on Internet • James Madison University • http://www.jmu.edu/assessment/ • University of North Texas • http://www.unt.edu/vpaa/Assessment/Assessment_Home.html • American Association of Higher Education • http://www.aahe.org/ • National Center for Postsecondary Improvement • http://www.stanford.edu/group/ncpi/index.html • NC State: Internet Resources for Higher Education Outcomes Assessment • http://www2.acs.ncsu.edu/UPA/assmt/resource.htm