460 likes | 546 Views
The Institutional Artifact Portfolio Process: An Effective and Nonintrusive Method for General Education Assessment. Renée M. Tobin, Derek J. Herrmann, & Kelly L. Whalen Illinois State University. Illinois State University: Who We Are. DRU Fall 2010 census day figures
E N D
The Institutional Artifact Portfolio Process: An Effective and Nonintrusive Method for General Education Assessment Renée M. Tobin, Derek J. Herrmann, & Kelly L. Whalen Illinois State University
Illinois State University: Who We Are • DRU Fall 2010 census day figures • Total enrollment: 20,762 • Undergraduate enrollment: 18,254 • 64.4% of all students are female • 13.1% of all students are minority
Illinois State University: Who We Are • 96.8% of undergrads are from Illinois • 93.1% of undergrads are < 24 years old • 94.6% of undergrads are full-time students
Illinois State University: Who We Are Fact Book (Planning and Institutional Research) – Fall 2010 • First-year students ACT mean composite scores: • Illinois State 24.1 • State of Illinois 20.7 • National 21.0
Illinois State University: Who We Are • Transfer Students • 1,838 new transfer students in Fall 2010 • ~50% of bachelor’s degrees granted annually • 85.0% retention from first-year to sophomore
University Assessment ServicesIllinois State University • UAS Office of the Provost • IR Finance and Planning The two offices work closely; however, UAS is responsible for the management of all program-level assessment.
History of ISU’s Gen Ed Program • Senate approval in 1992 • Full implementation in 1998 • Assessment task force in 2005 • Full implementation in 2008 • New task force in 2011-2012 • Assessment subcommittee!
Current Gen Ed Program • 12 Goals with 40 distinct skills/abilities • 42 credit hours • 190 potential courses • Approximately 13,410 enrollments per semester • 3 Cores: • Inner – (5 courses) • Middle – (5 courses) • Outer – (4 courses)
Current Gen Ed Assessment • Comprehensive, yet manageable • Purpose: To provide the Council for General Education with sound evidence to base decisions regarding the program.
Important ContributorsAKA Campus Buy-In • Director of General Education • Council for General Education • General Education Assessment Task Force • Assessment Advisory Council
Research • Reviewed past methodologies employed at ISU • Reviewed methodologies employed at other institutions
Initial Challenge • Goals of General Education • Numerous (12 with 40 abilities) • Difficult for faculty/staff/students to remember and reference • In some cases, challenging to measure
Solution • Four Shared Learning Outcomes • Common and integrated elements of the established goals of Gen Ed • Also aimed at eliminating some of the division that is present between Gen Ed and the Major
Selecting an Assessment Method:The Institutional Portfolio (Seybert) • Principles • Gen Ed is responsibility of entire campus • Be minimally intrusive (both faculty and students) • Use existing student work
Selecting an Assessment Method:The Institutional Portfolio (Seybert) • Process • “Artifacts” are collected • Faculty teams review using rubrics • Results are compiled and reported to committee • Committee makes decisions based on results
Selecting an Assessment Method:The Institutional Portfolio (Seybert) • Characteristics • Faculty review teams are multidisciplinary • Review is invisible to students and not intrusive to faculty • Process is labor intensive and requires resources • Process is dynamic and “messy”
Method – Requirements • Non-intrusive • Cover the 12 goals of general education • Comprehensive, Manageable • Institution-focused
General Overview for Assessment • Phase One: Obtaining the artifacts • Phase Two: Sampling the artifacts • Phase Three: Reviewing the artifacts
Phase One: Obtaining the Artifacts Instructor Participation: • Read the invitation letter from Provost • Identify artifact(s) that address at least 1 Primary Trait • Complete the online participation form • Allow UAS access to the artifacts for up to 24 hours
Phase Two: Sampling the Artifacts Goal – Obtain a random sample of 100 artifacts per core (300 total artifacts per Shared Learning Outcome) Behind the Scenes at UAS: • Calculate the total number of enrollments, based on class size • Remove all identifying information and oversample • End of the term, recalculate and use the appropriate proportions
Phase Three: Reviewing the Artifacts • Instructors solicited to apply to be IAP reviewers • Interdisciplinary two-person teams (3 per Shared Learning Outcome and an alternate) are trained in use of established rubrics • Review teams complete consensus analysis using developed rubrics for the 4 Shared Learning Outcomes
Review Process Review Week: • Day One: • Reviewers interpret the rubrics as a group • Calibration training using practice artifacts • Days Two through Four: • Reviewers split into teams and complete their binders • Final rubrics collected throughout
Phase Three: Reviewing the Artifacts • Instructors solicited to apply to be IAP reviewers • Interdisciplinary two-person teams (3 per Shared Learning Outcome and an alternate) are trained in use of established rubrics • Review teams complete consensus analysis using developed rubrics for the 4 Shared Learning Outcomes • The final data are reported to Council of General Education to formulate commendations/recommendations
Results • Reliability • Inter-rater reliability (calibration artifacts) • Percent agreement • Public Opportunity: 42% - 71% • Critical Inquiry and Problem Solving: 61% - 69% • Intraclass correlation coefficient • Diverse and Global Perspectives: .70 - .91 • Life-Long Learning: .26 - .63 • Test-retest reliability (repeat calibration artifacts) • Exploratory factor analysis
Results • Focus on trends for each Shared Learning Outcome • No consistent differences across freshmen, sophomores, juniors, and seniors • No clear differences emerged when comparing native and transfer students
Results • Maximum review value consistent with instructions
Discussion • Limitations of the results • Work in progress • Expand from 3 Cores to many course categories • Data based on courses from one semester only
Discussion • Future directions • More time calibrating during review process • Reports with data from two different semesters • General Education Task Force (and its Assessment subcommittee)
Conclusions • Advantages of the IAP • Nonintrusive to instructional faculty and staff and students • Allows for program-level assessment • Maintains the institution as the focus
Conclusions • Challenges of the IAP • Faculty participation • Locus of generalizability • Takes time
Conclusions • Lessons learned • Patience and public relations • Streamline the process • Oversample • Reviewer calibration is key • Alternate reviewer • Closing the loop
For More Information • http://gened.illinoisstate.edu/ • http://assessment.illinoisstate.edu/generaleducation/ • http://assessment.illinoisstate.edu/about/newsletter.shtml
Illinois State University University Assessment Services Normal, IL 309.438.2135 assessment@ilstu.edu Renée M. Tobin, Ph.D. Derek J. Herrmann Kelly L. Whalen Acting Director Coordinator Graduate Assistant rmtobin@ilstu.edudjherrm@ilstu.eduklwhal2@ilstu.edu