A Diagnostic Instrument to Support Student Writing and General Education Assessment

1. A Diagnostic Instrument to Support Student Writing and General Education Assessment HLC Annual Conference April 19, 2016 Ruth E. Cain, Ed.D., Director of Assessment Crystal R. Doss, Ph.D., Coordinator of Writing Assessment University of Missouri – Kansas City

2. Overview How the results of a directed-self placement writing assessment are used: • by students to enhance their ability to read and write critically, • by upper-division faculty to identify areas of needed support to enhance student writing achievement, and • by the institution to assess student achievement of the general education program’s written communication learning outcome.

3. University of Missouri - Kansas City • Member of the four-campus University of Missouri System • Founded in 1933 as the private University of Kansas City; became part of the UM System in 1963 • Located in mid-town KCMO • Member of APLU and its Coalition of Urban Serving Institutions • 13 academic units on two campuses • Degree programs: Bachelor’s (53), Master’s (52), Specialist (4), Doctoral (14) • Students - Fall 2015: 8,370 undergrad; 5,446 grad/professional • 24% are racial/ethnic minorities • Large percentages of undergrads are: transfer students (each year, over 50% of new undergrads); Pell Grant recipients (40%); first-generation college students Original Kasey the Kangaroo, Walt Disney, 1936

4. RooWriter History • 2009: College Writing Board was formed to reexamine writing assessment at UMKC; DSP chosen over barrier exam • 2010: College of Arts and Sciences approves RooWriter to replace previous assessment instrument • 2011-2012: College Writing Board become the University Writing and Reading Board; initial draft of website designed; RooWriter model proposed to and approved by Faculty Senate, Student Government Association, Deans’ Council, and all academic units. • 2013: Coordinator of Writing Assessment and evaluators are hired; RooWriter site launches in September.

5. UMKC General Education Core • Discourse I, II, and III(3-course communication sequence) • Gen Ed outcome: Communication Skills: • Students will: • develop effective written, oral and visual/spatial communication competencies and the ability to communicate with a variety of audiences. • interpret information presented in a variety of formats. • evaluate the context of their messages and use proper form and style to engage audiences using a variety of media. • demonstrate their critical engagement with audiences through reading, listening, reflecting and responding.

6. RooWriter in Undergraduate Curriculum • Students may submit essays at any time and multiple times throughout their undergraduate careers. • To satisfy the RooWriter graduation requirement, students must submit essays after passing Discourse 200 (or equivalent) and before completing 90 hours. • Transfer students with credit for Discourse 200 and 90 hours should take RooWriter within their first semester.

7. RooWriter Essay Process • Students choose and read a reading packet consisting of 5-7 sources (scholarly and reputable popular sources), 20-30 pages total. • Students log in to the RooWriter site, confirm their reading packet choice, and then receive an essay prompt. • If the student chooses a social category reading packet, they have the option to share that essay anonymously with relevant stakeholders (community organizations, student affairs) • Students have 72 hours to write a 750-1500 word essay that • addresses the prompt • uses at least three reading packet sources • cites sources in MLA or APA style

8. RooWriter Evaluation Process • Double-blind evaluationby two trained RooWriter evaluators • Graduate students from a range of disciplines • Training and professional development every semester • Rubric: • Feedback • Embedded comments • Summary comments: Overall Impression; Final Constructive Feedback • Resources

10. RooWriter and Student Feedback

15. Reasons for RW Visits to Writing Studio

16. Sample RooWriter Data – All Essays 2014-15 N = 1,727

17. Comparative RooWriter Data – 2013-14 and 2014-15 – Percent Achieving 3 or Higher

18. Student Variables • Age at time of essay • First-time college / transfer (institution) • Race / ethnicity • First-Generation status • Athlete / Non-Athlete • Class level • Academic unit / major • Gender • Citizenship • ACT score

19. DBA/Academic Unit Disaggregated by Academic Unit – Overall and by Each Criterion (Depth, Breadth of Analysis)

20. RooWriter and Lower Division Courses (Discourse I and II) • RooWriter taken after completion of Discourse II (or equivalent, for transfer students) • Use aggregate RooWriter results in conjunction with gen ed assessment and assessment of the Discourse series to inform alterations in the Discourse I and II experiences • Present strategies to students on ways to resolve writing concerns emerging from the data

21. RooWriter and Transfer Students • Transfer students who have completed an AA degree, the MO transfer core, or Discourse II equivalent are required to complete the RooWriter and to complete Discourse III • Utilize aggregate results for transfer students to ensure that they are arriving at UMKC at desired level of competency (three or higher on each criterion) • If level if not achieved, faculty will need to determine changes in gen ed requirements for transfer students

22. RooWriter and General Education Assessment • Gen Ed outcome: Written Communication Skills • Faculty panels review of student artifacts (Spring 2016) • Three-Year Cycles – Discourse I, II, or III each cycle • Track student development across the three years of Discourse • Compare aggregate General Education results to RooWriter results across the rubric criterion (similar but not identical) • Look for patterns – consistencies or inconsistencies across the two rubrics • Use in conjunction with indirect measures (NSSE Experiences with Writing Module; UMKC Graduation Survey) to identify ways to enhance support of student writing achievement

23. RooWriter and Upper Division Courses • Summary and custom reports for programs and units • Instructor-generated reports • Course • Adapt syllabus or assignments • Writing Studio consultations • Library consultations • Individual • Conferencing • Reflection • Portfolio • Advising

24. Lessons Learned and Next Steps • Reporting Features • 2013-2014: Development of RooWriter Custom Reportsand Faculty/Advisor Reports • 2014-2015: Development of Excel format for Custom Reports (Available November 2015) • 2015-2017: Develop resources for upper division faculty; close the loop

25. Lessons Learned and Next Steps • Placement options: • Writing Studio • ENG 351: Research Writing Studio • Honors 330: Honors Thesis/Project Practicum

26. Next Steps • Student retention and persistence efforts • UMKC member of the HLC Retention and Persistence Academy • Recently signed on with a company for predictive modeling software • RooWriter results will be one piece of student data that will go into the predictive model (each student’s score is recorded in the SIS) • Won’t know for several years how / if RooWriter contributes to the model

27. Discussion • Describe your institution’s DSP instrument: • At what point do students complete the instrument? • What is the nature of the instrument? • How is writing evaluated? • Who receives the results? • How could this tool be used to enhance general education assessment efforts on your campus? • How is the DSP used develop a writing culture on your campus? • Student resources • Curriculum changes • Writing center • Faculty development • Online resources

28. Questions? Suggestions? • Contact Information Crystal Doss, dosscr@umkc.edu Ruth Cain, cainre@umkc.edu