EAP and Student Placement

1. EAP and Student Placement Karen Borglum, Ed.D. Assistant Vice President Curriculum and Articulation

2. Walmart “GO” Grant • Spring of 2010: EAP program receives a two-year grant from Walmart • Outcomes of grant • Develop an EAP/Gen Ed curriculum designed to increase student academic success • Establish a common course repository • Implement an aligned curriculum during two consecutive semesters • Engage faculty in data and curriculum evaluation • Engage targeted students in community-building activities • Disseminate the curriculum model

3. Next Steps • Dollars were then used to address placement issues • A UCF Statistician was hired to review data and make suggestions

4. Present Placement Process • Four tests given: essay writing (graded by a faculty member) worth 50%; reading test, grammar test, and usage/mechanics test graded by Accuplacer system for the remaining 50%

5. Goals of Statistical Analysis • To determine if faculty assessments of student essays were reliable across graders. • To determine which of the computerized essay evaluation software packages, most closely agree with assessment of three panel graders

6. Methodology • Sixty students were given entrance essays and were graded by 11 faculty members collegewide. • The three faculty scores were averaged to produce a composite score for comparison in Accuplacer (present system) and the Compass (new system)

7. Findings • Inter-rater reliability when using our faculty to grade essays is .66 (n=60 essays) • Spearman’s Rho coefficients comparing faculty and computer scoring: • Accuplacer-.747 • Compass-.851 • There is substantial variability between faculty graders (especially across campuses), but the average scores produced by groups of graders match up with computerized grading

8. Meetings • Faculty and Deans met over the course of about a year to discuss results, review data, run tests, and determine next steps

9. Meeting History • May 3, 2012: Discussion of history of EAP, review of statistical analysis, preliminary discussion of systems. • Need to discuss Compass testing with Frank Potter from Compass • Need only 3 tests for placement instead of 4: Essay, Reading, and Listening • May 24, 2012: Sent e-mail to Campus Presidents explaining EAP Placement Issue • June 4, 2012: Met with Frank Potter to look at possible cut scores • Indian River, Jacksonville, Miami Dade and Palm Beach use Compass

10. Meeting History Continued • September 10, 2012: Discussion of how to establish cut scores in Compass • 300 units were provided to run a Beta test in reading similar to writing • November 12, 2012: Reviewed data from Beta tests • Determined that we needed linear regression done from Compass • February 6, 2013: James May reviewed his work with Compass Statistician to determine cut-scores

11. Cut-Score Decision • Two issues emerged from Compass data: • 1. Restriction of range issue: not a true population since our students can only place with a score of 66. • 2. Equitable forms issue: the line cut scores are an average of 4 tests, so the placement is obscured

12. Final Decision • Take an average of reading and essay scores to place students into reading, grammar and writing • Collect data on the listening tests to make a determination on a separate placement for the speaking course

13. Cut Scores

14. Cost • Previous year: • # of students tested – 1,100 • # of units used – 3,300 • Price per unit - $ 1.75 (Accuplacer ESL, AKA LOEP) • Total cost - $ 5,775 • Projection (with Listening test): • # of students to be tested for 1 year – 1,200 • # of units needed – 8,320* • Price per unit - $ 1.50 (ACT Compass ESL) • Total cost – 12,480 • Projection (without Listening test): • # of students to be tested for 1 year – 1,200 • # of units needed – 7,120* • Price per unit - $ 1.50 (ACT Compass ESL) • Total cost – 10,680 • *Based on projection that 1/3 of our students are retaking the test – there is a slightly lower cost for a retake for the listening and reading sections.  E-Write uses 3.5 units independently of whether it is a first time or retake.

15. Cost Analysis • The cost of Compass is more than Accuplacer; however, it is a better placement tool. There will be a more accurate and consistent placement for the essay portion of the test. • The Director of Standardized Testing is seeking approval to implement a retake fee of $10 or $15 to offset cost. • The retake fee is NOT being explored because of the possible new placement test; it was already in the works with the current system. We already have a retake fee for students retaking the PERT.

16. Additional Information • Valencia’s Continuing Ed Division is also looking to use Compass for their placement work (we can get a pricing discount) • Faculty and Deans have voted to approve cut scores • Deans have discussed results with their Campus Presidents • The Director of Standardized Testing is supportive of this switch • Agreement to review placement information January 2014 to determine if placement needs to be tweaked • There are mechanisms in place if students/faculty feel that a student is improperly placed upon initial testing

17. Voting • Supportive of new placement structure • Non-supportive of new placement structure