USING THE CLA TO INFORM CAMPUS PLANNING

1. USING THE CLA TO INFORM CAMPUS PLANNING Anne L. Hafner Campus WASC Faculty Coordinator Winter 2008

2. BACKGROUND • Collegiate Learning Assessment ( CLA) created by Council on Aid to Education, non-profit in NY • Performance assessment of critical thinking, analytic reasoning and written communication (see example) • Assesses “value added” by a college • Value added: standardized gain score that controls for students’ academic abilities(and for selectivity and SES)

3. CLA MEASURES I • Analytic writing task: Make an argument: “in our time, specialists of all kinds are highly overrated. We need more generalists - people who ca provide broad perspectives.” • Critique an argument

4. CLA Measures - Performance Task • Provide students with a real world scenario. Students have 90 minutes to advise the mayor on crime reduction strategies & evaluate potential policies: a.Invest in a drug treatment program b. Put more police on the street • Students provided w document library including memos, statistics, data tables, news articles, etc.

5. Performance Task • Students are required to use an integrated set of critical thinking, analytical reasoning, problem solving, and written communication skills

6. DETAILS I • Common models: cross sectional (freshman and senior cohorts) or longitudinal (follow same kids) • Students’ SAT scores are collected to enable comparison of student’s “expected” CLA score vs. student’s actual CLA score

7. DETAILS II • Both freshmen & seniors are tested. After seniors were tested, a gain score from freshman to senior class can be estimated. This informs the campus how much “value” was added by attending CSULA • Because sample of 100 drawn may not be representative of the class, CAE makes an adjustment so that a school’s “actual” CLA score is compared to its “expected” CLA score (based on students’ SATs)

8. INTERPRETING DIFFERENCES • Differences between actual and expected scores are reported in 2 ways: by points on CLA scale and by standard errors in terms of 5 performance levels (well below expected, below expected, at expected, above expected, and well above expected) At expected is between -1 and +1 standard errors from expected.

9. CSULA Fall 06 Administration • Fall 2006 - 110 students tested • Problems with getting students to participate, paid incentives • One half of students took performance task, one half took writing task • CAE sent report back with some interpretation, also sent data set

10. ANALYSIS: HOW DID OUR FRESHMEN SCORE? • After taking into account their abilities (SAT), our freshmen scored 1.4 s.e. higher thanexpected overall or above expected. • Students performed at expected on performance task and above expected on writing tasks. • CLA sample scored higher than freshman population on SAT and ELM, same on EPT

11. How CSULA Freshmen Did • Our college performed in the middle of the group of all schools nationwide that participated in CLA in 2006 (5th -6th decile)

12. Correlations • With dataset, correlations were run • CLA scale scores moderately to highly intercorrelated • Scale scores correlated moderately with EPT, SAT, HSGPA • Overall CLA performance level did not have any positive correlations other than with CLA scale scores

13. CONCLUSIONS I • CLA sample not equivalent to 2006 freshman class (CAE: not a problem) • CLA sample’s demographics similar to campus demographics • CSULA sample scored higher than expected on tasks • Overall, CLA performance level was not highly correlated with other tests

14. CONCLUSIONS II • Students with English as primary language do not score significantly higher than non-English speakers on CLA • CLA appears to be measuring something different from knowledge & skills measured by the SAT, EPT, ELM and grades

15. FINDINGS Summer 2007 • In summer 07, we received our senior findings and “gain” score • Seniors scored “at expected” • Gain was 55 pts (less than expected) Campus is at 4th decile (between 30th-40th percentile nationwide) • CAE can examine if improvement in average student performance between entry and graduation is in line with similar gains of comparable students at other colleges

16. FINDINGS II • Seniors did less well than freshmen on performance tasks, and on critique an argument tasks. • CSULA’s 6 year graduation rate was “at expected’ 32%

17. MAJOR PROBLEMS • Staff time to recruit students and administer the test • High cost of enrollment incentives • Time required to take test (90 minutes) • Cost to take the test is inexpensive ($20)

18. STRENGTHS • Value added is major focus, rather than absolute performance • CLA is well developed and assesses two key skills: critical thinking/reasoning, and writing skills • Strong psychometric properties, valid and reliable, standardized gain scores enable comparisons • Can work with small sample

19. CRITICISMS • Issue of junior transfers not dealt with in models; different population may need to be also assessed • Uncertainty about validity (consequential) and reliability • Heavy reliance on SAT score as measure of academic ability

20. THE FUTURE • It may be preferable to embed CLA tasks into a freshman required course such as Intro to College or orientation • Senior level: could be embedded in capstone courses or made graduation requirement • CLA could be a useful measure of some GE outcomes

21. THE FUTURE II • Pressure on universities from US DOE for accountability led to Voluntary System of Accountability (VSA) • 2007: CSU Chancellor mandated that all campuses use CLA and report annually for the VSA • CAE is working to create performance standards for CLA (e.g “Proficient” level); could be useful to colleges • CAE is working with CSU to deal with issue of junior transfers