Accountability Reporting for California Community Colleges

1. Accountability Reporting for California Community Colleges Patrick Perry Vice Chancellor of Technology, Research, & Info. Systems CCC Chancellors Office

2. Data Preamble • “Information is the currency of democracy.” -Thomas Jefferson • “Get your facts first, then you can distort them as you please.” -Mark Twain • “In the twenty-first century, whoever controls the screen controls consciousness, information and thought.” -Timothy Leary

3. The CCC System • 109 campuses, 72 districts, all locally governed • 2.6 million students (annual unduplicated) • 1.1 million FTES (annual) • 35% white; half over age 25; 70% part-time • No admissions requirements • $20/unit; 40% get fees waived • Highest participation rate of any CC system in US; 25% of all CC students are CCC

4. Participation (and Fees)

5. CCC Chancellor’s Office • Weak authority; powers vested locally • Unitary MIS data collection (1992) • Student, faculty, course, section, session, grade level detail • Data collected end of term, 3x/yr • Used for IPEDS, apportionment, accountability, research, online data mart

6. History of CCC Accountability • Simple reporting, fact books until 1998 • 1998: State provides $300m ongoing in exchange for accountability reporting • “Partnership for Excellence” was born • CCC developed report in isolation • CCC allowed to determine “adequate progress” • “Contingent funding” never triggered • Used 5 metrics to measure system and college-level performance

7. PFE Metrics • Annual volume of transfers to CSU/UC • Annual volume of awards/certificates • Rate of successful course completions • Annual volume of Voc. Ed. Course completions • Annual volume of basic skills improvements (lower to higher level) • 4 of 5 are volume metrics, only 1 rate

8. The State Said: • Your metrics allow for no adequate college comparisons • Your method of determining “adequate progress” is suspicious • You only look good because you are growing • Partnership over (2001), but keep reporting, (until 2004) • we have to spend your money buying energy from Enron

9. What Happened Next • Gov. Gray Davis: recalled for spending money buying energy from Enron • Replaced by “The Governator”

10. The Governator • Likes Community Colleges • Comes from a country that has European “academic bifurcation” (Austria)-university vs trade paths • Attended Santa Monica Community College • Took ESL, PE, bookkeeping, micro/macroeconomics • Transferred to U. Wisconsin-Superior

11. And Arnold Said: • We shall haves deez accountabeelity seeztem for de community collegez. • A bill was passed to create the framework, and eventually the framework was enacted. • Named: Accountability Reporting for Community Colleges (ARCC).

12. Arnold Said: • There shall be no pay for performance, but there will be the ability to compare performance.

13. We Said: • Some metrics will be system only; others will be at college-level • College metrics will be rates (to mitigate size for comparison) • No rankings—we will compare colleges against their “peers” • No $$$=ARCC is a “dashboard” accountability report.

14. Arnold Said: • Colleges need to address their performance annually to the State.

15. We Said: • Colleges are more responsive to their local district Board; annual requirement to take local ARCC results to local Board and submit minutes to State • Colleges must submit 500 word response, which becomes a part of the final report.

16. Arnold Said: • The report shall be done in collaboration with the State, not in isolation.

17. We Said: • The Dep’t of Finance, Leg Analyst, and Secretary of Education shall be a part of the technical advisory committee (along with CCC researchers and stakeholders). • We will either succeed or fail together. • This was a really smart move.

18. ARCC • The Model: • Measures 4 areas with 13 metrics: • Student Progress & Achievement-Degree/Certificate/Transfer • Student Progress & Achievement-Vocational/Occupational/Workforce Dev. • Pre-collegiate improvement/basic skills/ESL • Participation • “Process” is not measured

19. Student Prog. & Achievement: Degree/Cert/Xfer • College: • Student Progress & Achievement Rate(s) (SPAR) • “30 units” Rate for SPAR cohort • 1st year to 2nd year persistence rate • System: • Annual volume of transfers • Transfer Rate for 6-year cohort of FTF’s • Annual % of BA/BS grads at CSU/UC who attended a CCC

20. Student Prog. & Achievement: Voc/Occ/Wkforce Dev • College: • Successful Course Completion rate: vocational courses • System: • Annual volume of degrees/certificates by program • Increase in total personal income as a result of receiving degree/certificate

21. Precollegiate Improvement/Basic Skills/ESL • College: • Successful Course Completion rate: basic skills courses • ESL Improvement Rate • Basic Skills Improvement Rate • System: • Annual volume of basic skills improvements

22. Participation • College: • None yet…but coming. • System: • Statewide Participation Rate (by demographic)

23. Major Advancements of ARCC • Creating a viable alternative to the GRS Rate for grad/transfer rate. • Finding transfers to private/out of state institutions. • Doing a wage study. • Geo-mapping district boundaries. • Creating peer groups.

24. Defining Grad/Transfer Rate • Student Progress & Achievement Rate (SPAR Rate) • IPEDS-GRS for 2-yr colleges stinks: • No part-timers • How do you define degree-seeking? • Tracking period too short • Outcomes counting methodology terrible • AA/AS/Cert counted before transfer • Transfer to 2-yr college is counted

25. SPAR Rate • Defining the cohort: • Scrub “first-time” by checking against past records (CCC, UC, CSU, NSC)

26. SPAR Rate • Define “degree-seeking” behaviorally for CC populations • Not by self-stated intent; this is a poor indicator • Behavior: did student ever attempt transfer/deg-applicable level math OR English (at any point in academic history) • Students don’t take this for “fun”

27. Defining Degree-Seeking Behaviorally • Separates out remedial students not yet at collegiate aptitude • Measure remedial progression to this threshold elsewhere • Creates common measurement “bar” of student aptitude between colleges • Same students measured=viable comparison

28. SPAR Rate-Unit Threshold • CCC provides a lot of CSU/UC remediation • Lots of students take transfer math/Eng and leave/take in summer • Should not count these as success or “our” student • Set minimum unit completed threshold (12) for cohort entrance • Any 12 units in 6 years anywhere in system

29. SPAR Denominator: • First-Time (scrubbed) • Degree-seeking (at any point in 6 years, attempt transfer/degree applicable math or English) • 12 units (in 6 years) • This represents about 40% of students in our system

30. SPAR Numerator • Outcomes the State wants: • Earned an AA/AS/certificate; OR • Transfer: to a 4-yr institution; OR • Become “transfer-prepared”;OR • Completed 60 xferable units • Became “transfer-directed”: • Completed both xfer level math AND English • No double-counting, but any outcome counts • SPAR Rate=51%

31. Tracking Transfers • SSN-level matches with CSU, UC • Nat’l Student Clearinghouse for private, proprietary, for-profit, out of state • Match 2x/yr, send all records since 1992 • Update internal “xfer bucket” • Works great for cohort tracking • Needed method for “annual volume”

32. Tracking Transfers • Annual Volume of Transfers • CSU/UC: they provide these figures based on their criteria • We didn’t want to redefine this • Private/Out of State: NSC “cross-section” cut method • Validated against CSU/UC xfers from NSC source • Added another 30% to annual volumes

35. Transfer: Sector of Choice

36. Demography of Transfer

37. The Rise of The Phoenix

38. Who Transfers to Phoenix?

39. Wage Study • What was the economic value of the degrees (AA/AS/certificate) we were conferring? • Required data match with EDD • Had to pass a bill changing EDD code to allow match

40. Wage Study • Take all degree recipients in a given year • Subtract out those still enrolled in a CCC • Subtract out those who transferred to a 4-yr institution • Match wage data 5 years before/after degree

41. Wage Study • Separate out two groups: • Those with wages of basically zero before degree • Those with >$0 pre wage • The result: The Smoking Gun of Success

43. Mapping Districts • CC Districts in CA are legally defined, have own elections, pass own bonds • We did not have a district mapping for all 72 districts • So we couldn’t do district participation rates

44. Mapping Project • Get a cheap copy of ESRI Suite • Collect all legal district boundary documents • Find cheap labor—no budget for this

46. Peer Grouping • “Peers” historically have been locally defined: • My neighbor college • Other colleges with similar demography • Other colleges with similar size

47. Peer Grouping • Taking peering to another level: • Peer on exogenous factors that predict the accountability metric’s outcome • Thus leaving the “endogenous” activity as the remaining variance • Cluster to create groups • We picked 6 clusters, with a min of 3 in a cluster • Each metric produces different factors, peers, clusters

48. Peer Grouping: Example • Peering the SPAR Rate: • 109 rates as outcomes • Find data for all 109 that might predict outcomes/explain variance • Perform regression and other magical SPSS things • See how high you can get your R2

49. Finding Data • What might affect a grad/transfer rate on an institutional level? • Student academic preparedness levels • Socioeconomic status of students • First-gen status of students • Distance to nearest transfer institution • Student age/avg unit load

50. Finding Data • We had to create proxy indices for much of these (142 tried) • GIS system: geocode student zipcode/ZCTA • Census: lots of data to be crossed by zip/ZCTA • Create college “service areas” based on weighted zip/ZCTA values • Different than district legal boundaries