AIR 2004 Annual Forum May 30-June 2 Boston, Massachusetts Shuqin Guo, PhD

1. Issues in the Development of a Model to Adjustment Transfer RateAn Example of Adjusted Performance Measures AIR 2004 Annual Forum May 30-June 2Boston, Massachusetts Shuqin Guo, PhD Director, Research and Assessment Office University of Cincinnati 350 University Pavilion Cincinnati, OH 45219 guosn@ucmail.uc.edu

2. Why Is an Adjusted Model Needed? • Performance measurement is the key to the evaluation of a college or a program. • Questions raised: • What measurements should be used to evaluate a college or a program? • Is it fair to use the one-fit-all measurement for all the organizations if factors impacting performances are not controllable by an organization? • Adjusted Performance Measures (APM) may be one of the solutions.

3. What Is Adjusted Performance Measures (APM) (Stiefel, L. et al.) • It is a quantitative estimate of an organization’s output/outcomes. • It takes account of variables or factors that are outside the individual organization’s control.

4. Steps to Construct APM (See Stiefel, L. et al.) • Identify a set of variables that capture uncontrollable factors impacting an organization’s performance • Estimate a regression equation in which the performance measure of interest is regressed on this set of variables, pooling data on a set of similar organizations • Calculate a predicted performance for each organization by plugging in the regression equation • Calculate the difference (APM) between the predicted value and the actual value for each organization • APM is meant to measure performance while accounting for the organization’s working environments or characteristics

5. Selection of Organization’s Uncontrollable Factors • Select as many observable uncontrollable variables as possible • Be cautious of the correlation between the selected variables and unselected controllable variables. High correlation between them may cause the results misleading.

6. In the Case of Transfer Rate Adjustment • Transfer rate is one of the performance indicators for community colleges. • Factors that have impact on transfer are environmental or performance related in nature. Environment factors are uncontrollable by a college. • Controlling environmental factors is needed when transfer rate is counted as a performance indicator.

7. The Selection of Organizational Uncontrollable Factors in the Transfer Rate Adjustment • Academic preparedness of the incoming freshmen students • Distance to the nearest four-year public university • Geographic information of the college location • Local unemployment rate • Local median family income • Percentage of local population under poverty line

8. Uncontrollable Factor Selection (Continue) • Incoming students with academic disadvantage • Percentage of students taking general ESL, survival ESL, and Vocational ESL (LEP) • Percentage of students taking basic skill courses • Age of student population • Financial aid

9. Definitions of Transfer Rate Transfer rates used in other studies in the literature • Number of transfer/Number of students who completed an associate degree • Number of transfer/Total number enrolled in a transfer program • Number of transfer/Total number of headcount • Number of transfer/Total number of FTE • Number of transfer/Total number of entering high school students

10. Definition of Transfer Rate Used in This Study • Definition of Cohort (denominator) • First-time students entering a community college in the fall of the year with an intention to transfer by • Completing a minimum of 12 transferable credit units, and • Either attempting one or more transferable math courses, or • Attempting one or more transferable English courses • Each cohort was tracked for six years for transfer (numerator). • Transfer rate = number of transfer in the six year period/number in the cohort

11. Methodology Data • Four cohorts were selected (cohort 9394, cohort 9495, cohort 9596, and cohort 9697). • Cohort data were retrieved from the student database at a community system office. • Transfer data were obtained from the state university system offices and the National Student Clearinghouse. • The two sets of data were merged by SSN. • Financial data of the county a college is located was obtained from the website of the state Department of Finance. • Unemployment data was obtained from the EDD website.

12. Methodology (Continue) Student Preparedness Data • SAT9 scores for incoming freshman were obtained from Department of Education. The data were matched with student cohort data using fuzzy match method, which yielded about a 50% hit rate. Each college was then calculated a mean SAT9 index of incoming students’ academic preparedness.

13. Methodology (Continue) Variables • Dependent variable: Raw Transfer rates • Independent variables: The environmental factors mentioned above

14. Analysis • Normality of the data was checked and data were normalized accordingly. Log or square-root method was used for most of the variables that needed transformation. • Dummy variables were created to detect cohort effect. • The raw transfer rates were regressed on the environmental factors. • A search was done for the best regression model.

15. Results • The final model • The following independent variables were included in the final model: • An estimated student academic preparedness (SAAP_N) • Percentage of students age 30 and over (AGE 30PL_N) • Miles to the nearest 4-year public university (ML_UN_N) • County median family income (FAMINCOM) • Percentage of students taking Basic Skill courses (BS_N) • Percentage of students with financial aid based on need (FAID_N_N)

16. Model Summary

17. Coefficients for the Final Model to Adjust Transfer Rate

18. Check for the Multicollinearity of the Variables in the Model

19. Predict Expected Transfer Rate Expected transfer rate was calculated for each college based on the model. Expected transfer rate = constant + b1*SAAP_N + b2*AGE30pl_N + b3*ML_UN_N + b4*FAMINCOM + b5*BS_N + b6*FAID_N_N AGE30PL_N, ML_UN_N, FAMINCOM_N BS_N, FAID_N_N are normalized.

20. Interpretations of the Results • Student academic preparedness has a positive impact on transfer rate. That is, if a college has a higher average SAT9 scores for the first-time freshmen, the college tends to have a higher transfer rate; • Median family income also has a positive impact on the transfer rate. If a college is located in a county with higher median family income, the transfer rate for that college is more likely to be higher.

21. Interpretations of the Results (cont.) • Distance to a California 4-year public university has a negative impact on transfer rate. • Percentage of students taking basic skill courses has a negative impact on transfer rate. • Proportion of student population aging 30 and plus has a negative impact on transfer rate. • Percentage of students who are on financial aid based on need also has a negative impact on transfer rate.

22. Criterion for Identifying Consistent Low Transfer College Interquartile Range (IQR) Distance was used to identify colleges whose actual transfer rate is considerably lower than the expected transfer rate. Terms • Difference=actual rate-expected rate • 25th percentile=25th percentile of the differences • 75th percentile=75th percentile of the differences • IQR=a constant number of the interquartile range of the differences

23. Criterion for Identifying Consistent Low Transfer College (cont.) • IQR Distance = (difference – 25th percentile)/IQR, if the actual difference is equal or less than 25th percentile of actual difference, or • IQR Distance = (actual difference – 75th percentile)/IQR, if the actual difference is equal or greater than 75th percentile of the actual difference

24. Criterion for Identifying Consistent Low Transfer College (cont.) Consistent Low transfer college: IQR Distance <= -1.5 three years in a row

25. The Result of the Adjustment • Smaller and remote colleges got credits for their performance. • No colleges were identified as outliers three years in a row, but red flags were raised for some colleges who were identified as low transfer colleges for two years.

26. How to Use the Results of This Study • The study provides an adjusted transfer rate for each cohort at each college after controlling the environmental factors that have impacts on transfer. • The study helps to identify outliers (colleges) for transfer. • Each college could use the adjusted transfer rates for the 4 cohorts to see the performance improvement.

27. Limits of the Adjusted Transfer Rate Model • Controllable variables are not included in the model. • Results should be interpreted cautiously and reasonably.

28. Summary • The model takes accounts of the environmental factors that are outside of a college control, thus provides a fair measurement for each college. • Rather than ranking colleges, IQR difference is used to identify outliers for good or poor performance.

29. Limits of Adjusted Performance Measures • When observable but uncontrollable factors are highly correlated with the controllable variable(s) that impacts the performance, that variable(s) can be badly over- or underestimated. (Brooks, 2000) • Small sample size may result in misleading information. • Sometimes it is hard to make a distinction draw between controllable and uncontrollable variables.

30. Discussion: • To use raw or adjusted measures? • Adjusted Performance Measures could be better than raw measurement if used properly. • APM can be used in combination with other performance measures for program evaluation. • Good quality of APM depends on the quality of data and the content knowledge of the analyst.

31. References • Transfer Capacity and Readiness in the California Community Colleges (2002). Published report, Chancellor’s Office, California Community Colleges • Bahr, P., Perry, P., & Hom, W. (2002). Low Transfer Colleges: Methodology for Equitability in Identification. Paper presented at the 41st RP conference, Pacific Grove, CA • Rubenstein, R. Schwartz, A. E. & Stiefel, L. (2003). Better Than Raw: A Guide To Measuring Organizational Performance with Adjusted Performance Measures. Public Administration Review Vol. 63, No. 5 • Brooks, A. (2000) The Use and Misuse of Adjusted Performance Measures. Journal of Policy Analysis and Management, Vol. 19, No. 2 323-334

32. References (cont.) • Stiefel, L., Rubenstein, R. & Schwartz, A. E. (1999). Using Adjusted Performance Measures for Evaluating Resource Use. Public Budgeting & Finance Fall 1999 • Townsend, B. (2002). Transfer Rates: A Problematic Criterion for Measuring the Community College. New Directions for Community Colleges, No. 117, Spring 2002. • Mela, C. & Kopalle, P. (2002). The Impact of Collinearity on Regression Analysis: the Asymmemetric Effect of Negative and Positive Correlations. Applied Economics, Vol. 34 667-677

33. Contact Information Shuqin Guo, PhD Director of Research and Assessment University of Cincinnati 350 University Pavilion Cincinnati, OH 45219 Phone:513-556-1573 Email: shuqin.guo@uc.edu