Overreach in High-Stakes Testing: The case of Chile’s National University Admission Test (PSU)

1. Overreach in High-Stakes Testing: The case of Chile’s National University Admission Test (PSU) 9th ITC Conference, San Sebastian, Spain July 5, 2014 Mladen Koljatic Monica Silva Richard P. Phelps

2. Topics of the presentation Types of admission tests The PSU context and problems Lessons to be learned The role of international testing associations

3. Types of admission tests

4. Aptitude Tests or Achievement Tests in the Context of Admission? “ Where school inputs vary widely…academic achievement tests are likely to measure the opportunity to learn more than the ability to learn. [Developing] nations concerned about picking their future talent must consider the possibility that an aptitude test…may be more able to overcome the local differences in school quality” (Heyneman, 1987, p. 253) SOURCE: Phelps, Standardized Testing Primer, 2007

5. Aptitude Tests or Achievement Tests in the Context of Admission? “ Achievement tests are fairer to students because they measure accomplishment rather than ill-defined notions of aptitude; they can be used to improve performance; they are less vulnerable to charges of socioeconomic bias; they are more appropriate for schools, because they set clear curricular guidelines and clarify what is important for students to learn…the movement away from aptitude tests is an appropriate step for U.S. students, schools and universities” (Atkinson, 2001)

6. Purpose of Admission Tests To predict college success without providing an unfair or unjustified advantage to any particular group of test-takers SOURCE: Phelps, Standardized Testing Primer, 2007

7. 2. The PSU context and problems

8. School Context: 2 curriculums

9. The context of Chile´s PSU PSU focused on measuring the SH curriculum… Students who attend the TP tracks the poorest in the nation over 90% wish to pursue tertiary education social mobility highly related to university education in Chile * Source: Mineduc, 2009

10. The context of Chile´s PSU PSU test scores used Admission Financial aid Government subsidies to universities  very high-stakes associated to test scores

11. The context of Chile´s PSU Centralized admission system to all public funded universities (+ some private universities) PSU is run by a department within the largest public university (U. de Chile) Admission system requires: PSU test scores High-school GPA (HSGP) Class ranking No other criterion is considered for application to higher education institutions

12. History of Chile´s PSU From 1966-2003 : aptitude-type test (similar to SAT) From 2004- the present: PSU Achievement tests: measure 100%ofSH curriculum Mathematics Language Science (Chemistry, Physics and Biology) Social Sciences (History, Geography and Economics) Change of tests aimed at evaluating outcomes of an educational reform

13. Project to Develop the PSU TESTS Supported and promoted by the Ministry of Education and council of rectors of public universities (CRUCh) Developed using public funds Project led by an economist and a social psychologist with no training or expertise in the development of high-stakes tests

14. Multiplicity of Purposes for the PSU • Measure the implementation of a new curriculum • Fairly measure mastery of one of the two national curricula (SH) • Incentivize high schools to implement the new curriculum • Incentivize high school students to study more • Predict overall success of students in universities • Predict success across very different types of university programs • Reduce socio-economic disparities in performance

15. “ Evidence of the test´s technical quality should be provided for each purpose“ (St.13.2) as well as “adequate monitoring of potential negative consequences” (St.13.1) Multiplicity of Purposes for the PSU

16. How the PSU Runs

17. Main funder of the PSU since 2007. It holds the purse for CRUCh universities, since they are funded through public monies CRUCH Ministry of Education CRUCH´S COMMITTEE OF TECHNICAL ADVISORS FOR PSU Council of Rectors of Chilean Universities "owners" of the PSU designated by CRUCh as supervisors and official evaluators of the PSU PRESIDED BY CREATORS OF THE TEST U. de Chile agency responsible for developing test items, test assembly, tests administration, test scoring, for CRUCh and associated universities Source: adapted from the Pearson Report (2013)

18. Official Chilean Evaluations of the PSU Eight evaluative reports from CRUCh between 2004-2010: • Pseudo-evaluations: overly optimistic reports aimed to construct a positive image of the test • Do not report any analyses for the TP group • When reporting gaps, omit the TP data from the analyses (without saying so) • Deny the existence of a growing score gap between private paid and municipal school groups, especially those attending the TP track • CRUCh denies access to data bases to independent researchers

19. International Evaluations of the PSU • ETS (2005) • Pearson (2013)

20. Efforts to suppress information 2005: ETS Report not released, according to U. Chile because of secrecy clause imposed by ETS 2012: New audit. CRUCh intended: • Auditors to report to expert team chosen by CRUCh • …to curtail analyses for TP students • ... to control and editfinal report

21. Efforts to reveal information 2007: U. de Chile sued to release ETS report A group of faculty and students represented by a pro-transparency nonprofit organization go to court  judge rules in favor of U. de Chile

22. Efforts to reveal information • From 2007: 5 years of pressure by university student leaders, faculty and leaders of NGOs for new audit • 2012: Funded by the Ministry of Education (maximum amount suggested by CRUCh´s CTA: US$400,000) ETS report is finally released in 2012 thanks to TRANSPARENCY LAWS passed in the country

23. Pearson’s Final Report of the PSU (2013): Recommendations • PSU needs to be revised to advance equity for TP groups • reduction in contents • or study alternatives for the equitable assessment of CH and TP students • PSU is below predictive standardsof other international tests • belying claims from CRUCh’s experts as to the predictive capacity of the PSU

24. Pearson’s Final Report of the PSU (2013): SERIOUS DEFICITS • Inadequate use of PSU cut scores to assign financial help and scholarships • Inadequate use of scores to pass judgment on the quality of schools

25. Pearson’s Final Report of the PSU (2013): SERIOUS DEFICITS • Faulty piloting of test items • DIF analyses does not include factors as socioeconomic status, high school curricular branch (S&H and TP) • Inadequate difficulty level of the PSU Math Test (detected in ETS’s 2005 audit) • Actual reporting of PSU Science score deemed “untenable”: Science PSU requires not one, but three scores

26. Pearson’s Final Report of the PSU (2013): SERIOUS DEFICITS • Inadequate level of analysis expected when equating a high-stakes assessment: misleading information regarding year-to-year equating and pilot calibrations. “THESE ACTIVITIES ARE NOT TAKING PLACE EVEN THOUGH THE DOCUMENTATION PROVIDED SEEMS TO INDICATE THEY ARE.” (p.43) • PSU equating is below international standards ETC…

27. Pearson’s Evaluation of the PSU (2013)… • Comprehensive • Pearson met with stakeholders other than CRUCh • Incorporated some analyses for the TP group (at no extra cost) • Specified new studies that needed (e.g.,, to guide reduction of contents) However… • Recommendations are not prioritized • Highly technical report, not aimed at policy makers • Did not look at effects on enrollment

28. Pearson’s Final Report of the PSU (2013): • Revealed how easy it is to mislead the public when it comes to technical matters • CRUCh deceived the public as to the quality of the tests, in terms of prediction and fairness • CRUCH does NOT reduce contents in the PSU, against Pearson's reccomendation • CRUCh's new commitee of technical advisors (CTA) is controlled by an associate of the PSU test-makers

29. Aftermath of Pearson’s Report... Past Problems = Present Problems Vested interests No real changes in the governance system Lack of transparency No accountability Main problems remain unsolved…

30. Aftermath of Pearson’s Report Students demand an end to the PSU elimination of the test Only HSGPA and class ranking as selection criteria Moratorium not only to the PSU but to other educational tests as well …

31. 3. Lessons to be learned

32. Changes in tests… Some nations that are considering moving towards achievement tests might benefit from looking at the Chilean experience to avoid a proliferation of purposes, test corruption and abuse…

33. Changes entail risks… -Without transparency and full access to data it is tempting to those in power to produce pseudo-evaluations that construct an overly positive image of the quality of the new test -There is a need for a governance scheme that guarantees independent evaluations in order to protect applicants’ rights to be tested by good and fair tests

34. Data access is not only a problem in developing nations… “The SAT item-level information is not frequently available to independent researchers who are not affiliated with ETS or the College Board…” Source: Santelices & Wilson, 2010.

35. Governance Issues in Order to Prevent Test Abuse Test-developing agency should be professional and independent Account for funds received and quality of work Report to external boards of experts that can demonstrate independence from the agency Periodically audit tests for quality and fairness Audit reports & data bases should be public --> TRANSPARENCY AND ACCOUNTABILITY

36. 4. The role of international testing associations

37. Changes in tests… Question to ask: How can international organizations such as ITC play a more active role in promoting good consulting practices by test experts in developing nations?

38. Is there a larger role for testing associations such as ITC in the promotion of good practices worldwide?

39. The ITC Guidelines on Adapting Tests The ITC Guidelines on Test Use The ITC Guidelines on Computer-Based and Internet-delivered Testing The ITC Guidelines on Quality Control in Scoring, Test Analysis and Reporting of Test Scores… The ITC Guidelines for Test Review? The ITC Guidelines for International Consulting?

40. Overreach in High-Stakes Testing: The case of Chile’s National University Admission Test (PSU) Mladen Koljatic Monica Silva Richard P. Phelps 9th ITC Conference San Sebastian, Spain July 5, 2014

41. Predictive validities of the PSU (CRUCh vs Pearson estimates) SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013; CTA

42. Predictive validities: ACT and PSU SOURCE: ACT, Research Summary Services, 1997_1998; Pearson, Final Report Evaluation of the Chile PSU, January 2013

43. Predictive validities: SAT and PSU SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013

44. Predictive validities: SAT and PSU (BUSINESS MAJORS) SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013

45. Predictive validities: SAT and PSU (Education) SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013

46. Percentage-point drop in enrollment of municipal school graduates attending two top universities UC U. de Chile PAA (adm. 2003) 17 32 __________________________________ PSU (adm.2004) 17 29 PSU (adm.2005) 16 25 PSU (adm.2006) 1420 PSU (adm.2007) 13 -- PSU (adm.2008) 11 -- PSU (adm.2009) 11 -- PSU (adm.2010) 12 -- Source: official institutional info

47. Drop in enrolled students from Municipal Schools, other CRUCh Universities Fuente: E. Simonsen, 2009

48. Estimated average 10% drop in enrollment from students from municipal schools at top CRUCh Universities in three yearsNO measures taken…