Assessment, reporting accountability: Unproductive peeeroductive relationships

1. Assessment, reporting & accountability: Unproductive & peeeroductive relationships National Symposium on Assessing English as a Second/Other Language in the Australian Context, University of New South Wales, 20-21 February, 2009. Helen Moore University of New South Wales helenmmoore@optusnet.com.au

2. Moore Accountability 20 Feb. 2009 Outline Underlying tensions and their relation to accountability. The recent/current (?) solution to the problem of accountability. Application in the AMEP & LLNP. The centrality of assessment & reporting. Unproductive outcomes. Suggested beginnings in rethinking accountability in more constructive & productive ways.

3. Moore Accountability 20 Feb. 2009 Inherent Tensions in Modern States From Rose, 1993: State authorities rely on experts to provide the knowledge that underpins policy-making and implement their policies. These policies are legitimated, i.e. as seen as right and valid, precisely because expertise is seen as independent of state authorities.

4. Moore Accountability 20 Feb. 2009 Inherent Tensions in the Modern State (cont.) But at the same time as state authorities rely on independent expertise: they are held responsible for the policies they develop and their implementation and they resist centres/sites/bodies that are independent of their control (Hindess, 1997: 21).

5. Moore Accountability 20 Feb. 2009 ? A Problem/Paradox/Tension in the Relationship between State Officials and Experts How can experts be controlled without undermining the independence that legitimates the expertise that underpins government policies? i.e. the problem of accountability.

6. Moore Accountability 20 Feb. 2009 The Recent/Current (?) Administrative Solution to this Problem The New Public Management (NPM): 1980s: began in NZ late 1980s,1990s & 2000s: embraced by Australian governments; also USA, Canada, Britain & some other European countries.

7. Moore Accountability 20 Feb. 2009 New Public Management (NPM) (cont.) Some key elements (from Pollitt, 2003: 27-28): a focus on outputs and outcomes (rather than inputs and processes) use of contracts (rather hierarchical relationships) “deployment of markets (or market-type) mechanisms (MTM) for the delivery of public services” an emphasis on measurement using “performance indicators” and explicit “standards”

8. Moore Accountability 20 Feb. 2009 NPM: Underlying Principles Mobilising self-interest is the most effective way to influence human behaviour (public choice and agency theory) The proper role of state authorities is to “steer” rather than “row”. (Osbourne & Gaebler, 1993)

9. Moore Accountability 20 Feb. 2009 NPM (cont.)  For “Steerers” (= State Officials): Incentives should lie with: policy-making/advising regulating and monitoring service provision (as distinct from providing services). Their role: to protect state resources (“taxpayers money”) from “provider capture”.

10. Moore Accountability 20 Feb. 2009 NPM (cont.)  For “Rowers” (= “Providers”): Incentives should lie in: competition for state resources, i.e. “market discipline” coupled with evaluation against defined and measurable criteria, i.e. “performance indicators”. Their role: to deliver clearly defined services.

11. Moore Accountability 20 Feb. 2009 ? The “provider-purchaser split” 1996 National Commission of Audit: “Where the government wishes to provide access to a particular service for a group of clients, it should be a purchaser rather than a provider of services. (A purchaser/provider relationship is one in which the purchaser is the party who decides what will be produced, and the provider is the party who delivers the agreed outputs and outcomes).” cited in Department of Immigration, Multiculturalism & Indigenous Affairs, Report of the Review of Settlement Services for Migrants & Humanitarian Entrants, 2003, p. 35.

12. Moore Accountability 20 Feb. 2009 NPM Accountability Techniquesin Service Provision: the AMEP & LLNP Contracts (competitively awarded) Performance indicators

13. Moore Accountability 20 Feb. 2009 Accountability through Contract Specifications E.g. draft LLNP tender (2006/7-2008/9): 158 pages 24 chapters 7 page glossary of 106 terms 16 attachments

14. Moore Accountability 20 Feb. 2009 Key Performance Indicators (KPIs) ? the Central Role of Assessment & Reporting AMEP: reach, retention, results. LLNP: retention, participation, attainment, quality. Attainment: clients must achieve “at least one NRS indicator from any two macro-skills” Quality: rests on verifying assessments. All KPIs are quantitative measures — privileged because they are “a tangible means of assessing contract outcomes” (Zifcak, 2001: 95).

15. Moore Accountability 20 Feb. 2009 Do NPM approaches “work”? Pollitt (2003): NPM “preaches a strict doctrine of performance measurement but seldom applies this rule to itself” (p. 26) Large, well-resourced evaluations of NPM in Australia & NZ have been unable to demonstrate with any precision that costs have been cut or efficiency increased (p. 40) E.g. Auditor-General’s 2001 review of the AMEP found no appreciable cost savings following the introduction of competitive contracting (p. 44, para. 1.46).

16. Moore Accountability 20 Feb. 2009 Unproductive/Dysfunctional Outcomes A narrowing of accountability to detailed contract specifications and to what is precisely quantifiable— skews policy and program objectives against non-measurable values, including trust (Zifcak, 2001:95); silences feed-back loops between providers and government and between government agencies themselves (Althaus, 1997). displaces institutional intelligence in favour of “shallow rituals of verification” – audit threatens to become a form of learned ignorance” (Power, 1997: 123). “Numerical criteria admit of no contradiction except in their own terms. Against them, claims from professionalism, social science, values or welfare are rendered less persuasive.” (Zifcak, 2001:95)

17. Moore Accountability 20 Feb. 2009 An example re teaching low literacy learners Teacher (explaining why she couldn’t teach to learner needs as she saw them): “I’m concerned that DIMA’s going to look and say, ‘Oh, look at that group at [name of Centre]. How slack are they? What are they teaching? They’ve only achieved two LOs in the whole year.’ And there is that pressure. If we don’t perform, boy, we lose the contract and we don’t have a job.”

18. Moore Accountability 20 Feb. 2009 To what extent are reach, retention and results accurate measures/indicators of provider performance? Other independent & more decisive variables: Employment/unemployment situation in Australia Migrant source countries (educational infrastructure, culture, political stability) clients’ pre-embarkation experiences (torture & trauma) Clients’ previous education (shown in research to be the greatest influence on rates of progress and achievement, e.g. Watt & Lake, 2004; Spaventa, 2006 ).

19. Moore Accountability 20 Feb. 2009 NPM: Effects in its Current Form Current KPIs: doom the AMEP and LLNP to constant criticism and undermining of their legitimacy. Competition: instils fear ? blocks off feedback loops & truthful information that could improve the situation instals incentives that progressively undermine teacher professionalism and quality provision (e.g. employment of casuals; salaries cut from $49 to $35 per hour; sub-standard premises & resources). Current verification processes rely on paper trails that: are progressively more onerous & divert teachers from teaching bear little/no relation to the actuality of providers’ or clients’ experiences (see Moore, 2008, mimeo).

20. Moore Accountability 20 Feb. 2009 Towards More Productive Forms of Accountability Revisit and rethink: “Where the government wishes to provide access to a particular service for a group of clients, it should be a purchaser rather than a provider of services. (A purchaser/provider relationship is one in which the purchaser is the party who decides what will be produced, and the provider is the party who delivers the agreed outputs and outcomes).”

21. Moore Accountability 20 Feb. 2009 Rethinking the Provider-Purchaser Split Is the best way of ensuring accountability to define/describe: government as a purchaser? (—what about its role in ensuring the public good?) state officials as the “the party who decides”? (—how can good decisions be made if the decision-makers are insulated from the histories, contexts and short- and long-term consequences of their decisions?) educational programs as “what will be produced”? (—what is gained/lost by treating education as a production line?)

22. Moore Accountability 20 Feb. 2009 Rethinking the Provider-Purchaser Split (cont.) Is the best way of ensuring accountability to define/describe: educators as “the party who delivers”? (—what is the effect of imagining educators as delivery people?) English language development as “agreed outputs and outcomes”? (—why have outcomes been narrowed to outputs? to what extent are these outputs within providers’ control? to what extent do they capture providers’ actual performance? ) learner results, program reach & client retention as central to reporting outcomes? (—what should be the place of progress measures in a comprehensive approach to accountability? what are the effects on programs & teaching?) non-measurable outcomes as impossible/irrelevant to report? (—is it necessary for ways of reporting on non-measurable outcomes to be excluded/peripheral? how could non-measurable outcomes be included? what would be the effect?)

23. Moore Accountability 20 Feb. 2009 Re-working Accountability Relationships between State Authorities & Expert Service Providers Networked: relationships between government & providers, and between providers themselves ? (re)-building trust (re)-establishing dialogue

24. Moore Accountability 20 Feb. 2009 Re-working Accountability Relationships (cont.) Networked: knowledges across and between government & providers ? (re)-instituting respect (re)-valuing professionalism (on both sides) (re)-vitalising and opening up research.

25. Moore Accountability 20 Feb. 2009 Networked: experiences for government officials & service providers field visits to each others’ work sites? work placements in each others’ work sites? ???

26. Moore Accountability 20 Feb. 2009 References Alford, J., & O'Neill, D. (Eds.). (1994). The contract state. Melbourne: Centre for Applied Social Research, Deakin University. Althaus, C. (1997). The application of agency theory to public sector management. In G. Davis, B. Sullivan & A. Yeatman (Eds.), The new contractualism? (pp. 137-153). Melbourne: Macmillan. Auditor-General. (2001). Management of the Adult Migrant English Program Contracts: Department of Immigration and Multicultural Affairs (No. 40 2000-2001 Performance Audit). Canberra, ACT: Australian National Audit Office. Davis, G., Sullivan, B., & Yeatman, A. (Eds.). (1997). The new contractualism? Melbourne: Macmillan. DEST. (2005). Request for tender for the provision of Language, Literacy and Numeracy Programme (LLNP) services, RFT PRN 7767: LLNP 2006-07--2008-09 Draft programme guidelines as at 12 November 2005. Retrieved 04.23.08 from http://www.llnp.deewr.gov.au/NR/rdonlyres/502A386E-86B5-42D3-8512-8672A349EF6D/12801. Canberra: Australian Government Department of Education, Science & Training. DIMIA. (2003). Report of the review of settlement services for migrants and humanitarian entrants. Canberra: Department of Immigration Multiculturalism & Indigenous Affairs, Commonwealth of Australia. Hamp-Lyons, L. (2006). The impact of testing practices on teaching: Ideologies and alternatives. In J. Cummins & C. Davison (Eds.), The international handbook of English language teaching (Vol. 1, pp. 487-506). Norwell, MA: Springer. Hindess, B. (1997). Politics and governmentality. Economy and Society, 26(2), 257 - 272.

27. Moore Accountability 20 Feb. 2009 References (cont.) Howie, T. (2006). Struggling with assessment of ESL/EAL learners in Australian Technical and Further Education: Can identity fit in? In K. Cadman & K. O'Regan (Eds.), Tales out of school (pp. 86-103). Flinders Park, South Australia: Australian Council of TESOL Associations. Leung, C., & Rea-Dickens, P. (2007). Teacher assessment as policy instrument: Contradictions and capacities. Language Assessment Quarterly, 4(1), 6-36. McKay, P., & Brindley, G. (2007). Educational reform and ESL assessment in Australia: New roles and new tensions. Language Assessment Quarterly, 4(1), 69-84. McNamara, T. (2001). Language assessment as social practice: Implications for research. Language Testing, 18(4), 333-349. Moore, H. (1996). Why has competency-based training become the 'solution'? Prospect, 11(2), 28-46. Moore, H. (2008, June). Outsourcing, competitive contracting and codes of silence. Paper presented at the Paper presented at the 33rd Annual Conference of the Applied Linguistics Association of Australia, Sydney University. Moore, H. & Dumendin, I. , 2009. (mimeo) “Unveiling” the performance in a teacher-based assessment for an adult ESL program. Mulgan, R. (2003). Holding power to account: Accountability in modern democracies. Houndmills, Basingstoke, Hampshire: Palgrave Macmillan. Mulgan, R. (2006). Governmental accountability for outsourced services. Australian Journal of Public Administration, 65(2), 48-58. National Commission of Audit. (1996). Report to the Commonwealth government. Canberra: Australian Government Publishing Service.

28. Moore Accountability 20 Feb. 2009 References (cont.) Niskanen Jr., W. A. (1994). Bureaucracy and public economics. Aldershot: Edward Elgar. Osbourne, D., & Gaebler, T. (1993). Reinventing government: How the entrepreneurial spirit is transforming the public sector. New York: Plume. Pollitt, C. (2003). The essential public manager. Maidenhead: Oxford University Press. Power, M. (1997). The audit society: Rituals of verification. Oxford: Oxford University Press. Rose, N. (1993). Government, authority and expertise in advanced liberalism. Economy and Society, 22(3), 283-299. Shohamy, E. (2001). The power of tests: A critical perspective on the uses of language tests. London: Pearson. Smith, D. (1990). The conceptual practices of power: A feminist sociology of knowledge. Toronto: University of Toronto Press. Spaventa, M. (Ed.), 2006. Perspectives on Community College ESL, Volume 1: Pedagogy, Programs, Curricula and Assessment (pp. 201-218). Alexandria, VA: TESOL. Spurling, S., Seymour, S., & Chisman, F. P. (2008). Pathways and Outcomes: Tracking ESL Student Performance: A Longitudinal Study of Adult ESL Service at City College of San Francisco. NY: CAAL. http://www.caalusa.org/publications.html, viewed 10/07/08. Watts, D. L. E., & Lake, D.M. (2004). Benchmarking adult rates of second language acquisition & integration: How long and how fast? Final Report. Retrieved 4/12/2007 from http://www.language.ca/pdfs/Benchmarking%20Adult%20Rates%20of%20Second%20Language%20Acquisition%20and%20Integration1.pdf Zifcak, S. (2001). Contractualism, democracy and ethics. Australian Journal of Public Administration, 60(2), 86-98.