Why ‘o utcomes’ aren’t enough

1. Why ‘outcomes’ aren’t enough Tim Oates Group Director Assessment Research and Development (ARD)

2. Some system characteristics (indicators?) buoyancy in the labour market for young people in their late teens, until the recent recession a high level of ‘lifelong learning’ after cessation of compulsory schooling decline in employer use of FE relative to the volume of employer-funded training accompanying high flexibility in the structure and content of FE provision persistence in structural skill shortages collapse in provision of ‘classical apprenticeship’ the use of imported labour in skilled segments of the economy partial success in the attempt to use HE as a dominant VET route (including variability in return) poor resilience and responsive in the system in respect of offering places during a period of recession and high youth unemployment

3. Qualifications reform as a principal focus Schools Known locations of learning (schools) Relatively easy to govern Ownership clear – although parental/state responsibility an issue Clear populations and cohorts Legal obligation to attend Solid research on pedagogy Powerful instruments of curriculum control VET (note HE VET ambiguity) Diverse and shifting locations of learning (workplaces, colleges, PTPs) Difficult to control, governance institutions unclear Ownership of/responsibility totally unclear – employers, State, individual Constantly shifting participation patterns Complex patterns of requirements and incentives regarding participation Very little work on workplace pedagogy So qualifications reform will fix it? …. Unlikely

4. A reductive framework QCF Notional learning time (size and weight) Level Sector Rationale Reduction in number of qualifications Rationalisation of elements; new combination of common elements Stimulated explosion in qualifications – 22,000? No….6,450 VQs Increase in QCF category; no commensurate reduction in others 06-07 07-08 08-09 09-10 10-11 QCF 1000 1050 2480 6100 9700 Total 8150 9700 11500 15300 18100

5. ‘Curriculum thinking’ Aims Content Methods Assessment Evaluation After Michael Eraut Curricular and extra-curricular Taught and untaught curriculum Unstated curriculum (ethos, culture) The constructed curriculum Outcomes versus formation

6. Conceptual ground-clearing 1 The term ‘system’ – as in ‘education and training system’ is a metaphor ‘Systems-analysis’ is a theoretical and practical means of understanding the overall operation and affects of a diverse set of education and training arrangments Should we aspire to manage these arrangements as a system – do we have the theoretical and practical tools to do this? Are we prepared for the constitutional consequences of a ‘systems management’ approach?

7. The implications of ‘systems-thinking’ Singapore and Korea – the characteristics of modern industrial strategy UK attitude to industrial strategy With the race for international investment going global, how should Britain address its manufacturing sector in order to boost the economy? And how much is this the government’s responsibility, rather than that of industry and businesses? Industrial strategy has been considered an unfashionable term in the past, with some unable to see beyond British industry being in a slow decline since the 1980s. This view could hamper the focus of policy-makers on manufacturing as crucial to better-balanced growth. Coupled with a devastating manufacturing skills gap, this should be carefully considered in the lead-up to the Budget. Total politics Issue 56 March 2013

8. Conceptual ground-clearing 2 Implicit routes exist in seemingly ‘unified’ systems – you need to make choices of subject and focus Seemingly ‘routed’ systems possess different levels of real flexibility – issues of outcomes (attainments), destinations and switching between routes

9. Conceptual foundation 1 Situated cognition Lave & Wenger Gee Newble & Clarke Seely Brown, Collins & Duguid Sociolinguists - context and language acquisition Bhatkin Brown Boreham and Samurcay

10. Anderson Reder & Simon (1998) • issues of context –bound nature of learning, arguing that this is dependent on the precise content of knowledge being acquired • issues regarding the extent to which abstraction is inappropriate for initial acquisition; iteration of abstract concepts and concrete exemplification is vital • knowledge (and skill) is not resolutely context-bound; generalisation is possible • context is characterised by complexity; there are issues of orientation and selective engagement • Specific skills associated with managing equipment, materials and compounds, measuring, observing, reporting

11. Levels of analysis Generic issues of reliability Occupational Enterprise Job role Individual issues of currency False distinction in that functional competence (held by individuals) can be described as generic but is necessarily constituted in an individual manner and is linked to the circumstances of its formation Work process knowledge and distributed competence

12. Conceptual foundation 2 Learning as well as outcomes French VET system – ‘formation’ German Dual System – ‘Beruf’ English apprenticeship system – ‘timeserving’

13. ‘Curriculum thinking’ Aims Content Methods Assessment Evaluation After Michael Eraut Curricular and extra-curricular Taught and untaught curriculum Unstated curriculum (ethos, culture) The constructed curriculum Outcomes versus formation

14. ‘Curriculum thinking’ Aims Content Methods Assessment Evaluation After Michael Eraut Curricular and extra-curricular Taught and untaught curriculum Unstated curriculum (ethos, culture) The constructed curriculum Outcomes versus formation

15. Science - key stage 3 Materials and properties The National Curriculum 1995 Chemical Reactions • that when chemical reactions take place, mass is conserved; • that virtually all materials, including those in living systems, are made through chemical reactions; • to represent chemical reactions by word equations; • that there are different types of reaction, including oxidation and thermal decomposition; • that useful products can be made from chemical reactions, including the production of metals from metal oxides; • about chemical reactions, e.g. corrosion of iron, spoiling of food, that are generally not useful; • that energy transfers that accompany chemical reactions, including the burning of fuels, can be controlled and used; • about possible effects of burning fossil fuels on the environment.

16. National Curriculum 2007 Chemical and Material Behaviour In their study of science, the following should be covered: • chemical change takes place by the rearrangement of atoms in substances; • there are patterns in the chemical reactions between substances; • new materials are made from natural resources by chemical reactions; • the properties of a material determine its uses.

17. Securing precision of the construct – science practical work specific outcomes – the HE requirement for specific preparation – mastery? persistence (Deakin Crick) precision (Fotheringham) methods of measurement experimental design experimental control and threats to validity psychomotor co-ordination principles and practices of safe working ethics of investigation observation recording teamworking and collaboration ‘beruf’ concepts of proof, adequacy, verification, falsification

18. The number of qualifications in England Comparison with Germany 2005-2007 National Database of Accredited Qualifications (source: QCA-DfES) May 2007 5830 total 938 mainstream academic qualifications 4800 vocational and ‘other’ qualifications Vocational qualifications in the German system (source: BIBB) 4070 total 150 FT State-specific qualifications 420 vocational qualifications in the Dual System of Apprenticeship (3yr employment-based programmes in which around 50% of the 16-19 cohort participate) 3,000 chamber of commerce qualifications 500 continuing education qualifications Note – this is a careful comparison of ‘like with like’ – in neither case does it include industry-owned specific qualifications

19. The number of qualifications in England My conclusion in 2006-07: There is no basis for using 20,000 as the baseline figure for the English system. There is no compelling argument that the system is incoherent, compared with other benchmark nations Comments collected from industry at that time: there are issues regarding gaps – particularly maths - and some duplication in information technology qualifications. Rapid growth in NVQ2 care as a result of change of labour market requirement Oxford work on Skills Survey 2000

20. Qualifications supply and combination QCF implied combination and saliency Policy maker: ‘There are hundreds which are not used…’ Me ‘So what?...they’re not used...’ Policy maker: ‘Remove all with fewer than 500 candidates…’ Me ‘what about the 50 who qualified in bomb disposal?...and…” QCF lacked principles of combination QCF lacked criteria of need (utility and saliency) Material impediments to shared units Relatively unfettered escalation in qualifications Failure of principle or a failure of implementation? Neurotic obsession with system tidiness.

21. Combination: purpose and needs It is vital to differentiate: school-based VET (including work experience) as a component of compulsory general education initial VET in a full-time educational setting employment-based for young entrants to the labour market – focused VET for comprehensive labour market preparation continuing VET for adult employed workers VET for unemployed adults

22. ‘Frameworks’ as a policy instrumentThe importance of higher order criteria Education and training ‘system’ is a metaphor Qualifications arrangements Qualifications frameworks Clear progression routes Absence of inefficiencies Absence of problems of knowledge and skill supply High levels of recognition and trust High levels of motivation and attainment Frameworks have emerged as a relatively recent policy instrument

23. Understanding relations as well as objects Curriculum coherence Curriculum control Explanatory factors Control factors

24. Control factors 1 curriculum content (nc specifications, support materials, etc) 2 assessment and qualifications 3 national framework for qualifications 4 inspection 5 pedagogy 6 professional development 7 institutional development 8 institutional forms and structures (eg size of schools, education phases) 9 allied social measures (linking social care, health care and education) 10 funding 11 governance (autonomy versus direct control) 12 accountability arrangements 13 labour market/professional licensing • allied market regulation (eg health and safety legislation; insurance regulation) Oates T 2010 Could do better: using international comparisons to refine the National Curriculum in England Cambridge Assessment

25. Curriculum control A system must exercise control, it is not that individual agencies should take control: ‘…our purpose in introducing alternative ways to govern curriculum…is not to advocate one approach or another. As analysis by Cochran-Smith and Fries (2001) indicates, disagreements about teaching and, by implication, curriculum, often divides along ideological lines, an outcome that occurs no matter how pragmatic the veneer. A functional approach, by specifying in advance the criteria that an effective curriculum-governance system must meet, lessens the tendency to judge these systems in terms of the political values they represent (eg regulation vs deregulation, public interest vs private interest…’ Schmidt W & Prawat R 2006 p656

26. Qualification frameworks as policy instruments Key distinctions Descriptive frameworks (accommodation – ‘relating’ frameworks Prescriptive frameworks (change) – ‘realigning’ frameworks 30 levels in CINOP project 12 levels in Scotland – the rationality shown by empty cells Functions showing relationships (level, ‘weight’, worth) indicating progression routes rationalising qualifications

27. Qualification frameworks as policy instruments EQF – metaframework Analytic function – labour market mobility as prime aim) NQF – national framework Developing equivalences and rationalising the system as awhole QCF – formula for recognition in addition to framework Reducing the number of qualifications, duplication and redundancy Who is a framework for, exactly? QCA website statement: ‘for learners to navigate the system’

28. Erosion of concept of formation Ditch lessons, schools are toldGeraldine Hackett, Education Correspondent 24 06 2007 ‘State secondary schools are being told to ditch lessons in academic subjects and replace them with month-long projects on themes such as global warming. Quote from official “At the moment most schools are in the traditional mindset, which means they organise the day into a meaningful form take content and divide it up into fragments called timetables. They do it as it has always been done…The idea is to offer less education prescription and more opportunity to interpret the curriculum. Cutting across all subjects are curriculum dimensions; a set of themes including creativity, cultural understanding and diversity.”

29. Classical apprenticeship Internal economics of classical apprenticeship Merits of long duration Formation through long duration Formation as a removal of the need to specify and assess externally Material efficiency of inflexibilities Re-establishing curriculum thinking in general education as well as VET

30. QCF impacts and issues High cost of QCF compliance Small quals with genuine utility removed from the system Some ABs forced out of business - undue market restriction Material impediments to shared units Relatively unfettered escalation in qualifications Comments from contracted developers Failure of principle or a failure of implementation? Neurotic obsession with system tidiness.

31. Insights from German VET External appearance of incoherence: 364 training lines (high labour market mobility) Each set of standards revised on a sector basis (High saliency) Qualification is associated with ‘formation’ not with certification Restriction vital to the system Low complexity in QA arrangements around qualifications Social partnership essential to form of system (the institutional form of curriculum control) (Reuling on modularisation 1995) Is this for England? No naïve policy borrowing.

32. Modualisation and ‘outcomes orientation’ In general education Greater clarity in learning outcomes Early feedback on performance Gains in motivation and engagement Clear conceptual sequencing in material But A very significant increase in assessment load A truncation of learning processes Problems in establishing comparability A narrow focus on assessment objectives, contributing to narrow instrumentalism in schools and colleges

33. Modualisation and ‘outcomes orientation’ In vocational education and training Greater linkage between assessment and requirements of work High specificity, precision and transparency in required outcomes Validity leading to higher reliability More flexible provision, reduced programme cost, meeting LM needs Recognition of prior attainment Qualifications capable of being used across all VET routes But Escalation in cost and complexity of quality assurance Significant problems in reliability and comparability Reducation of employers’ incentives to train (erosion of apprenticeship) Failure to recognise more subtle ‘work process’ skills

34. The 40 functions Oates & Cole CEDEFOP ‘Changing Qualifications’ report (2010) 1 Documenting the outcomes of learning 3 Signalling 6 Controlling flows into specific occupations 7 Empowering citizens 8 Re-aligning control of professions 10 Measuring the performance of the education system 14 Guaranteeing the quality of provision, for funding agencies 16 Affecting the identity of learners 25 Influencing the content of learning programmes 35 Providing feedback to learners (formative and diagnostic function)

35. Post-16 Mathematics – an important case study An emergent gap in qualification arrangements identified through domestic research and international comparisons Research undertaken by King’s College London, sponsored by the Nuffield Foundation An increase in the maths requirements of a broad range of science and social science degree courses A level and AS qualifications too specialist and high level GCSE repeat or retake demotivating, higher level, refined content required ‘Functional skills’ qualis inappropriate; higher level, refined content required The NQF/QCF did not indicate or highlight deficit in qualifications ‘catalogue’ – specific research on needs and requirements was required

36. Concluding remarks 1 (of 3) System tidiness is a misleading preoccupation Vertical progression is vital; clarity in signalling and opportunity Coherence and knowledge symmetry in sectors is vital ‘Level 2’ hairdressing, ‘Level 2 Engineering’ – why is this not immediately a scandal and a problem in the economy? Labour market flexibility and skills supply vital – lessons from Germany Bridging at key points in the system are critical Real systems (‘arrangements’) are messy

37. Concluding remarks 2 (of 3) High quality qualifications arrangements are not dependent on the existence of a qualifications framework Dependence on qualifications frameworks can lead to over-simplistic approaches to the development of effective qualifications arrangements and effective learning provision – signs from the Richards Review Multiple purposes for qualifications are likely – but curriculum coherence is vital; it requires sophisticated and evidence-based policy formation, management of ‘over determination’ and adjuvant policy, combined with subtle monitoring of impact and system performance

38. Concluding remarks 3 (of 3) The assets of outcomes: drive to remove dysfunctional duplication drives analysis of the composition of competence But ‘level’ of analysis is crucial – theory matters robust and explicit method and aim are vital can encourage reductivist move from ‘formation’ and ‘beruf’ can drive policy towards a consideration of ‘policy objects’ and a neglect of ‘complex relations’