Topic 11 Policy Process Studies: Policy Evaluation Studies

1. Topic 11Policy Process Studies:Policy Evaluation Studies PEDU 6209 Policy Studies in Education

2. Conceptualization of evaluation Fitzpatrick, Sanders and Worthen define “evaluation as the identification, clarification, and application of defensible criteria to determine an evaluation object’s value (worth or merit) in relation to those criteria.” (2004, p.5) Conceptualization of Policy Evaluation

3. Conceptualization of policy evaluation: Accordingly, the constituents of evaluation consist of The evaluation object: It is of course public policy or more specifically education policy. The evaluation criteria: Define the criteria upon which the evaluations are based. They may include Policy objectives: To assess to what extent the identified policy objectives have been fulfilled. Policy measures and programs: To investigate to what extent the designed policy measures and programs have been implemented Policy outcomes: To measure to what extent the targeted outcomes have been realized Cost and benefit of the policy: To assess the cost of the implementation of policy means against the benefits entailed by the policy outcome Conceptualization of Policy Evaluation

4. Conceptualization of policy evaluation: the constituents of policy evaluation consist of … The judgment: The evaluation can be studies and judge by means of Analytical-technical study Interpretive-political study Discursive-critical study Conceptualization of Policy Evaluation

5. Typology of policy evaluation In regards to the purpose of policy evaluation, policy evaluation can be categorized into Formative evaluations aim to provide information for personnel implementing the policy or carrying out the program. Hence, their primary purposes are for the development and improvement of the policy and/or program Summative evaluations aim to pass judgment on the performance of the policy and/or program. They are primarily carried out as accountability mechanism or auditing measures. Conceptualization of Policy Evaluation

6. Typology of policy evaluation In relation to the policy stages, policy evaluation can be categorized into Need assessments: They the studies conducted at the formative stages of the policy process. They are conducted either to evaluate the urgencies of the policy problems or the needs that are required if the identified desirable states of affairs are to be realized. Process-monitoring studies: They are studies carried out to monitor to what extent are the designed policy measures .and programs are implemented in real world situations. They also aim to provide information for the improvement of the effectiveness of the policy instruments in use. Conceptualization of Policy Evaluation

7. Typology of policy evaluation In relation to the policy stages, policy evaluation can be categorized into Outcome studies: They are carried out to assess to what extent have the preconceived policy outcomes been attained. They can also provide information to the revelations of unanticipated consequences (desirable, undesirable or both) of the policy Conceptualization of Policy Evaluation

8. Typology of policy evaluation In terms of the policy context, policy evaluation can be classified into Endogenous policy evaluation: They are studies carried out to assess the internal effectiveness and coherence among various parts of the policy constituents. The most common types of the endogenous evaluation studies are those to assess whether the policy outcomes are in congruence with the policy objectives. Exogenous policy evaluation: They are studies assessing whether the policy substance are in congruence with the policy environment. These type of evaluation studies are especially informative in ever changing policy environment. Conceptualization of Policy Evaluation

9. Frank Fischer’s Conception of Four Levels of Policy Evaluation

10. The framework of practical logical of policy evaluation: Post-positivist methodological approach to policy evaluation: By post-positivism, it refers to "a contemporary school of social science that attempts to combine the discourse of social and political theory with the rigor of modern science. It calls for a marriage of scientific knowledge with interpretive and philosophical knowledge about norms and values." (Fischer, 1995, p. 243) Accordingly, Fischer advocates that policy evaluation research should extend beyond the dominance of empirical-positivism and incorporate practical discourse about rightness of normative claims into policy evaluation research.. Frank Fischer’s Conception of Four Levels of Policy Evaluation

11. The framework of practical logical of policy evaluation: Fischer names such a post-positivist approach in policy evaluation the framework of the "Practical Logic of Policy Evaluation". "It is design to illuminate the basic discursive components of a full or complete evaluation, one which incorporates the full range of both the empirical and normative concerns that can be brought to bear on an evaluation. Structured around four interrelated discourses, the approaches extends from concrete empirical questions pertinent to a particular situation up to the abstract normative issues concerning a way of life." (Fischer, 1995, p. 18) Frank Fischer’s Conception of Four Levels of Policy Evaluation

12. The framework of practical logical of policy evaluation: Accordingly, the framework of practical logic of policy evaluation is made up of First-order evaluation level: Technical-analytical discourse Contextual discourse Second-order evaluation level Systems discourse Ideological discourse Frank Fischer’s Conception of Four Levels of Policy Evaluation

13. Technical-analytical discourse: Program verification By program verification, it refers to concerns "with measurement of the efficiency of program outcome. …The basic questions of verification are Does the program empirically fulfill its stated objective(s)? Does the empirical analysis uncover secondary or unanticipated effects that offset the program objectives? Does the program fulfill the objectives more efficientlythan alternative means available?" (p. 20) Frank Fischer’s Conception of Four Levels of Policy Evaluation

14. Technical-analytical discourse: Program verification "Program verification employs such methodologies as experimental research and cost-benefit analysis. The goal is to produce a quantitative assessment of the degree to which a program fulfills a particular objective (standard or rule) and a determination (in terms of a comparison of input and output) of how efficiently the objective is fulfill (typically measured as a ratio of costs to benefits) compared to other possible means." (p. 20) Frank Fischer’s Conception of Four Levels of Policy Evaluation

15. Contextual discourse: Situational validation Validation focuses on whether or not the particular program objectives are relevant to the situation. …Instead of measuring program objectives per se, validation examines the conceptualizations and assumptions underlying the problem situation which the program is designed to influence. Validation centers around the following questions: Is the program objective(s) relevant to the problem situation? Are there circumstances in the situation that require an exception to be made to the objective(s)? Are two or more criteria equally relevant to the problem situation?" (Pp. 20-21) Frank Fischer’s Conception of Four Levels of Policy Evaluation

16. Contextual discourse: Situational validation "Validation is an interpretive process of reasoning that takes place within the framework of the normative belief systems brought to bear on the problem situation. Validation draw in particular on qualitative methods, such as those developed for sociological anthropological research." (p.21) Frank Fischer’s Conception of Four Levels of Policy Evaluation

17. Systems discourse: Societal vindication "In vindication… the basic task is to show that a policy goal (from which specific program objectives were drawn) addresses a valuable function for the existing societal arrangement. Vindication is organized around the following questions: Does the policy goal have instrumental or contributive value for the society as a whole? Does the policy goal result in unanticipated problems with important societal consequences? Does a commitment to the policy goal lead to consequence (e.g. benefits and costs) that are judged to be equitably distributed?" (p. 21) Frank Fischer’s Conception of Four Levels of Policy Evaluation

18. Systems discourse: Societal vindication As second-order vindication, this type of policy evaluation "steps outside of the situational action context in which program criteria are applied and implemented in order to assess empirically the instrumental consequences of a policy goal in terms of the system as a whole." (p. 21) Hence, the methodological approach adopted by societal vindication is macroscopic-institutional analysis which usually takes the forms of comparative and/or historical-sociological methods. Frank Fischer’s Conception of Four Levels of Policy Evaluation

19. Ideological discourse: Social choice "Social choice seeks to establish and examine the selection of a critical basis for marking rationally informed choices about societal systems and their respective ways of life. …Social choice raises the following types of questions: Does the fundamental ideal (or ideology) that organize the accepted social order provide a basis for a legitimate resolution of conflicting judgments? If the social order is unable to resolve basic values conflicts, do other social orders equitably accommodate the relevant interests and needs that the conflicts reflect?” Do normative reflection and empirical support the justification and adoption of an alternative ideology and the social order it prescribes?” (p. 22) Frank Fischer’s Conception of Four Levels of Policy Evaluation

20. Ideological discourse: Social choice "Social choice involves the interpretive tasks of social and political critique, particularly as practiced in political theory and philosophy. Most fundamental are the concepts of a 'rational way of life' and 'good society'. Based on the identification and organization of a specific configuration or values ― such as equality, freedom, or community ― models of the good society serve as a basis for the adoption of higher level evaluative criteria." (p. 22) The typical methodological approach adopted by this type of evaluative studies is critical approach to political, social and philosophical inquiry. Frank Fischer’s Conception of Four Levels of Policy Evaluation

21. Ideological Choice Because of Critical Discursive Perspective Systems Vindication Because of Interpretive Political Perspective Situational Validation Because of Warrant Since Analytic Technical Perspective Data Technical Verification So, (Qualifier), Conclusion

22. Evaluation Study of MOI Policy for Secondary Schools in HKSAR Topic 11Policy Process Studies: Policy Evaluation Studies

23. A policy measure in search of an objective: Summative or even judgmental evaluation of the efficiency of CMI in comparison with EMI Informative evaluation of the efficiency of preparing secondary-school leavers to become trilingual and bilateral Upholding mother-tongue instruction Summative evaluation of the effectiveness of eliminating mixed-coded teaching in secondary schools in HKSAR Technical-analytical Discourse: Program verification

24. “The Objectives of the Longitudinal Study (1999-2002) are • to trace the academic & personal development of students in schools adopting either Chinese or English as the MOI; • to compare the degree of improvement of students’ academic and personal development in schools adopting either Chinese or English as the MOI; • to compare the language ability (in both Chinese & English) of students in schools adopting either Chinese or English as the MOI; and • to identify facilitating & hindering factors affecting students learning in school adopting Chinese as the MOI.” (p.2 of Tender ref. ED/PR/EMICM/99)

25. The Objectives of the Longitudinal Study (2002-2004) • Study are to find out • The effects of different MOI arrangements at senior levels on students’ learning, e.g. their academic performance, personal development, language ability and high-order thinking skills; • the major factors that enhance/hinder students’ learning in Chinese or English MOI at senior secondary levels; and • the major factors that enhance/hinder the introduction of English as MOI or in part only at senior secondary levels. • based on the findings, to design/recommend measures that support students’ learning under different MOI arrangement for their whole secondary schooling. (Tender ref. ED/P&R/EMICMI/02)

26. To compare the academic and personal development outcomes of students in EMI and CMI schools: Summative, outcome and endogenous evaluation research Instructional paradox in MOI policy outcomes Socio-psychological paradox in MOI policy outcomes School-organizational paradox in MOI policy outcomes Technical-analytical Discourse: Program verification

27. Subjects • Cohorts of Students under Study

28. 00-01 99-00 01-02 02-03 03-04 04-05 05-06 98 Cohort HK CEE 98 AAI F. 2 Tests F. 3 Tests F. 4 Tests HK ALE 99 Cohort F. 1 Tests F. 2 Tests F. 4 Tests HK CEE 99 AAI HK ALE F. 3 Tests

29. Teachers’ Perception of Learning Environment A. Sense of efficacy B. Sense of administrative and collegial support C. Sense of reward & recognition from work D. Sense of autonomy & empowerment E. Feeling of stress in work Students’ Performance & Perception of Learning Environment A. Academic Development a. Language Subject - Chinese Language - English Language b. Content Subject - Mathematics - Science - Social Subjects B. Personal Development a. Self-esteem b. Citizenship Development c. Social Efficacy d. Language & learning attitudes C. Perception of Learning Environment a. Learning process & motivation b. Teacher’s efficacy c. Classroom climate d. Quality of school life EMI HKCEE Results EMI CHIG Students’ Academic & Socio-economic Background Implementation of the MOI Guidance CMI EMI CMID CMI EMI CLOW CMI School Administration & Learning Environment A. Principal’s Leadership B. Organizations of Academic & Disciplinary Orders C. Organizational Climate & Collaborative Culture. • Conceptual Framework of the Project

30. Instructional paradox in MOI policy outcomes (Junior Forms) Technical-analytical Discourse: Program verification

37. Instructional paradox in MOI policy outcomes (Senior Forms; HKCEE) Technical-analytical Discourse: Program verification

48. Instructional paradox in MOI policy outcomes (Access to University Education & HKALE) Technical-analytical Discourse: Program verification

49. Educational Attainment Measure: JUPAS Minimum entry requirement:1. Grade E or above in either 2 AL subjects  or 1 AL subject + 2 AS subjects (other than UE and Chi Lang & Culture)2. Grade E or above in AS UE3. Grade E or above in AS Chi Lang & Culture