MEASURING THE OUTCOME OF EDUCATION FRANZ SCHOTT UNIVERSITY OF TECHNOLOGY, DRESDEN, GERMANY

1. MEASURING THE OUTCOME OF EDUCATIONFRANZ SCHOTTUNIVERSITY OF TECHNOLOGY, DRESDEN, GERMANY ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

2. Introduction • The topic of this conference is: “Employability - Mobility - Flexibility. European demands challenging Religion and Education”. We read in the announcement of this conference: • “The aim of education and training is focused on a concept of flexible skills and competencies that serve employability, mobility and flexibility.” • “The Conference will encompass and discuss the European perspectives of education …” • One central perspective on education since the last decade is the view on the efficiency of the educational system • This leads to demand more output-orientation of the educational system instead of input-orientation (input is for example: curriculum, training of teachers, teaching materials) • The output of an educational system is what the addressees of this system have learnt, in other words: it is the outcome of education • International comparative studies of the outcome of education (e.g. Programme for International Student Assessment: PISA) brought questions about the efficiency of national educational systems and about measuring the outcome of education ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

3. Introduction • We will discuss the topic: measuring the outcome of education as follows: • On measuring • On the outcome of education • Purposes of measuring the outcome of education • Problems of measuring the outcome of education • Summary • Literature ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

4. 1. On measuring • We have to differentiate measuring in a narrow sense and in a broad sense: • Measurement in a narrow sense • This concept is used in the physical sciences and defines: Measurement is the process or the result of determining the magnitude of a quantity, such as length or mass, relative to a unit of measurement, such as a meter or a kilogram (see e.g.: Deutsche Norm DIN 1319-1, 1995). • Measurement in a broad sense • In social sciences there are no generally binding definitions of units of measurement such as meter or kilogram • Therefore we find in social sciences a broader concept of measurement: • Measurement is "the assignment of numerals to objects or events according to some rule“ (Stevens, 1946) • Stevens differentiated different levels of measurement: • Ordinal scale => the numbers represent a rank order (1st, 2nd, 3rd, etc.) • Interval scale => the interval between the numbers is equal • Ratio scale => like interval scale plus a well-defined zero point • We do not find ratio scales in social sciences ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

5. 1. On measuring • Stevens´ broad definition of measurement is accepted in all social sciences. • According to this broad definition of measurement, people use often measurements in everyday life • An example of a measurement in everyday life • In a conversation Elisabeth says: “Peter is much more self-confident than Mary”. • According to the above-mentioned broad definition of measurement as the assignment of numerals to objects or events according to some rule Elizabeth has made a measurement on the level of an ordinal scale: • Regarding the criterion of self-confidence, she has compared Peter and Mary: • Peter got rank 1st and Maria got the rank 2nd • What was Elizabeth´s rule of this measurement? Perhaps she will explain: • I observed Peter and Mary in situations which demand a self-confident behavior. In most of the cases, Peter performed better than Mary. • To generalize: • A personal trait or ability can be measured by assessing the behavior in question in a certain situation according to some rules • This measurement is possible and is actually proceeded in many domains , for example: ability in math or tennis, trust in God, etc. ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

6. 1. On measuring • Criteria for good measurement • The most important criteria for good measurement are: • Objectivity => the evaluation is independent of the assessing person • Reliability => a repeated evaluation has the same result • Validity => the evaluation reflects what is the intention of measurement ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

7. 1. On measuring • A personal trait or ability can be measured by assessing the behavior in question in a certain situation according to some rules. • This rules can be more or less strict: • Informal measuring • The rules are not well-defined, often intuitive and in some cases partially unconscious proceeded • This informal measuring is ever-presentwhen we evaluate our environment, neighbors and colleagues • Usually a great variability between low and high objectivity, reliability and validity • Low for example caused by prejudices • High for example caused by long-term observation • Semi-formal measuring • There are some explicit rules but not so well-defined as in a scientific developed test, for example exams in schools • Semi-informal measuring is the normal case in educational settings • Usually medium objectivity, reliability and validity • Formal measuring • The rules are well-defined by scientific requirements • This formal measuring isapplied by using scientific tests • Usually high objectivity, reliability and validity ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

8. 2. On the outcome of education • The outcome of education is what is learnt in a certain educational system, for example what students learn in a school • Important differentiations of outcome: • Various domains of the outcome: • for example outcomes concerning: • Mathematics • Geography • Religion • Employability • Achieved outcome: outcome as description, what is learnt • Intended outcome: outcome as norm, what should be learnt • Names used for intended outcomes: • Learning goal or instructional objective • Intended Competence • Educational standard • Short-term outcome: the outcome immediately after the lesson • Long-term outcome: the sustainable outcome, which outlasts the short-term outcome Increasing generality ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

9. 3. Purposes of measuring the outcome of education • There a several purposes of measuring the outcome of education, for example: • International comparison of the quality of schools (e.g. PISA: Programme forInternational Student Assessment) • Certification of the learner (e.g. diploma, university degree) • Feedback to the learner • Feedback for the teacher • Maintaining and improving the quality of instruction • The measurement of the outcome is essential to control and improve education ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

10. 3. Purposes of measuring the outcome of education The measurement of the outcome is essential to maintain quality control and improve education FEEDBACK INSTRUCTION MEASUREMENT OF THE ACHIEVED OUTCOME COMPARISON OF THE INTENDED OUTCOME WITH THE ACHIEVED OUTCOME • To evaluate the success of instruction, it is necessary to compare the intended outcome with the achieved outcome • The quality of this comparison depends on how precise the achieved outcome and the intended outcome can be analyzed • Therefore the assessment quality of the outcome depends on how precise the description of the intended outcome is • There is no chance to improve instruction without a sufficient precise description and measurement of the intended outcome! ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

11. 3. Purposes of measuring the outcome of education • We stated: There is no chance to improve instruction without a sufficient precise description and measurement of the intended outcome! • Here an example: • We asked:How can teachers ensure that their students have a good chance to learn during class the intended outcome of education? • To answer this question, we developed approaches (Schott & Seidl, 1994; Schott & Azizi Ghanbari, 2008, 2010, 2011) called PLANA and GovI (Goal-valid Instruction) • Here some aspects of these approaches: • Step 1: Describe the indented outcome of education (that is an instructional goal) as accurately as necessary by specifying the sets of tasks that can be solved by the learners if they have reached the instructional goal. To specify the sets of task we have developed a special method of task analysis. • Step 2: These task sets must occur in an appropriate manner during class • Step 3: These task sets must occur in an appropriate manner when measuring the outcome of education • These three steps ensure that both the lesson and the control of the learning success is content-valid to the intended outcome of education • We speak of “parallel content validity” ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

12. 3. Purposes of measuring the outcome of education • GovI a proposal on how to teach and measure the intended outcome of education: • => 3 Steps of an outcome-oriented instruction and measurement Step 1 PRECISE DEFINITION OF THE INTENTED OUTCOME Specifyingsetsoftasksdescribingtheintendedoutcomeofeducationt PARALLEL CONTENT VALIDITY The three steps ensure that both the lesson and the control of the learning success is content-valid to the intended outcome of education Step 2 Step 3 OUTCOME-ORIENTED INSTRUCTION OUTCOME-ORIENTED MEASUREMENT The task sets describing the intended outcome of education must occur in an appropriate manner during class The task sets describing the intended outcome of education must occur in an appropriate manner in the exam ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

13. 4. Problems of measuring the outcome of education • Arguments against the measuring of the outcome of education: • Objection: Measuring the outcome of education ins incompatible with human dignity • In thehumanitiesthereis a widespreadmistrustofmeasurementandempiricalresearch • Of course, we find also misuses and poor quality of measurements, but as we saw in the example of Elizabeth, we measure continuously every day • Measuring is a human characteristic • Measuring is ubiquitous not only in science and education but also in everyday life • Objection: Measuring the outcome of education is only possible in the case of basic skills such as factual knowledge but it is not possible in the case of advanced skillssuch as understanding and problem solving • Advanced skills like flying an airplane or playing Bach can be tested • Objection: Measurable are only cognitive abilities, emotions, attitudes and values are not measurable • In the case of measuring emotions, attitudes and values, we find also misuses and poor quality of measurements, but as we saw in the example of Elizabeth: • we measure continuously every day – also emotions, attitudes and values • Nevertheless it is often a hard challenge to measure advanced skills, emotions, attitudes and values ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

14. 4. Problems of measuring the outcome of education • Arguments against the measuring of the output of education: • Objection: Measuring the outcome of education prevents demanding lessons and sophisticated teaching, it leads to teaching to the test • This argument must be taken seriously. • The teacher must make sure that students are not prepared only for the test • Objection: If at all, only the short-term outcome is measurable. Long-term outcomes are not measurable • This argument must be taken seriously too. We usually cannot examine how the learners will behave after graduation • But we can do something to promote sustainability of the outcome of education ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

15. 4. Problems of measuring the outcome of education • A careful task analysis can help to promote sustainability and measurability of the outcome of education • An example: • In a research project concerning outcome-oriented instruction we developed with university students a lesson for ethics education at school. The topics of the lesson were ethical concepts like Kant's categorical imperative or concepts of utilitarianism. • When we began to specify the intended outcome of this lesson, my students insists that one of the most relevant outcomes of this lesson should be: • The learners should recognize the importance of the topics • The learners should see the sense of the lesson • The challenging question was: • How can one distinguish a learner who has achieved this outcome from a learner who did not achieve it? • Which set of tasks can measure that? ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

16. 4. Problems of measuring the outcome of education • A careful task analysis can help to promote sustainability and measurability of the outcome of education • Example continued • Very relevant outcome of the lesson: • The learners should recognize the importance of the topics; they should see the sense of the lesson. • Which set of tasks can measure this outcome? • After a long discussion we came to the following solution: • A learner who has achieved this outcome is able to solve the following 3 tasks: • Explain: If there is no controversy on moral norms, there are no ethical problems to solve. • Explain: If there is a disagreement concerning a certain moral norm, then ethical rules are necessary to solve this ethical problem • Explain: If there are various ethical rules (such as Kant's categorical imperative and utilitarianism), then philosophical methods are necessary to compare these different rules. • We decided: If a learner can solve the set of tasks consisting of these 3 tasks he or she has understood the sense of the lesson. • Specifying in this way may help to make an outcome of education, which is challenging because of its compexity, measurable and sustainable. ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

17. 5. Summary • The concept “measurement” • Measurement in a narrow sense • is used in the physical sciences and defined as the process or the result of determining the magnitude of a quantity relative to a unit of measurement • is not applicable to the outcome of education • Measurement in a broad sense • is defined as the assignment of numerals to objects or events according to some rule • is applicable to the outcome of education • Important criteria for good measurement • Objectivity • Reliability • Validity • According to the rules of assignment of numerals to objects or events we differentiate: • Informal measuring • Semi-formal measuring • Formal measuring ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

18. 5. Summary • The concept “outcome of education” • Different types of outcome of education • Various domains of the outcome • Achieved outcome: what is learnt • Intended outcome: what should be learnt • Short-term outcome • Long-term outcome ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

19. 5. Summary • Purposes of measuring the outcome of education • International comparison of the quality of schools (e.g. PISA) • Certification of the learner (e.g. diploma, university degree) • Feedback to the learner • Feedback for the teacher • Maintaining and improving the quality of instruction • Problems of measuring the outcome of education • We discussed arguments against the measuring of the output of education • Measuring the output of education • is incompatible with human dignity • is not possible in the case of • advanced skills • emotions, attitudes and values • long-term outcomes of education • leads to teaching to the test • We suggest that appropriate procedures of outcome-oriented instruction can help to teach and test the outcome of education • A careful task analysis supports measuring the outcome of education • => Measuring the outcome of education is in many cases useful and humanitarian ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

20. Literature • Schott, F. & Azizi Ghanbari, S. (2008). Kompetenzdiagnostik, Kompetenzmodelle, kompetenzorientierter Unterricht. Zur Theorie und Praxis überprüfbarer Bildungsstandards. Münster: Waxmann Verlag. • Schott, F & Azizi Ghanbari, S. (2010). Zur Theorie und Praxis kompetenzorientierten Lehrens und Lernens: Probleme und Lösungsmöglichkeiten. Report Psychologie. 11/12 2010, 480-490. • Schott, F & Azizi Ghanbari, S. (2011, in Vorbereitung). Der Beitrag von Bildungsstandards, kompetenzorientiertem Unterricht und Kompetenzdiagnostik zur Qualitätssicherung des Bildungswesens in der Schule: Eine problemorientierte Einführung. Münster: Waxmann Verlag. • Schott, F. & Seidl, P. (1994): PLANA - an ID model focussing on instructional task analysis. In: R. Tennyson, F. Schott, N. Seel & S. Dijkstra (Eds.) (1997): Instructional Design: International Perspectives. Volume I: Theory, Research, and Models. pp. 395 – 413. Lawrence Erlbaum Associates, Inc., Publishers: Hillsdale, New Jersey. • Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103 (2684): 677–80 ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education