Theme 10 Evaluation

1. Theme10Evaluation

2. In thistheme we discuss in detail the topic “evaluation”. This is a comprehensiveand a complex theme. Therefore, duringthissession, we discussonly a first part of te overall theme.

3. Evaluation: the concept Defining the concept evaluation is a difficultissue since the concept itselfonlyemphasizesone aspect of whatevaluationfullyembraces; namely the “giving a value” towhat is beingobserved . As we willsee, italsodoes not help toreplace the concept byotherpopularconcepts, such as “assessment”. Again, onlyoneparticular aspect of the wholeprocess is beingemphasized.

4. Evaluation: the concept Read the followingdescription of evaluation: “Evaluation is the entireprocess of collecting, analysingandinterpretinginformationaboutpotentiallyevery aspect of aninstructionalactivity, with the aim of givingconclusionsabout the efficacy, efficiency and or anyother impact” (Thorpe, 1988). Youcanobservethatevaluation is a comprehensiveprocessthatcanberelatedtopotentiallyevery element in oureducational frame of reference.

5. Evaluation: the concept In the literature, an important distinction is made betweenevaluationand assessment. • Assessment or “measuring” refersto the process of collectingandanalysing information (Burke, 1999 en Feden & Vogel, 2004) • Evaluationrefersto, as statedearlier, adding a valuetowhat has been collectedandanalyzed in view of comingto a conclusionabout the efficacy, efficiency or anyother impact.

6. Evaluation: the concept But in the literature, an even more detailed distinction is made between: • Measuring/testing: collecting information • Evaluating/valuing: what is this information worth? • Scoring/grading: depending on the « worth », what score will we give It is essentialtodistinguish these three approaches. Onecanmeasure without valuingor scoring. Andonecannotscore without collectingandvaluinginformation.

7. Evaluation: qualityrequirements Prior to a discussion of recent developments in the field of evaluation, we first deal withsomecriticalqualityrequirementsthat are central in discussionsaboutevaluation: • Validity • Reliability • Authenticity • Recency

8. Validity Validityrefersto the extentthat the content of what is beingmeasured, valuedandscored is relatedto the initialevaluationobjective. Typicalquestionsthat are raised in this context are: • Whatif we onlymeasuregeometry, when we want tocometoconclusionsaboutmathematics performance in primary school? • Whatif we only get questionsfromchapter5 duringanexam? • Whatif we onlyaskmemorizationquestions in a test when we alsoworked in the laboratoryandsolvedchemistryproblems?

9. Reliability Reliabilityrefersto the extentourmeasurement is stable. Typicalquestionsraised are: • If I repeat the same test tomorrow, willI get the sameresults (stability)? • Is there a large difference in the abilitytosolve the different questionsabout the same topic (internalconsistency)? • Ifsomeoneelsemeasured, valuedandscored the test, would he/she end up with the sameresults?

10. Authenticity Authenticityrefersto the extent the information we gather, mirrors in a relevant, adequate, andauthentic way reality. Examples of relatedquestions: • Is itsufficienttoask student nurses togiveinjections on a dolltoevaluatetheirinjection skills? • Is it adequate togive a flyinglicensetosomeonewho was onlytested in a flight simulator? • Is itsufficientto say thatone is ableto “teach” afterevaluating his/her capacitieswith small group teaching?

11. Recency Recencyquestions the “date” information has been collected, valued or scored in view of evaluation: • Can we accept creditsobtained 5 yearsagofromsomeonewhoasksbeingreleaved of courses in a new study program? • Can we hire a young house motherwhogot her degree 10 yearsago? • Are the Basic Life Support Skills masteredsixmonthsago, still relevant today in anactive first aidofficer?

12. Recent developments in evaluation Recent developments in evaluationcanbeclusteredalong five dimensions: • At whataggregation level is the evaluationbeing set up? • What are the functions/roles of the evaluation? • Whocarries out the evaluation? • When is the evaluationbeing set up? • Whatevaluationtechniques are beingadopted? We discusssomeexamples in relationtoeachdimension.

13. Dimension 1: aggregation levels Firstly, we observethatevolutionsin evaluation are relatedto the aggregation levels in oureducational frame of reference: • Micro level • Meso level • Macro level We look – in relationtoeachaggregation level – toparticular new developments.

14. Dimension 1: aggregation levels At eachaggregation level, the sameelements re-appear. Evaluationcanberelatedtoevery element in the educationalframe of reference • Responsiblefor the instruction • Learner • Learning activities • Organisation • Context • Instructionalactivities(objectives, learning content, media, didacticalstrategies, evaluation)

15. Micro level Example 1: evaluation of the extent the learningobjectives have been attained; Example 2: evaluation of didacticalstrategies.

16. Micro level: evaluationlearningobjectives Duringevaluation we measure the behavior, we valuethe behaviorandgive a score. The question is “What is the base of giving a certainvalue?”. • Based on a criterion? • Criterionreferenced assessment • Based on a norm, e.g., groupmean? • Norm referenced assessment • Based on earlier performance of learner? • Ipsative assessment or self-referenced assessment

17. Micro level: evaluationlearningobjectives Example: athletics, 15-year olds have to run 100 meter? • Criterionreferenced assessment • Every performance is comparedtoan a priori statedcriterion; e.g., lessthan 15 seconds • Norm referenced assessment • Every performance is comparedto the classroom mean (imagineyour are in a class withfast runners). • Ipsative assessment of self-referenced assessment • Every performance is comparedtothe earlier performance of the individuallearner; emphasis on progress.

18. Micro level: evaluationinstructionalstrategies Hattie (2009) discusses in his meta-analysis instructionalactivities. These analyses look whether different instructionalstrategies have a differential impact on learners. Do they matter? In the followingexampleyouseethat the didacticalstrategy “homework” representanaverage “effect size” d = .29. This is far below the benchmark d = .40.

19. Meso level: evaluation at school level • Recent developments at the school level look whether “schools” have a value-added; this means anadditionalvaluethatresults in betterlearning performance. • But can we simplycompare schools withoneanother? Does thisnot lead tosimple ranking as depicted in thisjournal

20. Meso level: evaluation at school level • Onecannotsimplycompare schools. • Calder (1994) puts forward in this context, the CIPP model toconsidereverything in balance: • Context evaluation: the geographicalposition of a school, the available budget, the legal base, etc. • Input evaluation: what the school actuallyuses as resources, its program, itspolicies, the numberand type of staff members, etc. • Processevaluation: the way a program is implemented, the strategiesbeingused, the evaluation approach, the professional development of the staff, etc. • Product evaluation: the effects, such as goal attainment, throughput, return on investment, etc.

21. Meso level: evaluation at school level • Comparing schools with the CIPP model can as suchimplythat: • A school with a lot of migrantsoutperforms a school withdominantlyupper class children. • A school canbegood in attainingcertain goals, but canbelessqualified in attainingother goals. • A school canbecriticized as toitspolicies. • Thatonewillconsider the geographicallocation of a school whendiscussingresults (e.g., anunsafeneighbourhood). • Thatwe willalsolook at what the learnersdo later whenthey go toanother school (e.g., success at university). • Schools are beingassessedby the inspection on the base of the CIPP model.

22. Meso level: evaluation at school level • The inspectionreports are public.

23. Macro level: school effectiveness Read the following description: • “The aim of school effectiveness research is todescribeandexplain the differencesbetween schools on the base of specific criteria. This research explores the differences in performance on the base of differences in thoseresponsiblefor teaching, the learners, the classes, the school.” Youcanseethat – as in the CIPP model – explanations are sought at the level of all schools in the educational system.

24. Macro level: school effectiveness This development started from very critical reports as to the value-added of schools: • Coleman report (1966, chapter 1): “Schools have little effect on students’ achievement that is independent of their family background and social context.” • Plowden report (1967, p.35): “Differences between parents will explain more of the variation in children than differences between schools. (…) Parental factors, in fact, accounted for 58% of the variance in student achievement in this study.” • Schools want – in contrast to these reports – proof they make a difference and contribute to learner performance

25. Macro level: school effectiveness A central critique on the Coleman and Plowden report is that they neglect the complex interplay that helps to explain differences; see the CIPP model. Instead of simply administering tests and comparing results, we have to look – next to “product effects” – atthe processes and variables that are linked to these results. This is labelled with the concept performance indicators.

26. Macro level: performance indicators • Performance indicators are: "statistical data, numbers, costsor anyotherinformationthatmeasuresandclarifiesthe outcomes of aninstitution in line withpreset goals.“ • You can notice that the emphasis in performance indicators is on the description and explanation of differences in performance. • One of the best known performance indicator studies is the three-yearly PISA study: Programme for International Student Assessment.E.g., in the PISA 2006 study, performance was compared of schools in 54 countries.

27. Macroniveau: performance indicators • Results of PISA in 2006 show – forexample – the high performance of Flemish schools forsciences, mathematics, and reading literacy.

28. PISA results are notonlydescribed.They are alsoexplained. In thisgraphic, oneseeshow the PISA results are associatedwith the socio-economic status (SES) of the learners. The higherthe status, the higher the results. SES is determinedby the educational level of the parents, theirincome, theirpossession of culturalgoods (e.g., books), etc.

29. Dimension2: Functions of evaluation Why do we evaluate? Theremightbe different reasons: • Formativeevaluation • Toseewhereoneis in the learningprocessandhow we canredirectthe learningprocess • Summativeevaluation • Todetermine the finalattainment of the goals. • Predictionfunction • Topredictfuture performance (e.g., success in highereducation) • Selectionfunction • Toseewhetherone is fit for a job or task.

30. Dimension 2: Functions of evaluation Abroad, there is a lot of attention for the selectionfunction; see the emphasis on entranceexams. In thisexample, onesees a luckycandidate (and his mother) whosucceeded in the entranceexamfor a Chinese university.

31. Dimension 2: Functions of evaluation Earlier, there was a major emphasis on summativeevaluation. Nowadaysthisemphasis has shiftedtowardsformativeevaluation. Why? • Does onelearnfromevaluative feedback; this is alsocalledconsequentialvalidity genoemd? • From the evaluationresults, does thisnotimplythat the teacher has toredirect the instruction, the support, the learningmaterials, etc? • Does a learneralreadyreach a preliminaryattainment level?

32. Dimension 3: Who is responsible? Traditionally, the teachers is responsiblefor the evaluation. But there are new developments: • The learnerhim/herselfcarries out the evaluation : self assessment • The learnerandpeerscarry out the evaluationtogether: peer assessment • An externalresponsiblecarries out the evaluation (e.g., other teacher). • An external company carries out the evaluation: assessment centers • …

33. Dimension 3: Who is responsible? • New development: self assessment • Self-assessment is seen as a type of evaluationthataims at fostering the learningprocess (Assessment-as-learning) : formativeevaluationfunction • Twomain steps tobe taken: • Initialtrainingtodevelop criteria and instrument/tool, anddiscuss the value of what is beingmeasured. • Next, usage of the tools/instrumentsanddeveloping a personal opinion. Scoring is notan issue here. • Veryusefultechnique: rubrics(seefurther on)

34. Dimension 3: Who is responsible? • Assessment centres: external company thatcarries out evaluation; mostlywithselectionfunction • “Standardizedprocedure toassess complex behavior on the base of multiple information bases. The behavior is assessed in simulatedcontexts. Multiple persons carry out the evaluationandcometo a shared vision.” • Differentevaluators are involvedandguarantee a 360° approach of the evaluation • Thistechniquefulfills a selectionfunctione.g., when screening candidatesfor a job

35. Dimension 4: When to evaluate? There is a shift in the moment the evaluation is being set up: towards « prior to » and « during » the learning process; serving formative evaluation function: • Prior • Prior knowledge testing • During • Progress testing • Portfolio evaluation • After • Final evaluation

36. Dimension 5: What technique? Next to traditional evaluation tests with multiple choicequestions, open answerquestions, fill-in questions, sortquestions, … we observe a series of new techniques. Examples: • Rubrics: attention is paidto criteria and indicators • Portfolio’s: file with letters, information, illustrations, products, … as the information base for the evaluation • …

37. Dimension 5: Technique rubrics Rubrics: • Defineclearcriteria: concrete element of a complex learningobjectivethat is beingmeasured, valuedandscored • Determineforeachcriterion a number of qualityindicators: indicators exemplify the level at which a certaincriterion is being met, answered, attained

38. Dimension 5: Technique Rubrics Performance indicators Examplerubric: “mixingcolours” In next steps of the learningprocess, we canadd criteria and/or performance indicators to the rubric Criteria

39. Dimension 5: Technique Rubrics Examplerubric: “Writing of a historical fiction story”

40. Dimension 5: Technique Portfolio Read thisdescription of a portfolio: A portfolio is a file with letters, information, illustrations, products, … that is used as an information base for the evaluation.

41. Dimension 5: Technique Portfolio • Types of portfolios: • A document portfolio or product portfolio: documentationthathelpstodescribe the activities in the training, intership, practical experience, … (measurement). In additiontothis info, learnerscanaddtheirreflections (valueing).Typicallyusedwith student doctors, nurses, teachers, … • A process portfolio: a logbook. Documentation of the progress in the learningprocess, enrichedwithreflections.Typicallyusedwith student doctors, nurses, midwives, teachers, … . • A showcase portfolio: “the best of …”. Bundle of the best work of a student thathelpstocometo a conclusionabout his/her performance.Typicaluse in decorative arts, music, theater, architects, … .

42. Dimension 5: Technique Portfolio Example of aprocess portfolio for student teachers

43. Einde van dit instructiepakketPak nu de eindtoets aan. Ga opnieuw naar je Minerva werkplek