Consider the Evidence

1. Consider the Evidence Evidence-driven decision making for secondary schools A resource to assist schools to review their use of data and other evidence 5 Getting Started

2. Evidence-driven decision making Today we will think about • the data and other evidence we can use to improve teaching, learning and student achievement • how to get started on a decision-making process by asking purposeful questions

3. A teacher has a hunch - poor writers might spend little time on homework Explore data Survey of students shows that this is only partially true Reflect How will we teach writing in the future? Trigger Significant numbers not achieving well in writing Question What are the characteristics of students who are poor at writing? Evaluate Has writing improved? Analyse NQF/NCEA results by standard Intervene Create multiple opportunities for writing; include topics that can use sport as context; connect speaking and writing. PD for staff. Assemble more data & other evidence: asTTle reading, homework, extracurric, Attendance, etc Interpret information Poor writers likely to play sport, speak well, read less, do little HW Analyse non NQF/NCEA data and evidence The evidence-driven decision making cycle .

4. Evidence-driven decision making Getting started Trigger Clues found in data, hunches Explore Is there really an issue? Question What do you want to know? Assemble Get all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

5. Evidence-driven eating You need to buy lunch. Before you decide what to buy you consider a number of factors: • how much money do you have? • what do you feel like eating? • what will you be having for dinner? • how far do you need to go to buy food? • how much time do you have? • where are you going to eat it?

6. Evidence-driven teaching I had a hunch that Ana wasn’t doing as well as she could in her research assignments, a major part of the history course. What made me think this? Ana’s general work (especially her writing) was fine. She made perceptive comments in class, contributed well in groups and had good results overall last year, especially in English. How did I decide what to do about it?

7. Evidence-driven teaching (cont…) I looked more closely at her other work. I watched her working in the library one day to see if it was her reading, her use of resources, her note taking, her planning, or what. At morning tea I asked one of Ana’s other teachers about Ana’s approach to similar tasks. I asked Ana if she knew why her research results weren’t as good as her other results, and what her plans were for the next assignment. I thought about all of this and planned a course of action. I gave her help with using indexes, searching, note taking and planning and linking the various stages of her research.

8. Consider the Evidence A resource to assist schools to review their use of data and other evidence What is meant by ‘data and other evidence’?

9. Evidence Any facts, circumstances or perceptions that can be used as an input for an analysis or decision • how classes are compiled, how classes are allocated to teachers, test results, teachers’ observations, attendance data, portfolios of work, student opinions… Data are one form of evidence

10. Data Known facts or measurements, probably expressed in some systematic or symbolic way (e.g. as numbers) • assessment results, gender, attendance, ethnicity … Data are one form of evidence

11. Which factors are data? Evidence to consider before buying lunch • how much money you have • what you feel like eating • what you’ll be having for dinner • how far you need to go to buy food • how much time you have • where you’re going to eat • what your diet allows

12. Evidence-driven decision making We have more evidence about what students know and can do than ever before – their achievements, behaviours, environmental factors that influence learning We should • draw on all our knowledge about the learning environment to improve student achievement • explore what lies behind patterns of achievement • decide what changes will make a difference

13. What evidence does a school have? • Demographics • Student achievement • Perceptions • School processes • Other practice

14. Demographics What data do we have now to provide a profile of our school? What other data could we create? • School • Students • Staff • Parents/caregivers and community

15. Demographics Data that provides a profile of our school • School- decile, roll size, urban/rural, single sex or co-educational, teaching spaces … • Students - ethnicity, gender, age, year level, attendance, lateness, suspension and other disciplinary data, previous school, part-time employment … • Staff - gender, age, years of experience, qualifications, teaching areas, involvement in national curriculum and assessment, turnover rate … • Parents/caregivers and community - socio-economic factors, breadth of school catchment, occupations …

16. Student achievement What evidence do we have now about student achievement? What other evidence could we collect? • National assessment results • Standardised assessment results administered internally • Other in-school assessments • Student work

17. Student achievement Evidence about student achievement • National assessment results - NCEA, NZ Scholarship - details like credits above and below year levels, breadth of subjects entered… • Standardised assessment results administered internally - PAT, asTTle … • Other in-school assessments - most non-standardised but some, especially within departments, will be consistent across classes - includes data from previous schools, primary/intermediate • Student work - work completion rates, internal assessment completion patterns, exercise books, notes, drafts of material - these can provide useful supplementary evidence

18. Perceptions What evidence do we have now about what students, staff and others think about the school? Are there other potential sources? • Self appraisal • Formal and informal observations made by teachers • Structured interactions • Externally generated reports • Student voice • Other informal sources

19. Perceptions Evidence about what students, staff, parents and the community think about the school • Self appraisal - student perceptions of their own abilities, potential, achievements, attitudes … • Formal and informal observationsmade by teachers - peer interactions, behaviour, attitudes, engagement, student-teacher relationships, learning styles, classroom dynamics … • Structured interactions - records from student interviews, parent interviews, staff conferences on students … • Externally generated reports - from ERO and NZQA (these contain data but also perceptions) … • Student voice - student surveys, student council submissions… • Other informal sources – views about the school environment, staff and student morale, Board perceptions, conversations among teachers …

20. School processes What evidence do we have about how our school is organised and operates? • Timetable • Classes • Resources • Finance • Staffing

21. School processes Evidence about how our school is organised and operates • School processes - evidence and data about how your school is organised and operates, including: • Timetable –structure, period length, placement of breaks, subjects offered, student choices, tertiary and workforce factors, etc • Classes - how they are compiled, their characteristics, effect of timetable choices, etc • Resources - access to libraries, text books, ICT, special equipment, etc • Finance - how the school budget is allocated, how funds are used within departments, expenditure on professional development • Staffing - policies and procedures for employing staff, allocating responsibility, special roles, workload, subjects and classes

22. Other practice How can we find out about what has worked (or not) in other schools?

23. Other practice How we can find out about what has worked in other schools? • Documented research – university and other publications, Ministry of Education’s Best Evidence Syntheses, NZCER, NZARE, overseas equivalents … • Experiences of other schools – informal contacts, local clusters, advisory services, TKI LeadSpace …

24. What can we do with evidence? Shane’s story A history HOD wants to see whether history students are performing to their potential. She prints the latest internally assessed NCEA records for history students across all of their subjects. As a group, history students seem to be doing as well in history as they are in other subjects. Then she notices that Shane is doing very well in English and only reasonably well in history. She wonders why, especially as both are language-rich subjects with many similarities. The HOD speaks with the history teacher, who says Shane is attentive, catches on quickly and usually does all work required. He mentions that Shane is regularly late for class, especially on Monday and Thursday. So he often misses important information or takes time to settle in. He has heard there are ‘problems at home’ so has overlooked it, especially as the student is doing reasonably well in history.

25. Shane’s story (cont...) The HOD looks at the timetable and discovers that history is period 1 on Monday and Thursday. She speaks to Shane’s form teacher who says that she suspects Shane is actually late to school virtually every day. They look at centralised records. Shane has excellent attendance but frequent lateness to period 1 classes. The HOD speaks to the dean who explains that Shane has to take his younger sister to school each morning. He had raised the issue with Shane but he said this was helping the household get over a difficult period and claimed he could handle it. The staff involved agree that Shane’s regular lateness is having a demonstrable impact on his achievement, probably beyond history but not so obviously. The dean undertakes to speak to the student, history teacher, and possibly the parents to find a remedy for the situation.

26. Thinking about Shane’s story What were the key factors in the scenario about Shane? What types of data and other evidence were used? What questions did the HOD ask? What happened in this case that wouldn’t necessarily happen in some schools?

27. Shane’s story - keys to success The history HOD looked at achievement data in English and history. She looked for something significant across the two data sets, not just low achievement. Then she asked a simple question: Why is there such a disparity between in these two subjects for that student? She sought information and comments (perceptions evidence and data) from all relevant staff. The school had centralised attendance and punctuality records (demographic data) that form teacher could access easily. The action was based on all available evidence and designed to achieve a clear aim.

28. Speculate A teacher has a hunch about a problem or a possible action Trigger Data indicate a possible issue that could impact on student achievement Explore Check data and evidence to explore the issue Reflect on what has been learned, how practice will change Question Clarify the issue and ask a question Evaluate the impact on the intervention Assemble Decide what data and evidence might be useful Act Carry out the intervention Intervene Plan an action aimed at improving student achievement Interpret Insights that answer your question Analyse data and evidence The evidence-driven decision making cycle

29. SPECULATE TRIGGER EXPLORE REFLECT QUESTION EVALUATE ASSEMBLE ACT INTERVENE INTERPRET ANALYSE The evidence-driven decision making cycle

30. Evaluate and reflect • Summative evaluation – assess how successful the intervention was; decide how our practice will change; report to board • Formative evaluation– at every stage in the cycle we reflect and evaluate Are we are on the right track? Do we need to fine-tune? Do we actually need to complete this?

31. The evidence-driven decision making cycle > Trigger Clues found in data, hunches Explore Is there really an issue? Question What do you want to know? Assemble Get all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

32. Asking questions Evidence-driven decision making starts with asking good questions You can tell whether a man is clever by his answers. You can tell whether he is wise by his questions. Nobel Prize winner, Naguib Mahfouz

33. Trigger questions • How good/poor is …? • What aspects of … are good/poor? • Is … actually changing? • How is … changing? • Is … better than last year? • How can … be improved? • Why is … good/poor? • What targets are reasonable for …? • What factors influence the situation for …? • What would happen if we …? Formative or summative?

34. Summative questions A target in the school’s annual plan is for all year 10 boys to improve their writing level by at least one level using asTTle (e.g. from 4B to 4A). Have all year 10 boys improved by at least one asTTle level in writing?

35. Questions about policy We have been running 60-minute periods for 5 years now. What effect has the change had?

36. Formative questions from data The data suggest our students are achieving well in A, but less well in B. What can we do about that?

37. Formative questions from data A significant proportion of our school leavers enrol in vocational programmes at polytechnic or on-job. How well do our school programmes prepare those students?

38. Questions from hunches • I suspect this poor performance is being caused by … Is this true? • We reckon results will improve if we put more effort into ... Is this likely? • I think we’d get better results from this module if we added … Is there any evidence to support this idea?

39. Hunches from raw data

40. Hunches from raw data • Is the class as a whole doing better in internally assessed standards than in externally assessed standards? If so, why? • Are the better students (with many Excellence results) not doing as well in external assessments as in internal? If so, why? • Is there any relationship between absences and achievement levels? It seems not, but it’s worth analysing the data to be sure.

41. The evidence-driven decision making cycle Trigger Clues found in data, hunches > Explore Is there really an issue? Question What do you want to know? AssembleGet all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

42. Question – Explore – Question It looks like our students are doing well in A but not in B. What can we do about it? EXPLORE … what else should we be asking? Is this actually the case? Is there anything in the data to suggest what could we do about it?

43. Question – Explore – Question We have been running 60-minute periods for a year now. Did the change achieve the desired effects? EXPLORE … what else should we be asking? How has the change impacted on student achievement? Has the change has had other effects? Is there more truancy? Is more time being spent in class on assignments, rather than as homework?

44. The evidence-driven decision making cycle Trigger Clues found in data, hunches Explore Is there really an issue? > Question What do you want to know? Assemble Get all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

45. A very good question • Specific and with a clear purpose • Able to be investigated through looking at data and other evidence • Likely to lead to information on which we can act

46. Questions with purpose What do we know about reported bullying incidents for year 10 students? MAY BE BETTER AS Who has been bullying whom? Where? What are students telling us? What does pastoral care data tell us? Were some interventions more effective with some groups of students than others?

47. Write more purposeful questions • What are the attendance rates for year 11 students? • What has been the effect of the new 6-day x 50-min period structure? • How well are boys performing in formal writing in year 9? • What has been the effect of shifting the lunch break to after period 4?

48. More purposeful questions • How do year 11 attendance rates compare with other year levels? Do any identifiable groups of year 11 students attend less regularly than average? • Is thenew 6-day x 50-min period structure having any positive effect on student engagement levels? Is it influencing attendance patterns? What do students say? • Should we be concerned about boys’ writing? If so, what action should we be taking to improve the writing of boys in terms of the literacy requirements for NCEA Level 1? • The new timing of the lunch break was intended to improve student engagement levels after lunch. Did it achieve this? If so, did improvements in student engagement improve student achievement? Do the benefits outweigh any disadvantages?

49. The evidence-driven decision making cycle Trigger Clues found in data, hunches Explore Is there really an issue? Question What do you want to know? Assemble Get all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

50. SPECULATE TRIGGER EXPLORE REFLECT QUESTION EVALUATE ASSEMBLE ACT INTERVENE INTERPRET ANALYSE The evidence-driven decision making cycle