How to assess the effectiveness of ICT use in schools and classrooms ?

1. How to assess the effectiveness of ICT use in schools and classrooms ? • Michel Pérez, Inspecteur général • de l’Education nationale (France)

2. Summary • Contributing to the conception and the testing of a EuropeanFramework for the evaluation of ICT in education (EUN projects: P2P and P2V) • Assessment of ICT use in France • how the French general inspectorate (IGEN) contributes to the promotion and the assessment of ICT use • how the French ministry of education tries to assess innovation (ICT-related or innovation in general)

3. Why inspectors need to assess the use of ICT? • Digital devices are more and more present in the teaching process. • Local authorities investing in ICT want to know what use is made of the equipment and installations. • New tools and systems appear every day on the market: VLEs (Virtual Learning Environments), interactive whiteboards, laptops, tablet computers, one-to-one computing, personal devices, etc. • We know the benefits of computers in learning : autonomy, self-assessment, personalization of learning, diversity and remote accessibility of contents, collaboration and sharing, continuity of monitoring, etc.     • BUT : • We are not sure that an appropriate use is made of the equipments. • It is currently impossible to accurately assess the gain provided by digital tools and measure their contribution to the improvement of students' skills - (cf. André Tricot, Matt Richtel Classrooms of the Future, Stagnant Scores - New York Times 2011). • Efficiency mainly depends on the learning context and the quality of the teacher is a key element.

4. Framework for the evaluation of ICT in educationA EUROPEAN SCHOOLNET program • PEER-TO-PEER (P2P) PROJECT FOR INSPECTORATES - 2006 • PEER-TO-VALORIZE (P2V) PROJECT - 2007/2008

6. History and aims of the project • The framework was developed as part of the P2P-Inspectorates project. • Its aim was to bring together standards and indicators as used by several education inspectorates throughout Europe, revise and reorder these criteria and present them as one shared model. • The framework could also be used by individual schools to self-assess their ICT use and by groups of schools to carry out peer evaluations. • P2P-Inspectorates was a cooperation between Ofsted (Office for Standards in Education, England), Skolverket (Swedish National Agency for Education, Sweden), HMIE (Her Majesty's Inspectorate of Education, Scotland), Department of Education and Science (Ireland), IGEN (Inspection Générale de l’Éducation Nationale, France) and Inspectie van het Onderwijs (The Netherlands). • P2P and P2V were co-financed by the European Commission. The inspectorates work packages of P2V and P2P were developed under the auspices of SICI (http://www.sici-inspectorates.eu/) and were coordinated by The Dutch Inspectorate of Education. 6

7. History and aims of the project • P2V project ("peer learning to valorize" which succeeded P2P) has enabled the development of a new framework in an evaluation toolkit containing instructions, forms, comments and additional forms of assessment. • The P2V framework has been used by inspectorates in several European countries to assess the use of ICT in primary and secondary schools. • The "toolbox" (P2V Work package 6 - entitled "Inspecting and Evaluating the use of ICT in schools in Europe“ was presented by Bert Jaap Van Oel (Dutch inspector and project coordinator). • Partners in the P2V project were the inspectorates of Scotland, Sweden, France, Catalonia, Lithuania and the Netherlands.

8. A quick guide to the framework • The framework is accompanied by a number of support documents that help its user in gathering relevant information: • a comprehensive user guide that helps the users of the framework (inspectors or schools) towards an accurate evaluation • a school questionnaire, which is sent to the school before the visit and provides useful information about the school’s ICT policy, hardware, etc. • a lesson observation form, which is used to make notes during the lessons visited • a guidance sheet for the meeting with teachers • a guidance sheet for the meeting with students • a guidance sheet for the meeting with school management/ICT leadership

9. The content of the framework The framework consists of three main themes: Conditions, Use and Outcomes. - There are a number of quality areas (eight in total) within each theme, - Quality indicators with corresponding evidence pointers are identified within each area. • Conditions: C1. Leadership, C2. Infrastructure and access, C3. Curriculum planning, C4. Quality assurance and improvement • Use: U1. Pupil use, U2. The teaching process, U3. Administrative use • Outcomes: O1. Impact on learning and standards When using the framework, evaluators can indicate which evidence they found by checking the appropriate boxes and can accordingly assign a score to each Quality Indicator.

13. A short review of the results • The last review meeting held in Brussels in October 2008 helped to show different lines of thought. • It should first be noted that ICT have transformed radically and irrevocably the way we teach and learn. It is therefore appropriate to examine carefully these teaching tools and the modes of learning. Reflection on the methodology of visits • The one-day (or more) visits for school assessment are conducted in several European countries as part of the evaluation of schools (which is not the norm in France). • However, in France the grid could be filled at the time of inspection of teachers and might be subject to analysis (a self-assessment grid is included in the “toolbox “). • At the end of the visit, a report could be made to the school management team extended to representatives of the school board : this report could be an opportunity for fruitful dialogue and would identifying areas for improvement.

14. A short review of the results • The analysis of the results of evaluations of the use of ICT in schools visited by country shows the diversity of situations that depend on various factors: • the size and equipment of schools, • the educational culture of the school and, concretely, whether or not the teachers work together in a dynamic way, • the “teaching authority” in the school, which is a factor of coherence, • the use of ICT by students for free access and personal work (no program involving one-to-one computing was detected).

15. Updating the framework • The ICT framework was intended to be applied during schools visits: however its administration is cumbersome and does not allow for instant reaction. • The framework could evolve into an online tool allowing : - a self-diagnostic evaluation of the use of ICT, showing the strengths and weaknesses of the school - a dialogue with external assessment in order to define tracks of progress

16. Some assessment activities for the use of ICT in France

17. IGEN’s activities for the assessment of innovative uses of ICT • Permanent monitoring within each “specialty group” of general inspectors • Monitoring mission within IGEN’s “ICT committee” (cellule TIC) • Counselling and advising DGESCO experts (DGESCO: General Directorate of School Education) • Supporting reforms and action plans,such as the “DUNE” action plan (Développement des Usages du Numérique à l’Ecole) or the Ecoles Numériques Rurales action plan (“Digital Rural Schools”) • Accompanying innovative experiences, the digital textbook, for example • Audits and reports at the request of Minister and local authorities

18. An experimental online tool for self-assessment of schools in France

19. Online school self-assessment OAPE Overview • This site is designed as a tool to help schools in the monitoring of their educational activity. • It includes a set of questions that focus on current practices involving several actors. • It is used by the head teacher, who is free, if he wishes to do so, to share the access to OAPE with other members of the school community. • It consists of four types of tabs: • 5 tabs for the “diagnosis” (corresponding to the 5 “pillars” of the school’s characteristics and specificities); • - 1 tab to summarize the various results of the diagnosis; • - 1 tab for the “radar chart”, a diagram which provides an immediate and global visualisation of the diagnosis; • - 1 tab for resources. • With this tool, the head teacher can identify the “pillars” of the school’s specificities that are heavily invested and conversely the “pillars” that are less so.

22. Experimental audit protocol for the assessment of a (regional) VLE • With the participation of IGEN, an audit protocol was recently devised in the region of Nantes (Académie), with a view to: • verifying how the regional VLE is developing in the schools • assessing the effective use of the VLE in schools • this protocol will be applied in order to : • help the schools in their self-assessment approaches • enter into a process of continuous improvement • for local authorities • to make an assessment and provide a tool for monitoring progress, • to manage the evolution of uses by bringing to light the points assessed.

23. The stages of the protocol • Collection of information • Centralised data collected on the VLE • Two questionnaires: a questionnaire for head teachers, a questionnaire for teachers. No questionnaire for students. • Visit that comprises verification, interview(s), instant / on-the-spot debriefing • Written audit report • Summary of recommendations

24. A BRIEF CONCLUSION • Effective use of ICT in teaching must obviously be checked ... • to support teachers’ innovative activity, • to help them make a fair use of digital devices in their teaching activity, • to promote substantial progress in the learning process. • Inspectors need to have some efficient tools to aid in the evaluation of the use of ICT in schools. • We need to provide a shared tool, fruit of a collaborative work. • It should be possible and relevant to develop this framework into an online tool for self-diagnosis and remote dialogue preparing the evaluators’ visit.