Context of Evaluation

1. 1

2. Context of Evaluation Technology can be applied in the classroom (and the community) in countless ways, and with numerous benefits. • The flexibility of technology presents both opportunities and challenges. • Evaluation must remain focused on measuring and monitoring the use of technology to fulfill the program’s goals. 2

3. Program Goals, Vision, Mission Provide Opportunity for underprivileged children in India to enhance their learning through the use of Digital Technology • Increasing Economic Potential • Promoting Global Camaraderie Future AIF/DE plans: • Expansion of program to additional schools. • Increase utilization of Internet Learning Centers (ILCs) through community involvement. • Focus on government schools & the most needy areas. 3

4. Current Measurements Quantifiable Metrics: • 55 centers • 1000 teachers • 25,000 high school students trained in basic computer and internet skills. • Almost 10 schools have relationships with the local community to support sustainability and help cover costs. • 39 Schools with Websites • 100 teacher projects • 70 student projects 4

5. DE Benefits to Students AIF-Reported Benefits of the Digital Equalizer program include: • Considerable improvement in academic performance of slow learners. • The Internet motivates students to learn English faster. • Increased confidence & skills to be successful in life. Our visits to 9 schools, including interviews and interactions with students, teachers, and school management, have led us to believe these claims are credible. In addition, an overwhelming amount of 3rd party research supports the value of Technology Aided Learning (TAL). 5

6. Independent Research on TAL Sivin-Kachala, J. (1998). Report on the effectiveness of technology in schools, 1990-1997. • Includes a review of 219 research studies from 1990 to 1997. • Students in technology rich environments showed increased achievement: • In all major subject areas • Among all different age groups • Among both regular and special needs children • Students attitudes toward learning and their own self concept improved consistently when computers were used for instruction 6

7. Additional Research on TAL Kulik, J.A. (1994). Meta-Analytic studies of findings on computer based instruction. In E.L. Baker, and H.F. O’Neil, Jr. (Eds.). Technology assessment in education and training. Hillsdale, NJ: Lawrence Erlbaum. 7

8. Additional TAL Benefits McKinsey Report: Connecting K-12 schools to the Information Superhighwayhttp://www.uark.edu/mckinsey • “In a review of 254 controlled studies, researchers found that CAI helped students to learn 30 percent faster than students receiving traditional instruction.” • “The use of computers seems to have even greater effects with low achieving and remedial students.” • It seems clear that the impact on student motivation levels is significant. The Christopher Columbus School, which extensively uses networked computers, has its district's best attendance record for both students and faculty. In the Carrollton District, the dropout rate declined from 19% to 5% after deployment of technology. 8

9. Summary of Benefits 9

10. Challenges Although the success of Technology Aided Learning is well documented, and the value of the Digital Equalizer Program is clear to anyone with the opportunity to visit AIF schools, the need for ongoing assessment and evaluation remains. More refined assessment and measurement will: • Increase effectiveness. • Provide assurance to donors, partners, and participants that the program is meeting its goals. Evaluation tools should also monitor progress and performance in areas where obstacles are expected. • Motivation of Staff, Teachers in Government Schools. • Parental Support. • Utilization of ILCs during after school hours. • Sustainability: • Skills development, continued training. • Financial costs of ILC operation and technology renewal. 10

11. Year Two Year Three Current DE program Year One • Detailed training plan • Standardized teacher training curriculum • Standardized review and assessment • Some flexibility incorporated into goal setting and assessment 11

12. Learnings from School visits • Schools are unique • Students, Teachers, Facilities, Finances. • Each school has different needs and challenges: • More computers, training, support. • Financial sustainability. • Teacher or Management motivation The DE Program must be customized to fit school needs 12

13. Digital Equalizer Program Identifying Schools Categorize School Establish Objectives Evaluate Explore Sustainability 13

14. Categorize School Establish Objectives Evaluate Explore Sustainability Identifying School Identifying Schools Identifying Schools 14

15. DE – Identifying Schools The selection of schools that are best positioned to benefit from TAL may be the most critical factor determining the success of the Digital Equalizer Program. Our research and observations suggest the following key considerations: • School’s commitment to training, participation, and integration of technology to improve the school. • Enthusiasm of principal and teachers. • Availability of DE coordinator (location). • Issues surrounding vernacular and English medium schools. We recommend tracking selection process success, failures, and evaluating whether criteria being used are appropriate predictors of DE program success. 15

16. Categorize School Identifying Schools Categorize School Establish Objectives Evaluate Explore Sustainability 16

17. DE – Categorize Schools To efficiently customize the DE program for each school, it is important to understand each school’s needs. By categorizing a school: • Comparisons between schools in similar situations become more meaningful. • Different objectives, measurements, and expectations are appropriate for schools in different categories. • DE coordinators can more easily manage the schools and prioritize DE resources and attention. 17

18. School Categories • Financial Position of School • Type of institution (Public, Private, Government) • Economic class of students’ families (ability to pay computer fees) • English or Vernacular Medium • Vernacular schools face different challenges, but also possess enormous opportunities to use the Internet and technology to teach English. • School Size and Infrastructure • A lab 10 unit computer lab cannot serve 3000 students (plus teachers) in the same way it serves a 300 student school. • Schools which already have some computer technology, or acquire additional computers, will have different capabilities. • Post Graduation Career Expectations • The Internet and computers can be best used in different ways, depending upon whether students plan to continue their education, enter the workforce, etc. • Boy, Girl, Coed, and Special Needs Education 18

19. Establish Objectives Identifying Schools Categorize School Establish Objectives Evaluate Explore Sustainability 19

20. Establish Objectives One group of consistent objectives and measurements across all schools: • Establishes minimum standards. • Can be used to assess and communicate the impact of the DE program. • Ties each school to the AIF/DE Mission. A second group of objectives and measurements that are unique to the school is also critical: • Allows school to participate in setting expectations and determining how to best adapt technology to its specific needs. • Since schools will be evaluated on the basis of meeting both sets of objectives, this process allows comparison of schools with similar resources, abilities, and obstacles. 20

21. School and Program Objectives As part of the AIF-DE Training Guide, the Implementation Plan (Phase 1, Units 1 and 2) provides Coordinators the opportunity to set both sets of Objectives. • Coordinators should agree with School Management on specific objectives and goals which are measurable and in line with the core objectives of the DE program. • As part of the objectives agreement process, schools should also be informed of (or cooperatively determine) the measurements and evaluation methods that will be used to track progress toward these objectives. • Schools should be encouraged to set challenging goals, and should not fear punishment for failing to reach a goal that is set very high relative to the goals set by similar schools. 21

22. Establishing Common Objectives Common Objectives should tie to the AIF/DE mission: • Access to Digital Technology • Enhanced Learning • Expanded Economic Opportunities • Global Camaraderie Sample School Objectives might be: • Provide each student X hours of technology instruction • Provide X hours per week of unstructured ILC availability. • Boost student subject area test scores X percent • Increase number of students attending +2 by X percent • Provide every student in class 10 the opportunity to correspond by email with a student in the United States. 22

23. Establishment of Unique Category/Objectives Unique Objectives should tie to the specific school: • Is attendance or enrollment a problem? • Is financial sustainability a reasonable goal for the school to accomplish on its own? • Is the school an English medium or vernacular school? Sample Objectives might be: • Increase attendance to X percent. • Creation of a student fee program which covers certain digital technology costs. • Increase English proficiency test scores by X percent. 23

24. Evaluate Identifying Schools Categorize School Establish Objectives Evaluate Explore Sustainability 24

25. Evaluation • Evaluation of individual schools must tie to both sets of agreed upon objectives. • The AIF-DE Training Guide currently includes assessment templates for teachers, students, schools, and the DE program. • Comprehensive checklists • Primary weakness is that the grading scale is subject to interpretation, making assessments completed by different individuals impossible to compare. Also may lack credibility to outside constituents such as donors. • However, the AIF-DE Training Guide also includes opportunities and guidelines that will allow collection of important information that can be used for quantifiable, unbiased evaluation (See Attachment 1). • Additional data collection will aid evaluation process (See Attachment 2) • The evaluation should be a two way process where all constituents get their roles and responsibilities reviewed. A tool for schools to evaluate DE coordinators should be developed with input from facilitators. 25

26. Evaluation Questions • Question Set A • Check points and success factors that are common in all schools • Basic School Info • Teacher Training • Curriculum and Academics • Facility Utilization and Reliability • School Management commitment and support • Financial Tracking Question Set B Check points tailored for each school 26

27. YEAR 1 YEAR 2 YEAR 3 Evaluation – Ongoing and Iterative Pre-program Evaluation YEAR 1 Evaluation YEAR 2 Evaluation YEAR 3 Evaluation • Initial evaluation provides baseline measurements that are needed for later comparison and that are critical to showing results of the DE program. • First, Second, and Third year assessments match each years activities and objectives, so there is some variation in the evaluation. • Plans for methods of continued evaluation and assessment of schools after the three year period should begin immediately. 27

28. Explore Sustainability Identifying Schools Categorize School Establish Objectives Evaluate Explore Sustainability 28

29. Sustenance Requirements: Money and Skills For a school to sustain an ILC, it needs both: • Staff with the technical skills needed to run a lab • Financial resources to pay costs associated with normal use and maintenance. All DE schools should possess staff and teachers with the skills needed to run and sustain an Internet Leaning Center by end of 3 years. We are uncertain whether all government schools will be capable of finding the money to pay for connectivity, hardware and software upgrades, maintenance, and other fees that will be associated with sustaining the ILC over the long-term. Although partnering with corporations and other community groups for sponsorships (adoption) of schools is one of the best options, the risk that such outside support will be focused on primarily the most successful model schools exists. 29

30. Summary of Costs 30

31. Recommendations: Overall Program All of our recommendations should be viewed in the context of how deeply impressed we are with the success of the DE program, the hard work of the staff, and and our awareness that in the short time we have had here we cannot expect to know more about this program than the AIF-DE team. • “AIF DE program,” “Schools Online,” “SOL” and “Digital Equalizer” identity should be further consolidated. • Provides maximum value and clarity for donors who are interested in participating in program • Financial assistance, rewards in kind, certificates, other motivating actions to keep ILCs open and available to students for additional hours once schools would normally close should be considered. • Spending by AIF is certainly justified to increase ILC utilization and maximize the value of the centers to students, teachers, and the community. 31

32. Curriculum Recommendations • Continued Focus on teacher professional development. • No need focus on 100% teacher training if some staff is near retirement. • Those resistant to computers should be paired with a computer trained teacher • Teachers with a desire to learn beyond AIF/DE training should be directed to Online Training or other development opportunities. • Create a national digital trust for content development • Portal development • Partnerships with other organizations to share content. • Share what works between schools. Email discussion lists, other forms of knowledge sharing. • Training curriculum should be flexible and adaptable to different schools • Emphasize that we are concentrating on learning through computers rather than learning about computers. 32

33. Infrastructure Recommendations • The current machines may be obsolete in three to five years. • Expensive upgrades of software and hardware will be required if the Digital Divide is to be eliminated. • AIF should verify arrangements with WIPRO, or make other plans needed to ensure sustenance after 3 year program. • Explore other technology options that can broaden impact of digital technology: • Computers outside the ILCs, or scale the DE investment based on number of students in grades 7 to 10. • Digital Projectors for large classroom/auditorium. • Traditional overheads, other types of study materials that can be constructed with the help of the Internet, PCs. • "Professionalize" technical support. • This could either be a dedicated person in the school or a key contact at the hardware or software company. • Having one facilitator responsible for training, running the lab,and all technical support (in addition to teaching responsibilities) may not be reasonable 33