The Online Academy

1. The Online Academy Evaluation Plan

2. Need for Evaluation • Inform and direct current and future development • Examine the usability of the checklist • Organization and self-regulation • Determine effectiveness of navigation

3. Need for Evaluation 4. Review course modules for the following: • Correlation to SOL • Clarity of instruction • Engagement and motivation of instruction • Content presented at the appropriate level • Adequacy of the resources, skills, and content-adequate for challenge • Pedagogical approach and format • Pedagogical re-engineering

4. Purpose/Goals • Analyze instructional strengths (pedagogical re-engineering) • Determine the effectiveness of the design and transfer of the materials from classroom to online. • Evaluate the usability and the benefits of the checklist to the learner • Locate ease of use problems

5. Strategy • The evaluation strategy is to determine the effectiveness of the initial modules of the seven courses being developed for TOA. Using expert and one-to-one reviews. • The results of this evaluation will be used to inform and direct the current and future design and development of TOA.

6. Objectives Expert review • Find a way to determine whether the site adheres to Clark and Mayer criteria and Gardner criteria. • Recruit experts. • Prepare questions. • Design data collection tools. • Set up testing area. • Conduct evaluation. • Analyze results. • Write evaluation.

7. Objectives One-to-One review • Recruit students. • Prepare evaluation questions. • Design data collection tools. • Write script. • Set up testing area. • Conduct evaluation. • Analyze results. • Write evaluation.

8. Social/Cultural Factors • Priscilla Norton, TOA Project Director • The Policy Board • School districts • Subject Matter Experts (SMEs) • Students

9. Levels of Evaluation • multi-level evaluation to explore as many perspectives as possible • evaluate at Levels 1, 2, and 3 of Kirkpatrick’s Levels of Evaluation. • the level of formality of the evaluations will vary from informal to formal

10. Data Collection • use observations during the one-to-one evaluations with the students • conduct interviews for the expert reviews • follow a script with a list of prepared questions, allowing for the introduction of additional questions or probing questions when needed. • evaluations will be recorded on either audiotape or videotape. • we will document observations and interviews in field notes.

11. Data Analysis Analyze data by reviewing field notes and transcripts to look for common themes, patterns, and trends.

12. Logistics • Identified potential evaluators • Assigned team members to conduct evaluation (2 to each evaluation, one person interviews while the other records) • Identified tentative location and time • Written interview questions • Developed a protocol and script for each type of evaluation.