1 / 36

Formative and Summative Evaluations

Formative and Summative Evaluations. Instructional Design For Multimedia. Implementation. Evaluation Phases. Summative Evaluation. Development. Analysis. Design. Formative Evaluation. Formative Evaluation. Occurs before implementation

clail
Download Presentation

Formative and Summative Evaluations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Formative and Summative Evaluations Instructional Design For Multimedia

  2. Implementation Evaluation Phases Summative Evaluation Development Analysis Design Formative Evaluation

  3. Formative Evaluation • Occurs before implementation • Determines the weaknesses in the instruction so that revisions can be made • Makes instruction more effective and efficient

  4. Formative Evaluation Is Especially Important When… • Designer is novice • Content area is new • Technology is new • Audience is unfamiliar • Task performance is critical • Accountability is high • Client requests/expects evaluation • Instructions will be disseminated widely • Opportunities for later revision are slim

  5. Formative Evaluation Phases Design Reviews Expert Reviews One-to-One Evaluation Learner Validation Small-Group Evaluation Field Trials Ongoing Evaluation

  6. Design Reviews • Should take place after each step of the design process • Goal Review • Review of Environment and Learner Analysis • Review of Task Analysis • Review of Assessment Specifications

  7. Design ReviewsGoal Review

  8. Design ReviewsEnvironment and Learner Analysis Review

  9. Design ReviewsTask Analysis Review

  10. Design ReviewsAssessment Specification Review

  11. Expert Reviews • Should take place when instructional materials are in draft form • Experts include: • Content Experts • Instructional Design Experts • Content-specific Educational Experts • Learner Experts

  12. Expert ReviewsContent Experts • Subject matter experts (SMEs) review for accuracy and completeness • Is the content accurate and up-to-date? • Does the content present a consistent perspective? Example: Physics expert

  13. Expert ReviewsInstructional Design Experts • Reviews for instructional strategy and theory • Is the instructional strategies consistent with principles of instructional theory? Example: Instructional Designer

  14. Expert ReviewsContent-Specific Educational Expert • Reviews for pedagogical approach in content area • Is the pedagogical approach consistent with current instructional theory in the content area? Example: Science education specialist

  15. Expert ReviewsLearner Expert • Reviews appropriateness such as vocabulary, examples and illustrations • Are the examples, practice exercises, and feedback realistic and accurate? • Is the instruction appropriate for target learners? Example: 6th grade teacher

  16. Expert ReviewsProcess • Distribute draft material to experts • Collect comments and prioritize into categories such as: • CriticalRevisions should be made immediately • Non-criticalDisregard or address at a later date • More InfoFind more data or information

  17. Learner Validation • Try instruction with representative learners to see how well they learn and what problems arise as they engage with the instruction • One-to-One Evaluations • Small Group Evaluation • Field Trials

  18. Learner ValidationOne-to-One Evaluation • Present materials to one learner at a time • Typical problems that might arise are: • Typographical errors • Unclear sentences • Poor or missing directions • Inappropriate examples • Unfamiliar vocabulary • Mislabeled pages or illustrations • Make revisions to instruction • Conduct more evaluations if necessary

  19. Learner ValidationOne-to-One Evaluation Process • Present materials to student • Watch student interact with material • Employ “Read-Think-Aloud” method • Continually query students about problems they face and what they are thinking • Assure student that problems in the instruction are not their fault • Tape record or take notes during session • Reward participation

  20. Learner ValidationSmall Group Evaluation • Present materials to 8-12 learners • Administer a questionnaire to obtain general demographic data and attitudes or experiences • Problems that might arise are: • Students have more or less entry level skills than anticipated • Course was too long or too short • Learners react negatively to the instruction • Make revisions to instruction • Conduct more evaluations if necessary

  21. Learner ValidationSmall-Group Evaluation Process • Conduct entry-level and pretests with students • Present instruction to students in natural setting • Observe students interacting with materials • Take notes and/or videotape session • Only intervene when instruction cannot proceed without assistance • Administer posttest • Administer attitudinal survey or discussion • Reward participation

  22. Learner ValidationField Trials Evaluation • Administer instruction to 30 students • Problems that might arise: • Instruction is not implemented as designed • Students have more or less entry-level skills • Assessments are too easy or difficult • Course is too long or too short • Students react negatively to instruction • Make revisions • Conduct more field trials if necessary

  23. Learner ValidationField Trials Evaluation Process • Administer instruction students in normal setting, in various regions and with varying socioeconomic status • Collect and analyze data from pretests and posttests • Conduct follow-up interviews if necessary • Conduct questionnaire with instructors who deliver the training

  24. Formative EvaluationOngoing Evaluation • Continue to collect and analyze data • Collect all comments/changes made by teachers who deliver the instruction • Keep track of changes in learner population • Revise instruction or produce new material to accompany instruction as needed

  25. Conduct design reviews after each stage of design including goals, environment and learner analysis, task analysis and assessment specifications Conduct expert reviews with content, instructional design, content-specific educator and learner experts Conduct one-to-one evaluations with students Conduct small-group evaluations with 8-12 students Conduct field trials with 30 or more students Conduct ongoing evaluations Formative Evaluation Summary

  26. Summative Evaluation • Occurs after implementation(after program has completed full cycle) • Determines the effectiveness, appeal, and efficiency of instruction • Assesses whether the instruction adequately solves the “problem” that was identified in the needs assessment

  27. Summative Evaluation Phases Determine Goals Select Orientation Select Design Design/Select Evaluation Measure Collect Data Analyze Data Report Results

  28. Summative EvaluationDetermine Goals • Identify questions that should be answered as a result of the evaluation • Does implementation of the instruction solve the problem identified in the assessment? • Do the learners achieve the goals of the instruction? • How do the learners feel about the instruction? • What are the costs of the instruction, what is the return on investment (ROI)? • How much time does it take for learners to complete the instruction? • Is the instruction implemented as designed? • What unexpected outcomes result from the instruction?

  29. Summative EvaluationDetermine Goals • Select indicators of success • If program is successful, what will we observe it in: • Instructional materials? • Learner’s activities? • Teachers knowledge, practice and attitudes? • Learner’s understanding, processes, skills, and attitudes?

  30. Summative EvaluationSelect Orientation • Come to an agreement with client on most appropriate orientation of evaluation • Objectivism – Observation and quantitative data collected to determine the degree to which the goals of the instruction have been met • Subjectivism – Expert judgment and qualitative data not based on instructional goals

  31. Summative EvaluationSelect Design of Evaluation • What data will be collected, when, and under what conditions? • Instruction, Posttest • Pretest, Instruction, Posttest • Pretest, Instruction, Posttest, Posttest, Posttest

  32. Summative EvaluationDesign or Select Evaluation Measures • Payoff Outcomes - Review statistics that may have changed after instruction was implemented • Learning Outcomes - Measure for an increase in test scores • Attitudes - Conduct interviews, questionnaires, and observations • Level of Implementation - Compare design of program to how it is implemented • Costs - Examine costs to implement and continue program, personnel, facilities, equipment, and material

  33. Summative EvaluationCollect Data • Devise a plan for the collection of data that includes a schedule of data collection periods

  34. Summative EvaluationAnalyze Data • Analyze the data so that it is easy for the client to see how the instructional program affected the problem presented in the needs assessment.

  35. Summative EvaluationReport Results • Prepare a report of the summative evaluation findings that includes: • Summary • Background • Description of Evaluation Study • Results • Discussion • Conclusion and Recommendations

  36. Determine the goals of the evaluation Select objective or subjective orientation Select design of evaluation plan Design or select evaluation measures Collect the data Analyze the data Report the results Summative Evaluation Summary

More Related