270 likes | 478 Views
Effective Implementation Formative and Summative Project Evaluation. When Bad things happen to Good projects!. Great ideas become ruined with bad implementations! Brad Wiley English teacher case-text Innovations get bad press with bad implementations! Failed “good ideas”.
E N D
Effective ImplementationFormative and Summative Project Evaluation
When Bad things happen to Good projects! • Great ideas become ruined with bad implementations! • Brad Wiley English teacher case-text • Innovations get bad press with bad implementations! • Failed “good ideas”
Role of Implementation • Diffusion • Dissemination • Adoption • Stakeholders
Adoption Process • Awareness • Interest • Evaluation • Trial • Full Adoption • Integration
Adoption Models • Research based • CBAM--Concerns-Based Adoption Model • Focuses on the concerns of the users • Stages of Concern Questionnaire • Level of Use Questionnaire
Implementation • Role as a designer • Recommendations • Experiences • Role as a teacher/user/learner • Limitations • Experiences
Assignment • Using the case of Brad Wiley in Chapter 17, identify what principles of good implementation were ignored or violated and write a short implementation plan for Mr. Wiley’s project.
Formative and Summative Evaluation of Design • Assessments were used to determine student achievement • Evaluation of the design and the instructional materials is the current topic.
Occurs at 2 times • Evaluation of the materials and processes for the purpose of revisions--Formative • Once materials and processes in place-they are evaluated in terms of their effectiveness-Summative
Formative • Occurs during development of materials • Revisions are expected! • In business-skipped due to cost!
Formative evaluation is especially important if… • Designer is novice • Content is new • Technology is new • Audience is new • Designer new to ID • New or unfamiliar instructional strategy • Task performance is critical • Client expectations and later revision is not possible
Formative Review Stages • Design reviews--before development • Expert reviews--after development before use • Learner validation--during use • Ongoing evaluation--during use
Design Reviews • Goal Review • Environment and Learner Analysis Review • Task Analysis Review • Assessment Specifications Review
Expert Reviews • Content experts • ID experts • Educational specialists • Teachers • Target audience specialist
One-to-One Evaluation • Tries materials with 2-3 members of target audience • Sits down one-to-one--observation and questions • Typos, unclear directions, bad examples, poor illustrations, bad URLs
Small Group Evaluation • Uses more learners…usually after the one-to-one revisions were made. • Usually only observation, data gathered through charting • Survey after can be used with learners. • pg336
Field Trials • Determine effectiveness of revisions made after small group eval • Find problems with administration of materials in real life situations • Validate instruction with larger group to predict effectiveness
Which stage of the formative evaluation procedure does each statement describe? Teachers/trainers report the appropriateness of the documentation that accompanies a package of computer based instructional software..
Which stage of the formative evaluation procedure does each statement describe? Experts in the content area for which a slide-tape set was developed review the materials for accuracy and currency.
Which stage of the formative evaluation procedure does each statement describe? Individual students review the storyboards of an instructional videotape with the designer.
Which stage of the formative evaluation procedure does each statement describe? At least 30 students go through the instruction.
Which stage of the formative evaluation procedure does each statement describe? Designer only assists students when they encounter severe difficulties.
Which stage of the formative evaluation procedure does each statement describe? Students read and think aloud as they interact with written instructional materials.
Which stage of the formative evaluation procedure does each statement describe? Test materials are tried out with groups of individuals some of whom are “masters” of the content, some of whom are “nonmasters.”
Which stage of the formative evaluation procedure does each statement describe? Data are collected regarding problems with the instructionafter the instruction is implemented in the target contexts.
Summative Evaluation • Final evaluation of effectiveness of the design, materials and learning. • Usually will impact continued use • Determine goals of the evaluation--usually determined by client
Group Assignment • Determine what formative evaluation process you could use with your instructional design. • Discuss and decide as a group • Complete Formative Evaluation Documents and submit to Cheryl by e-mail…one per group