Assignment 4.1 Formative Evaluation of an Internet Searching Tutorial

1. Assignment 4.1Formative Evaluation of an Internet Searching Tutorial Q410 – Formative and Summative Evaluation March 15, 2003 Submitted By: Sun Hong Hwang Paul Kelly Meredith Rogers

2. Formative Evaluation of an Internet Searching TutorialIntroduction • Front-End Analysis • Problem • The Park Hill School District (Kansas City, Missouri) identified a gap between teacher performance utilizing the Internet (actual) and an optimal expectation that teachers be able to effectively and efficiently find information to support instructional processes. • Proposed Solution • Based upon the facts obtained via a front-end analysis (performance analysis, environmental analysis, learner analysis, and needs assessment), it was determined that K-12 teachers in the Park Hill School District needed training in effectively searching the Internet to close the gap between the “optimals” and “actuals” found in during the front-end analysis.

3. Formative Evaluation of an Internet Searching TutorialIntroduction • Instructional Materials/Product • Materials • “Take the Garbage out of Internet Searching” was developed specifically as a training tool to address the performance problem outlined on the previous slide. • Learner Objectives • Learners will. . . • . . .explore a variety of search engines and online directories and note differences of each. • . . . effectively utilize a search engine and a directory for online searches of information. • . . . improve the reliability of simple searches with advanced operators. • . . . explore natural language searches. • . . . effectively choose the proper search engine for a given online search task.

4. Formative Evaluation of an Internet Searching TutorialIntroduction • Instructional Materials/Product • Status of Materials • “Take the Garbage out of Internet Searching” is currently being developed, awaiting the results of this formative evaluation prior to distribution. • Media • “Take the Garbage out of Internet Searching” iscurrently available for preview in an online version and a CD-ROM format. The product/materials were developed using Macromedia Authorware 6.0, which can be exported in a web format or as an executable runtime program from CD-ROM. • The CD-ROM version was utilized for the purposes of this report. • The URL to the online version is: http://content.parkhill.k12.mo.us/users/pvkelly/Web/internetsearching-beta2.htm

5. Formative Evaluation of an Internet Searching TutorialFormative Evaluation Framework • Goals of Evaluation • The goal of this formative evaluation is to improve the effectiveness of the instructional materials prior to usage by a large population of K-12 teachers. The goals of this formative evaluation process include . . . • . . . identifying the deficiencies in learning effectiveness of the instructional materials; • . . . locating ease of use problems with the instructional materials; and • . . . evaluating the efficiency of the instruction within the materials. • Evaluators • An Expert Reviewer • Two One-on-One Reviewers

6. Formative Evaluation of an Internet Searching TutorialExpert Review • Reviewer • Mr. Robert Borst, Middle School Mathematics Teacher at Gentry Middle School, Columbia, Missouri • Expertise • Teacher of courses for the SISLT at the within the track of technology in schools • Web page designer & creator of multi-media presentations for the purpose of conducting teacher in-services for the Columbia Public School District. • Evaluation Goals • Evaluate the strengths and weaknesses of the program’s efficiency of operation • Evaluate the usefulness of the included content. • Evaluation Tool • A single three-page questionnaire was to Mr. Borst to complete on his own as he worked through the various screens of the program. (Appendix 01)

7. Formative Evaluation of an Internet Searching TutorialExpert Review • Recommendations of Expert Review • The information is too wordy. There is too much to read through with some of the topics outlined in the program. He said that the amount of information is far more in-depth that most K-12 teachers will ever need to know or will ever use. • The information is “presented” for the most part rather than given in an interactive mode for the user to demonstrate his or her skills. • The transitions between screens are inconsistent and very aggravating at times. A single type of quick transition needs to be considered, rather than a different type of screen change with every new topic.

8. Formative Evaluation of an Internet Searching TutorialOne-on-One Reviews • Reviewer A • Amy Carso, Middle School Language Arts Teacher at Gentry Middle School, Columbia Public School District • Expertise • A novice when it comes to understanding how the computer works and all its capabilities. • She describes herself as being “pretty comfortable [with technology now]” • Reviewer B • Keri McConnell, 5th Grade Teacher, Park Hill School District, Kansas City, Missouri • Expertise • She serves as a trainer for many technology-based staff development initiatives. • She is well respected by her peers for her use of technology in the classroom, as well as her ability and willingness to show others.

9. Formative Evaluation of an Internet Searching TutorialOne-on-One Reviews • Evaluation Goals • The reviewers evaluated . . . • . . . the directions and commands of this instruction program for clarity and operability from a learner’s perspective. • . . . the aesthetics design of the program for engaging the learner. • . . . the level of interaction for the learner. • . . .the content with regards to meeting the needs that K-12 teachers generally desire in order to improve the efficiency of their Internet searching abilities. • Evaluation Tool • A single three-page questionnaire was reviewed each user during their sessions and given to them to complete on their own as they worked through the various screens of the program (Appendix 02).

10. Formative Evaluation of an Internet Searching TutorialOne-on-One Review • Recommendations of One-on-One Reviews • Font size is too small making the user having to strain to read some of the material. • There needs to be more interaction for the learner as they are reading through the information. The reviewers suggest the inclusion of sample exercises, pre-test questions or actual connections to the Internet to practice some of their skills. • There was no real challenge to this program for the user. Both reviewers summarized the program, as just basically being a reading exercise, which many educators know will not keep a learner engaged in any learning process for very long.

11. Formative Evaluation of an Internet Searching TutorialRecommended Revisions • Recommendations of Formative Evaluation • Based on the data collected by the expert reviewer and the two one-on-one reviewers, the following items should be considered for revision in order to improve the program with the hope of addressing the objectives of this program more efficiently. • Font • Needs to be larger or more bold; change style to a more squared shape. • Directionality • Either enlarge the arrows on the quiz screens or remove them and just make the words the link; be consistent with what are links within the program, add a skip; create ways to restart the program more easily; add a skip button on opening screens to bypass; fix or remove the backwards button on the first two screens. • Aesthetics • Use only one kind of transition between screens; use a lighter background (e.g. white or ecru) and/or a deeper font color (e.g. navy or black).

12. Formative Evaluation of an Internet Searching TutorialRecommended Revisions • Recommendations of Formative Evaluation (continued) • Based on the data collected by the expert reviewer and the two one-on-one reviewers, the following items should be considered for revision in order to improve the program with the hope of addressing the objectives of this program more efficiently. • Content • Content assume the user has background knowledge on the topic; simplify key terms or provide examples to explain the terms directly following the term in the write-up; put in correct answers to incorrectly answered questions to give clearer feedback. • Operation • A claim statement needs to be made explaining that Authorware software is needed in order to run the program and that it can be easily downloaded to run the program form the web version, or that it is included on the CD-ROM already. • Interaction • Build in more interactive features such as practice examples where the user could be practicing on the Internet with the instructional materials open; cut down the content and focus on the key elements used in everyday searches; add more quiz questions, provide a pre-test on some skill; add a help menu.

13. Formative Evaluation of an Internet Searching TutorialSelf-Evaluation • Strengths of our Evaluation • The organization of the evaluation tools • The use of the scripts during both types of reviews • Communication within our evaluation team and the analysis of our results. • Areas of Improvement • The development of clear and specific goals and criteria for our reviewers to ensure they meet our needs for evaluation • The clarity in the objectives of the instructional materials so that the tools developed for the evaluation are better aligned.

14. Formative Evaluation of an Internet Searching TutorialReferences • Fitzgerald, G., & Koury, K. (2001-2002). The KidTools Support System. U.S. Department of Education. Project #K033271. http://www.coe.missouri.edu/~vrcbd • Tessmer, M. (1993). Planning and conducting formative evaluations: Improving the quality of education and training. London, England: Kogan Page Limited.