Evaluating Open Educational Resource (OER) Objects

1. Evaluating Open Educational Resource (OER) Objects Rubric III: Utility of Materials Designed to Support Teaching CC BY Achieve 2013

2. Open Educational Resources: [With Webster’s Definitions] O Open: Containing an open license - no restrictions on remixing or reusing [Having no enclosing or confining barrier; not restricted to a particular group or category of participants] E Educational: Used for teaching and learning [Pertaining to the action or process of educating or being educated] Resource: Object used to support an effort or task [A source of support or aid, especially one that can be readily drawn upon when needed] R

3. Open Educational Resources (OER) • What are OER? • OER are teaching, learning, and research resources that contain an open license. • They provide extraordinary opportunities for educators to freely share knowledge and resources. • They offer great potential for instructional innovation and networks for sharing best practices. • There are, literally, millions of OER objects available online.

4. Open Educational Resources (OER) • OER objects may include (but are not limited to): • Images • Applets • Games • Worksheets • Lesson plans • Original-source texts • Assessments • Units • Textbooks • Teacher/learner support materials • NOTE: Any smaller component of a more complex object, that can exist as a stand-alone, also qualifies as an “object” (e.g. a unit in a textbook, a lesson in a unit, or an activity in a lesson).

5. The Achieve OER Evaluation Rubrics • How and why were the Achieve OER Rubrics created? • Educators using the vast system of sharing need a method for filtering OER to meet the needs of their students. • The purpose of the rubrics is to provide a structure for evaluating an online resource in a systematic, purposeful and comprehensive way. • There are two ways of approaching the evaluation of a resource: • Holistically evaluating the object with a single rating, (similar to the star-rating systems used on Netflix, Yelp and Amazon) • Separately evaluating each of the components of quality, as with the Achieve OER Rubrics. • The rubric criteria are based on Achieve protocols used to assist states in alignment and quality review studies. • The Achieve OER Evaluation Tool, hosting the rubrics on OERCommons.org, allows a user to identify, evaluate and sort objects based on the specified essential elements of quality.

6. Applying the Achieve OER Rubrics • How do the Achieve OER Rubrics work? • The rubrics represent an evaluation system for objects found within Open Educational Resources. • They are hosted as an online evaluation tool on the repository, OERCommons.org, but can also be used independently. • They are applied to any content area. • NOTE: At this stage only Common Core Standards for English Language Arts/Literacy and Mathematics are available in the Achieve OER Evaluation Tool. Content standards for History, Social Studies, Sciences, and technical subjects are not yet available. • Each rubric is applied independently to an object. • They rate the potential, not the actual, effectiveness of an object. • They use a five-point scoring system to describe levels of quality.

7. The Achieve OER Evaluation Tool Rubrics • This presentation will focus on Rubric III: • Rubric I. Degree of Alignment to Standards • Rubric II. Quality of Explanation of the Subject Matter • Rubric III. Utility of Materials Designed to Support Teaching • Rubric IV. Quality of Assessment • Rubric V. Quality of Technological Interactivity • Rubric VI. Quality of Instructional Tasks and Practice Exercises • Rubric VII. Opportunities for Deeper Learning • NOTE: Sometimes one or more of the rubrics may not be relevant to a particular object. In those cases, a rating of N/A is appropriate.

8. Rubric III: Utility of Materials Designed to Support Teaching • How and Why is Rubric III Applied? • Applied to objects designed to support teachers in planning or presenting subject matter. • Primary user would be the teacher. • Evaluates the potential utility of an object for the majority of instructors at the intended grade level. • Not applicable if the object is not designed to support teachers in planning and/or presenting subject matter, even though it might be possible for an educator to find such an application.

9. Rubric III: Utility of Materials Designed to Support Teaching • 3: An object is ratedsuperioronly if all of the following are true: • The object provides materials that are comprehensive and easy to understand and use. • The object includes suggestions for ways to use the materials with a variety of learners. These suggestions include materials such as “common error analysis tips” and “precursor skills and knowledge” that go beyond the basic lesson or unit elements. • All objects and all components are provided and function as intended and described. For larger objects like units, materials facilitate the use of a mix of instructional approaches (direct instruction, group work, investigations, etc.). • For example a superior object for this rubric includes clear descriptions of the lesson’s activities with suggestions for interventions, extensions, and for use with English language learners. Also the time needed for lesson planning and presentation appears to be accurately estimated, materials list is complete, and all explanations make sense.

10. Rubric III: Utility of Materials Designed to Support Teaching • 2: An object is ratedstrongif it offers materials that are comprehensive and easy to understand and use but falls short of “superior” for either one of two reasons: • The object does not include suggestions for ways to use the materials with a variety of learners (e.g., error analysis tips). • Some core components (e.g., directions) are underdeveloped in the object. • For example an object would be considered strong if it includes clear lesson directions, with suggestions for students with special needs, but does not include a list of materials needed or time estimates, or the object has clear directions for the teacher, including student materials, but does not provide suggestions for intervention, extension, or for English language learners students.

11. Rubric III: Utility of Materials Designed to Support Teaching • 1: An object is ratedlimitedif it includes a useful approach or idea to teach an important topic but falls short of “strong” for either one of two reasons: • The object is missing important elements (e.g. directions for some parts of a lesson are not included). • Important elements do not function as they are intended to (e.g. directions are unclear). Teachers would need to supplement this object to use it effectively. • For example an object, a lesson plan for teaching division of fractions has clear directions and general suggestions for the types of activities for students but has no specific student materials, assessments, or suggestions for how to adjust the lesson for students with special needs. It is considered limitedsince, in order to use the object, the teacher will need to create questions or problems, student worksheets and/or assessments and will need to make adjustments for intervention and extension.

12. Rubric III: Utility of Materials Designed to Support Teaching 0: An object is ratedvery weak or no valuefor instructional purposes if it is confusing, contains errors, is missing many important elements, or is for some other reason simply not useful. For example an object, a lesson plan for teaching division of fractions, would be considered very weak if there are some materials included designed to support teaching but several student support documents are missing and they include only a few of the needed solution keys, with errors throughout.

13. Rubric III: Utility of Materials Designed to Support Teaching • N/A: This rubric is not applicable(N/A) for an object that is not designed to support teachers in planning and/or presenting subject matter. It may be possible that an educator could find an application for such an object during a lesson, but that would not be the intended use. • For example a game that is designed to be used by students as an intervention or pre-assessment of skill knowledge would not necessarily need to have teacher support materials included. In this case Rubric III would be not applicableto the object.

14. Using the Achieve OER Evaluation Tool On OERCommons.org Rubric III

15. Achieve OER Evaluation Toolon OERCommons.org • OERCommons.org hosts the rubrics as the Achieve OER Evaluation Tool • NOTE: Specific instructions for Using the OERCommons Website to search for and evaluate objects, using all of the rubrics, can be found in another slide presentation of this training package.

16. Rubric III: Utility of Materials Designed to Support Teaching Achieve OER Evaluation Tool Using OERCommons.org to evaluate an object for its utility of materials designed to support teaching: After you have evaluated the object using Rubrics I and II, you are ready to apply Rubric III. Begin by clicking on the title line for Rubric III or on at the end of Rubric II.

17. Rubric III: Utility of Materials Designed to Support Teaching Achieve OER Evaluation Tool To view a video about Rubric III click here…

18. Rubric III: Utility of Materials Designed to Support Teaching Achieve OER Evaluation Tool Hover over the rating numbers to see an abbreviated version of the rating descriptors. Review the object with your focus on the utility of the teacher support materials and select the appropriate rating.

19. Rubric III: Utility of Materials Designed to Support Teaching Achieve OER Evaluation Tool To download a full text version of the rubrics click here:

20. Rubric III: Utility of Materials Designed to Support Teaching Achieve OER Evaluation Tool • Across the bottom of Rubric III you see options to: • Leave a [Comment], • Clear rating, • [Save & Go to the Next Rubric] (You can also click on any rubric title to leave Rubric III, but your ratings will not be saved.)

21. Achieve OER Evaluation Toolon OERCommons.org: Rubric III Now you are ready to rate the object using the Achieve OER Evaluation Tool’s other rubrics. Rubrics can be used in any order and can be selected by simply clicking on the rubric title. Remember to click after each rubric to ensure that your work is saved. You can tell if your work was saved by the symbol at the beginning of each rubric title: indicates not yet rated or rating was not saved indicates your rating has been saved

22. Slides developed by Achieve: www.achieve.org 1400 16th Street, NW / Suite 510 Washington, DC 20036