Evaluating Open Educational Resource (OER) Objects

1. Evaluating Open Educational Resource (OER) Objects Rubric I: Alignment to Standards CC BY Achieve 2013

2. Open Educational Resources: [With Webster’s Definitions] O Open: Containing an open license - no restrictions on remixing or reusing [Having no enclosing or confining barrier; not restricted to a particular group or category of participants] E Educational: Used for teaching and learning [Pertaining to the action or process of educating or being educated] Resource: Object used to support an effort or task [A source of support or aid, especially one that can be readily drawn upon when needed] R

3. Open Educational Resources (OER) • What are OER? • OER are teaching, learning, and research resources that contain an open license. • They provide extraordinary opportunities for educators to freely share knowledge and resources. • They offer great potential for instructional innovation and networks for sharing best practices. • There are millions of OER objects available online.

4. Open Educational Resources (OER) • OER objects may include (but are not limited to): • Images • Applets • Games • Worksheets • Lesson plans • Original-source texts • Assessments • Units • Textbooks • Teacher/learner support materials • NOTE: Any smaller component of a more complex object, that can exist as a stand-alone, also qualifies as an “object” (e.g. a unit in a textbook, a lesson in a unit, or an activity in a lesson).

5. The Achieve OER Evaluation Rubrics • How and why were the Achieve OER Rubrics created? • Educators using the vast system of sharing need a method for filtering OER to meet the needs of their students. • The purpose of the rubrics is to provide a structure for evaluating an online resource in a systematic, purposeful and comprehensive way. • There are two ways of approaching the evaluation of a resource: • Holistically evaluating the object with a single rating, (similar to the star-rating systems used on Netflix, Yelp and Amazon) • Separately evaluating each of the components of quality, as with the Achieve OER Rubrics. • The rubric criteria are based on Achieve protocols used to assist states in alignment and quality review studies. • The Achieve OER Evaluation Tool, hosting the rubrics on OERCommons.org, allows a user to identify, evaluate and sort objects based on the specified essential elements of quality.

6. Applying the Achieve OER Rubrics • How do the Achieve OER Rubrics work? • The rubrics represent an evaluation system for objects found within Open Educational Resources. • They are hosted as an online evaluation tool on the repository, OERCommons.org, but can also be used independently. • They are applied to any content area. • NOTE: At this stage only Common Core Standards for English Language Arts/Literacy and Mathematics are available in the Achieve OER Evaluation Tool. Content standards for History, Social Studies, Sciences, and technical subjects are not yet available. • Each rubric is applied independently to an object. • They rate the potential, not the actual, effectiveness of an object. • They use a five-point scoring system to describe levels of quality.

7. The Achieve OER Evaluation Tool Rubrics • This presentation will focus on Rubric I: • Rubric I. Degree of Alignment to Standards • Rubric II. Quality of Explanation of the Subject Matter • Rubric III. Utility of Materials Designed to Support Teaching • Rubric IV. Quality of Assessment • Rubric V. Quality of Technological Interactivity • Rubric VI. Quality of Instructional Tasks and Practice Exercises • Rubric VII. Opportunities for Deeper Learning • NOTE: Sometimes one or more of the rubrics may not be relevant to a particular object. In those cases, a rating of N/A is appropriate.

8. Rubric I: Degree of Alignment to Standards • Alignment to Standards is Required Before Evaluation is Possible • Before the degree of alignment can be rated, an object must be aligned to standards. • Some objects will not have an alignment identified, either in the object or in the online tool. • Alignments suggested by earlier users may need to be verified. • In the online tool alignment to standards is required so that accurate data for Rubric I can be collected.

9. Rubric I: Degree of Alignment to Standards • The following steps are not part of the OER evaluation process but would be required for reviewers who need to align an object, or to verify alignments made by previous reviewers, before rating. • Six Steps in Alignment to the CCSS: • Review the object thoroughly, working through tasks and/or exercises. • Compare your work with keys/solutions and other instructional support materials. • Identify the specific content and performance(s) required in the object. • Identify the grade level for the object. • Match the grade level, content, and performance requirements to the appropriate CCSS, including the Mathematical Practices and texts for ELA/Literacy. • Make notes about any requirements in the object that are not addressed in the standard(s) and any requirements in the standard that are not addressed in the object.

10. Rubric I: Degree of Alignment to Standards • Evaluation of Alignments Using Rubric I • Aligning objects, and rating the alignments accurately, allows future users to sort by subject, grade level, and even by a specific standard. • Only number ratings (3, 2, 1, 0) will be averaged in the overall results for a particular object. If one or more standards are poorly aligned a 0- or 1-rating is appropriate, rather than N/A. • N/A ratings in Rubric I are rare. However if no CCSS can be found that align to the object, or if the resource is not intended to align to standards (e.g. a raw set of data or a map), then you may rate Rubric I as N/A and move on to Rubric II. • All selected standards must be rated before moving to Rubric II.

11. Rubric I: Degree of Alignment to Standards • Each Standard is Rated Separately Using the Following Criteria: • 3: An object has superioralignment only if both of the following are true: • All of the content and performance expectations in the identified standard are completely addressed by the object. • The content and performance expectations of the identified standard are the focus of the object. While some objects may cover a range of standards that could potentially be aligned, for a superior alignment the content and performance expectations must not be a peripheral part of the object. • Math Alignment Rating Example: An object is aligned to CCSS 2.NBT.2, “Count within 1000; skip-count by 5s, 10s, and 100s.” The rating is a 3 if the primary focus of the object is counting to 1000 and skip counting by 5’s, 10’s, and 100’s, with any other content/performance requirements secondary to this. For example if an object focuses on all these requirements, but also includes a peripheral activity addressing other concepts, such as counting by 2’s, relating counting to addition and subtraction, or a measurement activity, this would not detract from the 3-rating. • [Examples of ELA/Literacy alignment ratings can be found at the end of this section.]

12. Rubric I: Degree of Alignment to Standards • 2: An object has strong alignment for either one of two reasons: • Minor elements of the standard are not addressed in the object. • The content and performance expectations of the standard align to a minor part of the object. • Math Alignment Rating Example: An object is aligned to CCSS 2.NBT.2, “Count within 1000; skip-count by 5s, 10s, and 100s.” If the object requires counting to 1000 and skip counting by 5’s and 10’s, it would be considered a strong alignment as it aligns with the majority of the standard’s requirements, with only a minor part not addressed (counting by 100’s). A 2-rating would also be appropriate if all of the standard’s content and performance are addressed but some in a peripheral way. For example, counting by 100’s might be addressed but only in an extension activity, which users may opt not to use or may not even see. • [Examples of ELA/Literacy alignment ratings can be found at the end of this section.]

13. Rubric I: Degree of Alignment to Standards 1: An object has limited alignment if a significant part of the content or performance expectations of the identified standard is not addressed in the object, as long as there is fidelity to the part it does cover. Math Alignment Rating Example: An object is aligned to CCSS 2.NBT.2, “Count within 1000; skip-count by 5s, 10s, and 100s.” However if it addresses only counting with numbers to 500 (without skip-counting), it would be considered to have limited alignment, i.e., the object aligns very closely with a limited part of the standard. This would also be the case if the object addresses only counting by 5s and 10s, but does not specify counting within 1000 or by 100s. [Examples of ELA/Literacy alignment ratings can be found at the end of this section.]

14. Rubric I: Degree of Alignment to Standards • 0: An object has very weakalignment for either one of two reasons: • The object does not match the intended standards. • The object matches only to minimally important aspects of a standard, cluster or domain. These objects will not typically be useful for instruction of core concepts and performances covered by the standard. • Math Alignment Rating Example: An object is aligned to CCSS 2.NBT.2, “Count within 1000; skip-count by 5s, 10s, and 100s.”If the object only addresses skip-counting by 10’s; requires only counting within 100; or requires counting (no skip-counting) but has no clear indication as to the limitation (count to 1000) it would be considered a very weak alignment. • [Examples of ELA/Literacy alignment ratings can be found at the end of this section.]

15. Rubric I: Degree of Alignment to Standards • N/A: This rubric does not apply for an object that has no suggested standards for alignment. • Math Alignment Rating Example: An N/A rating for Rubric I is appropriate only if no standards can be found that align to the object or if the resource is not intended to align to standards. In those cases you may rate Rubric I with an N/A and move on to Rubric II. For example Rubric I would likely not be applicable to a set of raw data, a map, or a primary source text. • A 0-rating is preferable to N/A for those misaligned standards selected by previous raters since only numerical ratings (0, 1, 2, or 3) are averaged into the overall results for an object.

16. Rubric I: Degree of Alignment to Standards ELA ALIGNMENT RATING EXAMPLES: An object is aligned to CCSS RI.9-10.8, “Delineate and evaluate the argument and specific claims in a text, assessing whether the reasoning is valid and the evidence is relevant and sufficient; identify false statements and fallacious reasoning.” Ratings would be… 3 – If the object requires that students evaluate the argument of a text, assessing the validity of its evidence and reasoning including identifying false statements and reasoning. The text and instructional material must be constructed to include some false statements and reasoning. 2 – If the object requires that students evaluate the argument of a text, assessing the validity of its evidence and reasoning, but did not include identifying false statements and reasoning. OR if the object included all the range of evaluative performances, but the evaluation was a minor part of the activities. 1– If the object requires only delineating the argument but does not require students to evaluate its reasoning or evidence. 0– If the text of the object includes an argument, but none of the activities require that students delineate or evaluate it.

17. Using the Achieve OER Evaluation Tool On OERCommons.org Rubric I

18. Achieve OER Evaluation Toolon OERCommons.org • OERCommons.orghosts the rubrics as the Achieve OER Evaluation Tool • NOTE: Specific instructions for Using the OERCommons Website to search for and evaluate objects, using all of the rubrics, can be found in another slide presentation of this training package.

19. Achieve OER Evaluation Toolon OERCommons.org An Object’s Launch Page After any object has been evaluated, the average of ratings for all previous users, followed by the number of users, is shown on the object’s launch page. For Rubric I averages are shown for each of the selected standards. (If the object has not previously been evaluated there will be no average ratings.) The list of previously aligned standards can be seen below the average ratings.

20. Achieve OER Evaluation Toolon OERCommons.org • Rubric I requires two steps: • Align the object to standards • Rate the alignment for each aligned standard • Click on at the bottom of the launch page if alignment is needed. • NOTE: If previously selected standards are satisfactory, you can skip the alignment process and begin evaluating.

21. Achieve OER Evaluation Toolon OERCommons.org After you are satisfied with aligned standards, click on You will also be able to align to standards at any time during the application of Rubric I.

22. Achieve OER Evaluation Tool: Guidelines & Reminders • After selecting and object and clicking on from its launch page, you will be taken to the Achieve OER Evaluation Tool. • On the introduction page, seen here, you will find: • A link to a pdf version of the rubrics on Achieve’s website

23. Achieve OER Evaluation Tool: Guidelines & Reminders • Tool • A link to Achieve.org

24. Achieve OER Evaluation Tool: Guidelines & Reminders • An explanation of the Tool, including guidelines and reminders.

25. Achieve OER Evaluation Tool: Guidelines & Reminders • A link to the Common Core State Standards

26. Achieve OER Evaluation Tool: Guidelines & Reminders • And… • A link to a video providing an overview of the rubrics

27. Achieve OER Evaluation Tool: Guidelines & Reminders When you are ready to begin click on

28. Achieve OER Evaluation Toolon OERCommons.org: Rubric I The Achieve OER Evaluation Tool Home Screen: This is what the OER Evaluation Tool home screen looks like on OERCommons after you click [Start Evaluating].The rubrics can be applied in any order but it is best to start with Rubric I.

29. Achieve OER Evaluation Toolon OERCommons.org: Rubric I The Achieve OER Evaluation Tool Home Screen: To begin applying Rubric I click on align resource if alignment is needed or on skip as not applicable if Rubric I is does not apply. IMPORTANT: Selecting skip as not applicableindicates an N/A rating, which, for Rubric I, would apply only if there are no CCSS that align to the object or there is no intention by the object to align to standards. If previous users have already aligned standards they will need to be rated before moving to other rubrics.

30. Achieve OER Evaluation Toolon OERCommons.org: Rubric I The Achieve OER Evaluation Tool Home Screen: Click here to view an instructional video about Rubric I.

31. Achieve OER Evaluation Toolon OERCommons.org: Rubric I The Achieve OER Evaluation Tool Home Screen: Click here to read more about Deeper Learning from the Hewlett Foundation’s website.

32. Achieve OER Evaluation Toolon OERCommons.org: Rubric I • The Alignment Process: • After you click on align resource you will be asked to select the curriculum standard for alignment: • First identify the • subject area: • NOTE: at any time during the process you can click on [Return to Evaluation] to return to the opening evaluation screen.

33. Achieve OER Evaluation Toolon OERCommons.org: Rubric I Next select the grade level:

34. Achieve OER Evaluation Toolon OERCommons.org: Rubric I Then the learning domain:

35. Achieve OER Evaluation Toolon OERCommons.org: Rubric I And finally the alignment tag:

36. Achieve OER Evaluation Toolon OERCommons.org: Rubric I After selecting an alignment tag you can read the full text in the yellow box at the upper right.

37. Achieve OER Evaluation Toolon OERCommons.org: Rubric I Click on to add the new alignment tag, Or to go back to the rubric selection list and/or start the process over.

38. Achieve OER Evaluation Toolon OERCommons.org: Rubric I If you click on [Add Selected Tag] you will see your selection’s title and code at the upper left corner of the selection screen. Now you can search for more standards by selecting a different alignment tag or changing learning domain or even grade level.

39. Achieve OER Evaluation Toolon OERCommons.org: Rubric I You can remove a standard you have selected from the alignment by clicking the [x] next to the title.

40. Achieve OER Evaluation Toolon OERCommons.org: Rubric I When you are satisfied with your alignment selection(s), click on

41. Rubric I: Degree of Alignment to Standards Achieve OER Evaluation Tool After you click [Return to Evaluation], you will see your selected standards listed in the center of the screen for Rubric I. Click the standard you want to rate and it will be highlighted in orange.

42. Rubric I: Degree of Alignment to Standards Achieve OER Evaluation Tool Hover over the rating numbers to see an abbreviated version of the rating descriptor. Select the most appropriate rating of the for each standard identified.

43. Rubric I: Degree of Alignment to Standards Achieve OER Evaluation Tool • Across the bottom of Rubric I you see options to: • Leave a [+ Comment], • Clear rating, • Move to the [Next/Previous Standard] (or you can click on any un-highlighted standard), • [Save & Go to Next Rubric] (You can also click on any rubric title to leave Rubric I, but your ratings will not be saved.)

44. Rubric I: Degree of Alignment to Standards Achieve OER Evaluation Tool To read the full text of the rating descriptors you can click on the link for the full version of the OER Rubrics.

45. Achieve OER Evaluation Toolon OERCommons.org: Rubric I After rating your object against the criteria and descriptors of Rubric I, you will be ready to rate the object using the Achieve OER Evaluation Tool’s other rubrics. Rubrics can be used in any order and can be selected by simply clicking on the rubric title. Remember to click after rating for each rubric to ensure that your work is saved. You can easily tell if your work was saved by the symbol at the beginning of each rubric title. The indicator will appear for any of the rubrics you have viewed. The screen shot below shows the first three rubrics on the Tool’s cover page after a user has started rating: indicates Rubric I was viewed but either not yet rated or the rating was not saved indicates the rating for Rubric II was saved • NOTE: Rubric III in this view has not yet been viewed.

46. Slides developed by Achieve: www.achieve.org 1400 16th Street, NW / Suite 510 Washington, DC 20036