Evaluating Open Educational Resource (OER) Objects

1. Evaluating Open Educational Resource (OER) Objects Rubric IV: Quality of Assessments CC BY Achieve 2013

2. Open Educational Resources: [With Webster’s Definitions] O Open: Containing an open license - no restrictions on remixing or reusing [Having no enclosing or confining barrier; not restricted to a particular group or category of participants] E Educational: Used for teaching and learning [Pertaining to the action or process of educating or being educated] Resource: Object used to support an effort or task [A source of support or aid, especially one that can be readily drawn upon when needed] R

3. Open Educational Resources (OER) • What are OER? • OER are teaching, learning, and research resources that contain an open license. • They provide extraordinary opportunities for educators to freely share knowledge and resources. • They offer great potential for instructional innovation and networks for sharing best practices. • There are millions of OER objects available online.

4. Open Educational Resources (OER) • OER objects may include (but are not limited to): • Images • Applets • Games • Worksheets • Lesson plans • Original-source texts • Assessments • Units • Textbooks • Teacher/learner support materials • NOTE: Any smaller component of a more complex object, that can exist as a stand-alone, also qualifies as an “object” (e.g. a unit in a textbook, a lesson in a unit, or an activity in a lesson).

5. The Achieve OER Evaluation Rubrics • How and why were the Achieve OER Rubrics created? • Educators using the vast system of sharing need a method for filtering OER to meet the needs of their students. • The purpose of the rubrics is to provide a structure for evaluating an online resource in a systematic, purposeful and comprehensive way. • There are two ways of approaching the evaluation of a resource: • Holistically evaluating the object with a single rating, (similar to the star-rating systems used on Netflix, Yelp and Amazon) • Separately evaluating each of the components of quality, as with the Achieve OER Rubrics. • The rubric criteria are based on Achieve protocols used to assist states in alignment and quality review studies. • The Achieve OER Evaluation Tool, hosting the rubrics on OERCommons.org, allows a user to identify evaluate and sort objects based on the specified essential elements of quality.

6. Applying the Achieve OER Rubrics • How do the Achieve OER Rubrics work? • The rubrics represent an evaluation system for objects found within Open Educational Resources. • They are hosted as an online evaluation tool on the repository, OERCommons.org, but can also be used independently. • They are applied to any content area. • NOTE: At this stage only Common Core Standards for English Language Arts/Literacy and Mathematics are available in the Achieve OER Evaluation Tool. Content standards for History, Social Studies, Sciences, and technical subjects are not yet available. • Each rubric is applied independently to an object. • They rate the potential, not the actual, effectiveness of an object. • They use a five-point scoring system to describe levels of quality.

7. The Achieve OER Evaluation Tool Rubrics • This presentation will focus on Rubric IV: • Rubric I. Degree of Alignment to Standards • Rubric II. Quality of Explanation of the Subject Matter • Rubric III. Utility of Materials Designed to Support Teaching • Rubric IV. Quality of Assessment • Rubric V. Quality of Technological Interactivity • Rubric VI. Quality of Instructional Tasks and Practice Exercises • Rubric VII. Opportunities for Deeper Learning • NOTE: Sometimes one or more of the rubrics may not be relevant to a particular object. In those cases, a rating of N/A is appropriate.

8. Rubric IV: Quality of Assessments • How and Why is Rubric IV Applied? • Applied to objects which seek to determine what a student knows. • Assessments may be used to assess knowledge before, during, or after a topic is taught. • Applied to entire sets of assessment items when multiple items are included in an object. • Not applicable for an object not designed to have an assessment component, even when one might imagine aspects of the object that may be used for assessment purposes.

9. Rubric IV: Quality of Assessments • 3: An object is ratedsuperioronly if all of the following are true: • All of the skills and knowledge assessed align clearly to the content and performance expectations intended, as stated or implicit in the object. • Nothing is assessed that is not included in the scope of intended material unless it is differentiated as extension material. • The most important aspects of the expectations are targeted and are given appropriate weight/attention in the assessment. • The assessment modes used in the object, such as selected response, long and short constructed response, or group work require the student to demonstrate proficiency in the intended concept/skill. • The level of difficulty is a result of the complexity of the subject-area content and performance and of the degree of cognitive demand, rather than a result of unrelated issues (e.g. overly complex vocabulary used in math word problems). • For example a superior object for Rubric IV has appropriately focused and weighted assessments included, of varying types and modes, with accurate solution keys provided.

10. Rubric IV: Quality of Assessments 2: An object is ratedstrongif it assesses all of the content and performance expectations intended, but the assessment modes used do not consistently offer the student opportunities to demonstrate proficiency in the intended concept/skill. For example an object with an assessment component intended to determine a student’s ability to write a proof or present an argument uses only multiple choice or matching assessment items. This object is considered strong for Rubric IV because it has clear requirements that students know the structure and parts of the presentation of a proof or argument, but lacks an assessment mode that offers students an opportunity to fully demonstrate the intended skill/knowledge: writing a proof or argument.

11. Rubric IV: Quality of Assessments • 1: An object is ratedlimited if it assesses some of the content or performance expectations intended (as stated or implicit in the object), but omits some important content or performance expectations and/or fails to offer the student opportunities to demonstrate proficiency in the intended content/skills. • For example an object’s stated intention is to teach students to write fractions, with denominators of 2, 4, 5, 8, 10, and 100, as decimals, but assessment items address only measurements with the denominators 2, 4 and 8. The object is considered limitedin the information and performance expectations it assesses.

12. Rubric IV: Quality of Assessments • 0: An object is ratedvery weak or no value if assessments contain significant errors, do not assess important content/skills, are written in a way that is confusing to students, or are unsound for other reasons. • For example the following are considered very weak for Rubric IV: 1) an object that includes general suggestions for the type of assessments to be used (formative, summative, self-, pre-, etc.) but is missing specific questions or even question types; 2) one that has attached student assessments that address content not included in the lesson; or 3) one that includes some assessment items with a lesson, but with incorrect solutions. • NOTE: This list is not meant to be exhaustive. There are other possible examples of 0-rated objects for Rubric IV.

13. Rubric IV: Quality of Assessments N/A: This rubric is not applicable (N/A) for an object that is not designed to have an assessment component. Even if one might imagine ways an object could be used for assessment purposes, if it is not its intended use, not applicable is the appropriate score. For example an object is an interactive student-directed game that introduces a new concept. There is no intention by the designer to include assessments so Rubric IV is not applicable to the object.

14. Using the Achieve OER Evaluation Tool On OERCommons.org Rubric IV

15. Achieve OER Evaluation Toolon OERCommons.org • OERCommons.org hosts the rubrics as the Achieve OER Evaluation Tool • NOTE: Specific instructions for Using the OERCommons Website to search for and evaluate objects, using all of the rubrics, can be found in another slide presentation of this training package.

16. Rubric IV: Quality of Assessments Achieve OER Evaluation Tool Using OERCommons.org to evaluate an object for its quality of assessments: After you have evaluated the object using Rubrics I, II, and III, you are ready to apply Rubric IV. Begin by clicking on at the end of Rubric III or on the title line for Rubric IV.

17. Rubric IV: Quality of Assessments Achieve OER Evaluation Tool To view a video about Rubric IV click here…

18. Rubric IV: Quality of Assessments Achieve OER Evaluation Tool Hover over the rating numbers to see an abbreviated version of the rating descriptors. Review the object with your focus on the quality of the object’s assessments and select the appropriate rating.

19. Rubric IV: Quality of Assessments Achieve OER Evaluation Tool To download a full text version of the rubrics click here:

20. Rubric IV: Quality of Assessments Achieve OER Evaluation Tool • Across the bottom of Rubric IV you see options to: • Leave a [+ Comment], • Clear rating, • [Save & Go to the Next Rubric] (You can also click on any rubric title to leave Rubric IV, but your ratings will not be saved.)

21. Achieve OER Evaluation Toolon OERCommons.org: Rubric IV Now you are ready to rate the object using the Achieve OER Evaluation Tool’s other rubrics. Rubrics can be used in any order and can be selected by simply clicking on the rubric title. Remember to click after each rubric to ensure that your work is saved. You can tell if your work was saved by the symbol at the beginning of each rubric title: indicates not yet rated or rating was not saved indicates your rating has been saved

22. Slides developed by Achieve: www.achieve.org 1400 16th Street, NW / Suite 510 Washington, DC 20036