1 / 24

Educational Standard Assignment: Some Findings Working with CAT & SAT NSDL 2010 Annual Meeting

Ren é Reitsma 1 , Anne Diekema 2 Byron Marshall 1 , Trevor Chart 1 1 Oregon State University 2 Utah State University. Educational Standard Assignment: Some Findings Working with CAT & SAT NSDL 2010 Annual Meeting. Need for automated educational standard assignment in TeachEngineering.org.

mahala
Download Presentation

Educational Standard Assignment: Some Findings Working with CAT & SAT NSDL 2010 Annual Meeting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. René Reitsma1, Anne Diekema2 Byron Marshall1, Trevor Chart1 1Oregon State University 2Utah State University Educational Standard Assignment: Some Findings Working with CAT & SATNSDL 2010 Annual Meeting

  2. Need for automated educational standard assignment in TeachEngineering.org. Part 1: Comparative analysis of standard assignment by CAT and human catalogers (René & Anne). Part 2: What about standard crosswalking? Analysis of 4,790,801 Science SAT alignments (René, Byron and Trevor). Educational Standard Assignment: Some Findings Working with CAT & SAT… Overview

  3. www.teachengineering.org: 578 hands-on science and math K-12 activities. 339 lessons 54 multi-lesson, curricular units Explicitalignments: by author, supervised by collection catalogers: cover only one state  mean 4.5 stds./document. Similar coverage across all states: 917 * 4.5 * 50 = 200,000+ assignments. 200+ per document 917 * 4.5 * 10 = 40,000+ annual updates Automated Standard Alignment in TeachEngineering

  4. TE, ASN, CAT, TD, NSDL ‘Ecosystem’ BIG!! thank you to CNLP and friends for CAT. FYI, ‘new’ CAT (August 2010) is really fast  and includes ITEEA* & Common Core Math *Intern. Techn. & Engr. Educators Association

  5. 4,165 explicit alignmentsin TE 400,000+ (unsupervised) CAT assignments (science, math, ITEEA, common core math). Q-1: How are CAT assignments different from human (explicit) assignments? Q-2: Do the differences tell us something about how humans assign these standards in the first place? Q-3: Do the differences inform CAT and/or human improvements? BTW: What do we really mean when we say that a standard and a curricular item ‘align?’ (Reitsma, Marshall, Zarske (IPM – 2010)) Part 1: Content Assignment Tool (CAT) & Explicit Standard Assignment in TeachEngineering

  6. Approach: build networks of standards; layout the networks, interpret their spatial arrangements: Networks are based on how standards have been assigned to curriculum. Any two jointly assigned standards are considered ‘linked.’ Compare and contrast the networks for clues. Data: TeachEngineering collection – Jan. 2009. CAT & human standard assignments of CO 2007 Science standards. (Inductive) Method & Data

  7. CO 2007 Science Standard Assignments

  8. CO 2007 Science Standard Assignments... Cont.’d – CAT recall = 25 / 324 = .077* – CAT precision = 25 / 139 = .18* *if the humans did it right (?)

  9. ‘Curricular units’ – Human network is denser and more clustered. –Human clusters are curricular units – Human clusters link through common standards. – CAT: open structure; less clustering. Has no knowledge of curricular units.

  10. FR diagrams consider the network unweighted; i.e., all links have equal value/weight. Two weights: TF/IDF-like: weigh a standard link inversely proportional to the size of its company. ‘Fidelity:’ weigh a link between standards proportional to their mutual fidelity across the collection. Compute the KK network layouts Weighted or unweighted?

  11. Resulting KK diagrams showed essentially the same properties as the FR diagrams (hierarchical cluster analysis of two-dimensional positions)

  12. World standards (W): express facts and principles about the empirical world. E.g., S103EC87: Light and sound waves have distinct properties: frequency, wavelength and amplitude. Methodstandards (M!): express ways and means of conducting science. E.g., S103ECE9: A controlled experiment must have comparable results when repeated. Some method standards ‘contaminated’ with world terms and/or examples (M): E.g., S103ECD4: Technology is needed to explore space (for example: telescopes, spectroscopes, spacecraft, life support systems). Question: How do CAT and human catalogers compare on World vs. Method? CO Standards: ‘Method’ vs. ‘World’

  13. Standards: ‘Method’ vs. ‘World…’ Cont.’d

  14. W = world M! = (pure) method M = method with world examples – CAT under-assigned method. – Humans: method standards as curricular hubs – CAT central method hub: S103EC77: “physical properties of solids, liquids, gases and the plasma state and their changes can be explained using the particulate nature of matter model“

  15. Once again, thanks for CAT! TeachEngineering needs it. Tools such as CAT can benefit from contextual knowledge; e.g., that certain lessons are part of a larger set of lessons or a curricular unit. TeachEngineering curriculum is organized around both world and method standards. Hence, it would be nice if tools such as CAT become better at recognizing method standards. Contrast in standard re-use rate sends a signal to human catalogers not to be ‘complacent.’ Part 1: TeachEngineering & CAT Conclusions

  16. Standard crosswalking as a third source of standard alignment: Transitive logic: Learning object X aligns with standard P of state S Standard P of state S aligns with standard Q of state T Learning object X aligns with standard Q of state T CNLP’s Standard Alignment Tool (SAT) Send it an ASN PURL Send it the standard body to which to align Wait for the aligned standards Part 2: TeachEngineering & SAT

  17. Number of science standards (ASN leaves only): about 35,000. Number of authors: about 50 Mean number of standards per author: 700 Number of author combinations: 50(50 - 1) / 2 = 1,225 Total queries needed to collect a full set of SAT alignments: 700 * 1,125 = 787,500 Total required time: 787,500 * 5 seconds / 3600 seconds / 24 hours = 45 days of querying (assumes no down time). If instead, each of the authors is only aligned with one or more intermediaries, the total amount of querying per intermediary would be reduced to 50 * 700 = 35,000 queries. Total required time per intermediary: 35,000 * 5 seconds / 3600 seconds / 24 hours = 2.02 days. TeachEngineering & SAT Problem

  18. Aspect 1: How good are SAT alignments? Aspect 2: Assuming SAT alignments are good—whatever that really means—are the intermediary-based, transitive crosswalking alignments as good as the direct ones? Can we reliably use SAT for intermediary-based crosswalking? Test intermediaries: AAAS Project 2061 Science Benchmarks (AAAS) National Science Education Standards (NSES) Question: Does SAT-based Intermediary Crosswalking Work?

  19. Well respected; often (positively) referenced by states’ DOE standard documents. Why AAAS & NSES as intermediary?

  20. Aggregate Results (SAT alignment limit <= 5)

  21. Aggregate Results… Cont.’d

  22. How About Different States? Recall

  23. How About Different States?... Cont’d Precision

  24. Aggregate: …perhaps AAAS & NSES intermediary: AAAS U NSES recall ≈ 42%; precision ≈ 14% AAAS & NSES intermediary: AAAS ∩ NSES recall ≈ 14%; precision ≈ 43% Individual state: …perhaps Standards modeled to (one of the) intermediary; e.g., RI: recall ≈ 70%; precision ≈ 50% Size effects? Part 2: Does SAT-based Intermediary Crosswalking Work?

More Related