Exploring Educational Standard Alignment: In Search of ‘Relevance’

1. René Reitsma*, Byron Marshall*Michael Dalton*, Martha Cyr *Oregon State University Worcester Polytechnic Institute Exploring Educational Standard Alignment:In Search of ‘Relevance’

2. Problem: aligning DL learning objects with educational standards Need & tantalizing promise National Science Digital Library (NSDL) efforts & accomplishments Early results show low Inter-Rater Reliability (IRR) Hypothesis: low IRR is partially a methodological artifact Proposal: multifactor concept of ‘alignment’ Experiment: 10-factor alignment model High IRR Four factor regression model (R=.75) of ‘overall’ alignment Exploring Educational Standard Alignment:In Search of ‘Relevance’

3. Aligning DL Learning Objects with Educational Standards • Expanding DL learning resource base; e.g., • National Science Digital Library (NSDL): 928 collections • K-12: TeachEngineering.org, TeachersDomain.org, Engineering is Elementary, etc. • NSF-GK-12 program (ongoing). • ≈84,500 math, science & technology standards (changing frequently)

4. Curriculum Standard Alignment Efforts • NSDL leadership: • Jes&Co: • Achievement Standards Network (ASN) • Center for Natural Language Processing (CNLP): • Curriculum Alignment Tool (CAT) • Standard Alignment Tool (SAT) • WGBH Teachers’ Domain: standard alignment & lexicon • Others: • Academic Benchmarks • AAAS/NSDL Strandmap server • Etc.

5. NSDL-based Curriculum Alignment Services

6. How Good are these Alignments?

7. Low Inter-Rater Reliability (IRR) • Devaul, H., Diekema, A.R., Ostwald, J. (2007) • Bar-Ilan, J. Keenoy, K., Yaari, E., Levene, M. (2007) “There is no average user, and even if the users have the same basic knowledge of a topic, they evaluate information in their own context…” • Hypothesis: Low IRR is partially a methodological artifact • Alignment is a multifactor, multidimensional concept • Learning objects may align with certain dimensions but not with others • Levins, R, Lewontin, R.C. (1980): “Abstraction becomes destructive when the abstract becomes reified… so that the abstract descriptions are taken for descriptions of the actual objects”

8. Dimensions of Alignment • Seracevic, T. (2007): ‘Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II: Nature and Manifestations of Relevance’

9. Hypotheses • H: One-dimensional alignment/relevance IRR is partially a methodological artifact. • H-1: At least some dimensional IRRs will be high(er) • H-2: Dimensional IRR will vary • H-3: ‘Overall alignment/relevance’ IRR will be low, even when asked in the context of dimensional relevance testing.

10. Experiment

11. Experiment Cont.’d • 14 Subjects all familiar with the TeachEngineering system • Two teaching tasks: • “As a third grade Massachusetts teacher you are assigned to teach material related to the standard “Relate earthquakes, volcanic activity, mountain building, and tectonic uplift to plate movements.” You have two hours of class time to spend on instruction.” • Judge the alignment of three curricular objects (R-1 – R-10, six-point Likert scale)

12. Results • 91 IRR comparisons × 10 alignment dimensions × 6 alignments • H-1: IRRs are relatively high; IRR-1 (binary): 64%-95% • H-2: IRR variability • H-3: Overall relevance (R-10) among the weaker ones

13. How about Overall Relevance (R-10)?

14. MLR Model of Overall Relevance (R-10) R2 = .75 • ‘Overall alignment/relevance’ is meaningful as a complex variable. • Some high IRR alignment dimensions do not contribute to overall alignment.

15. Conclusion • K-12 educational DL content is expanding; educational standard alignment is needed. • Innovative and promising resources are available but reported IRR of assessment of alignments is low. • Propose that ‘Alignment’ is a complex concept: • Recognize alignment dimensions • Experiment suggests that dimension-specific IRR will be (much) higher • ‘Overall’ alignment has a very specific interpretation. • What do we need: • Continued assessment and IRR collection • Collections making their assessment data available. • Alignment methods that can assimilate ‘evidence’ from the multiple dimensions that comprise ‘alignment’ of a learning resource with a teaching standard.