Workshop on Quality/Selectivity of the DLESE Collections

1. Workshop onQuality/Selectivity of the DLESE Collections • Framing the Question • History of the Discussion Kim Kastens, June 30, 2003

2. Framing the Question • DLESE has Broad Collection and a Reviewed Collection. We are (mostly) talking about the Broad Collection.

3. Framing the Question • Resources enter the DLESE Broad Collection via two routes: • Individually, via the DLESE Cataloging Tool (the “Community Collection”) • As part of an aggregated or themed collection, a collection accessioned into DLESE in its entirety. • We are concerned with quality and relevance of resources entering via both routes.

4. Framing the Question • Anyone can submit a resource to DLESE via the cataloging tool, which is an open set of web forms. • This has given rise to concerns that “junk” could get into the DLESE Collections.

5. Framing the Question • This workshop needs to make recommendations on two issues: • What should be the criteria by which resources are approved for inclusion in the DLESE Broad Collection? • What should be the procedures by which these criteria are implemented?

6. Framing the Question: Criteria • Resources submitted for the DLESE Broad collection currently must meet two criteria: • The resource is relevant to Earth System Education • The resource works (i.e. it has no conspicuous bugs).

7. Framing the Question: Criteria • Other possible criteria that have been suggested for the DLESE Broad Collection: • No cost or low cost for educational users • Resource is in English • No commercial message • No intrusive advertising • No blatant religious message • No blatant political message • No blatant errors of fact • Educational effectiveness • Well documented

8. Framing the Question: Procedures • With respect to procedures, we have two issues: • By what process shall we identify problematic resources? • What shall we do when we find a problematic resource?

9. Framing the Question: Procedures • By what process shall we identify problematic resources? • Ask the resource contributor (current system)? • Screening by the community? • Screening by paid staff?

11. Framing the Question: Procedures • What shall we do when we find a problematic resource? • Exclude it from the collection? • Include it in the Broad Collection but annotate it?

12. This is what an annotation might look like in the Discovery System:

13. Framing the Question: Procedures

14. History of the Discussion • Coolfont: August 1999 • Collections Policy drafted • There shall be a Reviewed Collection and an “Unreviewed” Collection • Collection Committee established

15. History of the Discussion • Coolfont: August 1999 (cont’d) • Rationale for Reviewed Collection: • Users’ Perspective: “…. recognized, efficient source for quality teaching and learning materials.” • Creators’ Perspective:“…. a recognized stamp of professional approval at the level of publication in a peer-reviewed journal.” • Rationale for the “Unreviewed” Collection: • “Users are seeking materials on a huge range of topics. The DL provides added value by being inclusive while providing powerful search and classification capability.”

16. History of the Discussion • Coolfont: August 1999 (cont’d) • Criteria for Reviewed Collection: • Accuracy, as evaluated by scientists • Importance/significance • Pedagogical effectiveness. • Well documented. • Ease of use for students and faculty • Inspirational or motivational for students • Robustness/sustainability

17. History of the Discussion • Coolfont: August 1999 (cont’d) • No Criteria established for “Unreviewed” Collection • After debate, it was decided that there would be a human-mediated step between submission of resource and ingestion into library.

18. History of the Discussion • Spring 2000: Academic Career Recognition Task Force Web Survey • Seven selection criteria for the Reviewed Collection met approval of prospective DLESE users, resource creators, and department Chairs.

19. History of the Discussion • Mid-late 2000: Collecting began • DPC: testbed collection for exercising metatdata framework • Montana State: Dave Mogk & students • Foothill College: Chris DiLeonardo & students

20. History of the Discussion • October 2000: Collections Meeting at Boulder: • DLESE Community Cataloger tool introduced to non-DPC collecting groups (AGI, Montana State, others?)

21. History of the Discussion • November 2000: Steering Committee Meeting at Lamont: • Contentious discussion about “filters” at the gateway to the Broad Collection • Agreement on only two of the discussed “filters”: (1) relevant to Earth System Education (2) “It works”, e.g. no conspicuous bugs • Contentious discussion of how to apply “filters”; clarity seemed to emerge when John Snow described a “holding tank” system used in his history group.

22. History of the Discussion • Nov-Dec 2000: Steering Committee Meeting at Lamont (cont’d): • Meeting Minutes: • “The general concept of a 30-day public comment period on new resources was agreed to. This will allow a time for the community to review resources….” • “In the short term, partners collecting resources …. will review them to make sure they are appropriate • “….the Collections Committee, collection proposal team and the DPC will work together to investigate mechanisms for encouraging review….”

23. History of the Discussion • February 2001 Collections Meeting: • Joint meeting of Collections Committee, “Collections Partners”, and Community Review System Editorial Review Board • DLESE Community Cataloging Tool open to the world • Collections Committee drafted Deaccession Policy

24. History of the Discussion • February 2001 Collections Meeting (cont’d): • Collections Committee discussed “filters” at gateway to Broad Collection. Imperfect consensus: • Relevance Filter • Is the resource relevant to Earth System Science education? • Integrity Filter • Are there no blatant errors of fact in the resource? • Are there no blatant political, religious, or commercial messages in the resource? • Does it function reasonably; i.e., seem to be basically bug-free?

25. History of the Discussion • April 2001 Steering Committee meeting at Biosphere 2: • Collections Committee/DPC Collections group presented fleshed out version of the “holding tank” or “provisional status” plan. • Many questions and issues. Who are reviewers? How mobilized and overseen? No $ to oversee the “army of filterers.” • No resolution.

26. History of the Discussion • July-Aug 2001 Steering Committee meeting at Flagstaff: • 850 resources in library. Metadata QA streamlined. • “Mike Mayhew indicated a concern …about the broad collection. ….Where is the quality control in developing the collection? Do we dilute the value of library with variable quality?” • Holding tank idea revisited, in simpler form without “designated reviewers” • Action item: “Boyd …. will develop a draft proposal/set of guidines to implement a holding tank in which resources are discoverable in the system and identified as accessioned within a 30-day period with some mechanism to accept comments. The proposal for implementation will not include a designated reviewer”

27. History of the Discussion • February 2002 Steering Committee meeting at Boulder: • Draft Collections Accession Policy presented • Revised throughout spring • DLESE oversight would be review of review process, rather than review of individual resources

28. History of the Discussion • July 2002 Steering Committee & Annual meeting at Cornell: • Deaccession Policy approved • Interim Collection Accession Policy approved • First annotation service demo’d within DLESE • Faulker reported that NSDL content philosophy was: “Educational value …to be manifest in capabilities for annotation and selective filtering, rather than an accession threshhold” • Possibility raised that annotation option might be solution to ongoing dilemma about quality of DLESE Broad Collection.

29. History of the Discussion • Fall 2002: • Sumner et al focus group study of Educators’ perceptions of Quality. • Best Practices for Resources summited to the DLESE Reviewed Collection begins to take shape.

30. History of the Discussion • Spring 2003: • Ad hoc Collections group met in Boulder, worked on how to implement Interim Collections Accession Policy and on Pathways to Reviewed Collection document • June 13: 12 collections met documentation requirements to be accessioned as collections.