2003 Annual NSDL PI Survey A Collaborative Formative Evaluation

1. 2003 Annual NSDL PI SurveyA Collaborative Formative Evaluation

2. Purpose of Presentation • Describe the community-supported formative evaluation efforts to evaluate the NSDL’s distributed library and community organizational processes • Elicit feedback on the results and recommendations from the 2003 Annual NSDL PI Survey

3. Timeline of the Evaluation Effort • Dec. 2001: Evaluation Working Group Meeting articulated four questions to be addressed in a pilot study • February 2002: Evaluation Workshop developed a work plan to answer the four questions. An online survey would be used to identify: How are the distributed library building and community governance processes working? • Feb.-July 2002: Survey developed and delivered. Preliminary survey data reported at JCDL. • Dec. 2002: Final survey data presented to the community during the EIEC meeting. Committee agreed that an NSDL-wide implementation of the survey should be undertaken • Apr.-June 2003: Survey revised, tested for validity and usability • June-July 2003: Survey delivered • Oct. 2003: Survey analyzed and presented at NSDL Annual Meeting

4. Survey Details • Primary Question: How are the distributed library building and community governance processes working? • Developed measures in four areas to answer this question: • NSDL communications infrastructure • Extent and success of NSDL projects’ collaboration • Projects’ participation level in NSDL organizational structures • Information needs and motivations factors for NSDL community • Note: The questions and scales have evolved with each iteration, but the four areas being evaluated have remained the same. • Questions sought to identify: • Levels of use, effectiveness, and extent of participation • Information needs and motivations • Barriers to use, effectiveness, participation and suggestions for improvement • Types of scales: Likert, multiple choice, open-ended

5. Survey Details (continued) • Primary audience: NSDL PIs (though all project members were encouraged to contribute to survey completion) • Distribution: survey available online four weeks during June-July 2003; notification and reminders sent to All-Projects and PI listservs • Verification: one survey per project was included in final analysis; used NSF award number to verify results, then disassociated the award number from further analysis Examples of survey questions->

6. Summary of Key Findings from the Four Areas • 1. Communication Infrastructure: The NSDL Annual Meeting is still the best place for projects to establish communication and identify potential collaborators. Other communication tools are perceived as not supporting projects accomplishing their goals. • 2. Collaboration: 80% of respondents indicated that they had collaborated with a median of 2 other NSDL projects since the beginning of 2003. 58% of respondents incorporated something they learned through a collaborative relationship into their project’s design and use. Respondents overwhelmingly identified lack of time and being unclear about other projects’ goals and activities as barriers to further collaboration. • 2. Organizational Structures: A majority of the respondents had not interacted with NSDL organizational structures since the beginning of 2003. 36% of respondents had a mid-level rating on the overall perceived effectiveness of NSDL groups. “I'm sure all this activity is necessary but it hasn't trickled down to the individual projects.” • 4.a. Information Needs: A majority of all respondents were highly aware of the standards and technologies needed to integrate projects into NSDL. 62% of respondents indicated that they had implemented some part of their project’s evaluation plan in the past year. 100% of respondents reported that they had disseminated information about their projects through an average of 8 publications or presentation from June 2002 to June 2003. • 4.b. Motivation: 85% of respondents indicated that it was important to participate in NSDL communication, organizational groups and library building activities because they wanted to help establish a National Science Digital Library

7. Results, Recommendations & Questions • Individual projects are not aware of whose expectations they should meet, what level of participation is expected or what the outcomes of participation should be when using the NSDL communications infrastructure, pursing collaborative opportunities and participating in NSDL organizational structures. • Recommendation: Establish a process for defining and communicating shared priorities and expectations. • Recommendation: Provide targeted “entry points” for project or individual participation • The motivation to expend extra time and resources on participation, collaboration and communication is often redirected by deadlines imposed by a two-year funding cycle. • Recommendation: Identify tangible benefits for projects to participate, aside from building NSDL. • Question: Is it necessary that every project participate in order for NSDL to be built?

8. Results, Recommendations & Questions • Respondents identified what type of support they would like from the Core Integration; however, this could be a moving target because projects are at different stages of development. • Recommendation: Identify specifically what projects need in terms of technical support. • Recommendation: Define and communicate what levels of support can be realistically expected from the Core Integration team. • Recommendation: Leverage the expertise of NSDL projects. • Respondents indicated that they were unsure about how collaboration was being defined and what were the expected results of collaborative efforts. • Recommendation: Establish a process to define and measure collaboration in a distributed environment. • Question: How much emphasis should be placed on collaborative achievements as factor in the success of NSDL?

9. Results, Recommendations & Questions • Projects are not sure how to identify others working on similar topics, outside the NSDL Annual Meeting. They expressed the need for a “human” connector to point them to the right collaborator; and, the need for direction about how to be involved in organizational structures. • Question: How could the current NSDL organizational structure adapt to meet the demand for connections on a more granular level than now exists? • The current communications infrastructure is not meeting projects’ information needs and the multitude of communications channels present too many choices when deciding what to read regularly and where to look for information to support project work. • Recommendation: Establish one resource that is continually maintained and updated with basic technical information and other information relevant to project development.

10. Design & Delivery Issues • Audience: • Pilot - everyone working on a project – result was conflicting responses from one project. • 2003 - one response per project, though everyone working 20 hrs/wk for past 3 mos. could contribute opinions. This is easier to validate, verify and analyze, but are we missing out? • Moving target topics… • Ur-question remains the same: How are the distributed library building and community governance processes working? • four issues to evaluate remain the same: NSDL communications infrastructure; the extent and success of NSDL projects’ collaboration; the projects’ level of participation in the NSDL organizational structures; and, the information needs and motivations of the community. • definitions, issues, tools have changed (i.e., governance -> organizational structure; CommPortal ->re-designed)

11. Design and Delivery Issues (continued) • Reliability, Validity, Sampling • Pilot - Didn’t do. • 2003 - extensive review and comments from EIE Committee members and the Core Integration Team; expert review and validity testing with NSDL PIs; lastly,think-aloud usability testing • Distribution: • Pilot – used proprietary software for distribution and analysis. • 2003 – CI (Casey & Alex) hand built a form. There has to be a better way. • Timing • Distribute when the maximum number of respondents are near their computers; both the Pilot and 2003 surveys were delivered in early summer • Coordinate with other NSDL-wide data-gathering efforts • Re-Use (of instrument and of data) • Design so the same basic instrument can be used regularly – cost and time savings • Pilot: sparked the idea of creating an “Evaluation” category for the Collaboration Finder and for creating the NSDL Progress Report • 2003: Provided data for the NSDL Progress Report

12. Process Issues & Questions • Clarify the purposes and timing of the Annual PI Survey relative to other data-gathering activities occurring across NSDL: 1) surveys from other standing committees; 2) the Collaboration Finder; and, 3) NSF reporting requirements. • The Annual PI Survey has functioned well for gathering data about perceptions and beliefs about organizational aspects of NSDL. Does it function well as a tool for identifying projects’ immediate technical or informational needs? • The results from this Annual PI Survey were used in the NSDL Progress Report. Re-purposing survey data reflects a larger organizational need for both formative evaluation and for data to support outreach efforts. How should the Annual PI Survey respond to shifting NSDL organizational needs and goals?