Dr. Julia Melkers, Associate Professor of Public Policy Georgia Institute of Technology

1. Evaluating the Alaska EPSCoR Phase III:Resilience and Vulnerability in a Rapidly Changing North: The Integration of Physical, Biological and Social Processes Dr. Julia Melkers, Associate Professor of Public Policy Georgia Institute of Technology Dr. Eric Welch, Associate Professor of Public Administration University of Illinois at Chicago Year 3 Evaluation Presentation Alaska EPSCoR All Hands Meeting May 2010 Fairbanks, Alaska

2. Stating the Obvious • Collaboration is difficult, problems of: • disciplinary language, • distances, • funding, • methodological, • conceptual and theoretical approaches • lack of clarity about why research questions are “important”. • Requires commitment to learn from other fields • Variation in goals and interests of stakeholders and rural campuses • Institutional barriers and facilitators are key to success

3. Three Year Evaluation of Alaska EPSCoR Phase III • A formative and interactive approach. • Can also provide lessons learned for future EPSCoR proposals and initiatives. • An summative evaluation process to serve as an accountability mechanism. • A research driven evaluation, based on studies of science, collaboration, networks, among others.

4. Evaluation Questions

6. Social Networks & Evaluation • Social network analysis can add to the program evaluation methodological toolbox. • Why use SNA in program evaluation? • Allows us to capture knowledge development, social and human capital, and other interim outcomes that cannot be captured in other data. • Adds rigor to attitudinal and self reported data regarding behavior, outputs, outcomes, and relationships. • Adds detailed dyadic data on specific relationships and exchange. • Can be combined with other data to provide a comprehensive picture. Data may be drawn from surveys, interviews and existing documentation.

7. Some Network Characteristics • Network Size • Number of nodes that are connected to others • Number of Ties • Number of ties that link these nodes individuals can have multiple ties) • Network Density • measures the percentage of ties that exist compared the number of possible ties. • This measure provides one way of describing the extent to which potential ties are unexploited.

8. Some Network Characteristics • Average Degree Centrality • measures the average number of immediate connections that each individual has in the network. • Provides a way to examine the level of participation in network activity by the ‘average’ person in the network.

9. Some Network Characteristics • (E-I) Index • the extent to which the network is made up of individuals outside as compared to inside a particular environment or context. • The EI index is calculated as: (external ties – internal ties) / (external ties + internal ties) and ranges from negative one to one.

10. Our Final Report • Today’s Presentation • Collaborative interactions – Year 3 and Overtime • Integration with rural campuses and communities • Student Impacts • Observations and some lessons learned • Written Report • Collaborative Interactions and Outcomes • Student benefits and learning • Integration with native communities • Rural Campus issues • Attitudes and feedback regarding EPSCoR issues • Lessons Learned • Recommendations for future EPSCoR efforts

11. Overview of Survey Respondents • Year 1 • Faculty: 86% response rate (n=59) • Students: 85% response rate (n=57) • Year 2: • Faculty: 75% response rate (n=79) • Students: 72% response rate (n=48) • Year 3 • Faculty: 75% response rate (n=111) • Students: 57% response rate (n=52)

12. Year 3 Faculty/Researcher Respondents (n=111) Rank: Assistant Professor = 35% Associate Professor = 26% Full Professor = 21% Research faculty & other = 24%

13. Year 3 Student Respondents (n=52) • 84% respondents at UAF

14. Comparing Developments Across Time • Useful to track the EPSCoR community • General observations on community of researchers. • More meaningful to track core group of individuals who have been involved over time. • Longitudinal Faculty Comparison Groups • Survey Respondents from Years 1 & 3 (n=44) • Year 1 respondents and “active” individuals who responded to Year 3 survey (n=65) (59% of Year 3 respondents) • Changes in Year 2 & 3

15. Collaboration & Integration

16. Sole Authoring by Component Y3 (mean number of faculty reporting these products)

17. Sole Authoring by Institution Y3(mean number of faculty reporting these products)

18. What became of Desired Collaboration Year 1? Who would you like to collaborate with, but have not done so to date?

19. Reasons for Successful Collaborations • Sharing common research interests in permafrost related arctic engineering and science issues. • Common interest in phylogenetic methodology, especially methods of DNA sequence alignment. • We are both interested in future trends in moose and caribou populations. • We are both interested in developing programs that link arts with the sciences. • Collaborative style and our complementarity • [That person’s] energy and enthusiasm for doing challenging empirical research.

20. Production: Journal Articles Red=Biological Science Blue= Social Science Green=Physical Science Square= Year 1 & 3 participant

21. Impacts on Communication Red=Biological Science Blue= Social Science Green=Physical Science Square= Year 1 & 3 participant

22. Impacts on Research Red=Biological Science Blue= Social Science Green=Physical Science Square= Year 1 & 3 participant

23. Growing Collaboration Around Key Research Themes Red=Biological Science Blue= Social Science Green=Physical Science Square= Year 1 & 3 participant

24. Coalescing of Research Themes Change in the Number of Collaborative Ties over Time

25. Do Networks Matter for Continued Activity? • Dependent Variable – Indicator of Continuation • Have you developed any new ideas or plans for research as a result of your interaction with EPSCoR Phase III? (1/0) • Independent Variables • Familiarity “I understand this person’s knowledge and research skills. This does not necessarily mean that I have these research skills or am knowledgeable in these domains, but that I understand the skills this person has and domains in which they are knowledgeable.” (Strongly Disagree to Strongly Agree) • Size of Collaboration Network: Since July 2007, with which of the following EPSCoR faculty members did you work with on a journal article. (1/0) • Other controls: Tenured, Biology, UAF

26. Example Findings • Have you developed any new ideas or plans for research as a result of your interaction with EPSCoR Phase III? (1/0) *= p<0.10, ** = p<0.05)

27. Integration with Rural campuses and Communities

28. Stakeholder Interaction • Faculty report limited importance of interaction with stakeholders. • Faculty report few meetings with stakeholders over the course of EPSCoR Phase III (1= never, 2= once, 3= 2 to 3 times, 4= more than three times)

29. Research in Native Communities • Native organizations and local contacts critical. • Provide access • Experiences with researchers are varied • EPSCoR can facilitate the capacity of faculty to interact with native communities • Nature of outreach matters • Input / agreement of community • Follow-up is key • Relevant research has implications for attraction and retention in STEM

30. Rural Campuses • Play key role as liaison to native communities • Faculty are important nodes for connection • Student are important nodes for connection • Serve multiple roles • Skills and training • Research for understanding local context and for ensuring community security • Capacity building in community • EPSCoR is an important supporter of research dissemination through Western Alaska Interdisciplinary Science Conference (WAISC).

31. Student Impacts

32. EPSCoR Faculty and Students • 28% of faculty report that they are currently working with an EPSCoR supported student. • 50% of those report that the research would not have been conducted without EPSCoRfunding.

33. Students Report Positive Impacts:Research Interests • 75% of student respondents are required to do a thesis. • Of these: • 65% report that their thesis or dissertation includes some aspects of climate change research. • 55% report that their thesis or dissertation topic “has been influenced by their EPSCoR work.”

34. To what extent has your Alaska EPSCoR experience helped you to develop skills and knowledge in the following areas?(1= not at all, 2= some, 3= a great deal)

35. PrOSPECTS FOR THE FUTURE

36. Faculty Collaboration Interests • Generally positive attitudes across institutions regarding collaboration. (4 point scale, 1=strongly disagree)

37. Faculty Collaboration Interests • Faculty component-based responses show similar variation in interest in interacting with faculty and students in other disciplines.

38. Production: Grant Proposals to Extend Phase III Research Red=Biological Science Blue= Social Science Green=Physical Science Square= Year 1 & 3 participant

39. Capacity Development in EPSCoR Phase III • Research: • Evidence that collaborative networks have increased in size and diversity over Phase III. • Research theme-based networks are gaining more research ties across components. • Production and grant activities to sustain the capacity developed through Phase III are evident. • Faculty report positive impacts of EPSCoR engagement: • Production • Connections • Research ideas • Changes in research processes

40. Capacity Development in EPSCoR Phase III • Education : • Students report positive impacts of EPSCoR Phase III • Level of student research interests in climate change, as evidenced by thesis and dissertation topics, indicates potential capacity impacts on new generation of climate change researchers. • Strong student attribution of research skill development to EPSCoR Phase III, particularly among undergraduates.

41. Capacity Development in EPSCoR Phase III • Rural Campuses and Communities: • Significant interest in developing stronger ties to bring research and education opportunities and resources to rural campuses. • Researchers in the larger campuses express concern about “access to” rural communities. • Increasing exposure of climate change research in rural communities may attract and retain students there. • There are key individuals in rural campuses and communities that can help catalyze productive interaction.

42. Structural and Other Issues Relevant to Future EPSCoR • Importance of support of EPSCoR at all levels of administration and management. • In smaller institutions it is not always obvious how to best benefit from EPSCoR. • Administrative cultures constitute barriers to effective participation in and management of EPSCoR. • Ambitious efforts to cross disciplines and institutions require strong institutional support. • Student network development across institutions can also assist in this.

43. Questions and Comments