Learning about Learning Themes for GROOVE Final Impact Assessment

1. Learning about Learning Themes for GROOVE Final Impact Assessment

2. Steps in Process for Capturing Learning about Learning Data • Define what “Success” looks like for learning networks. Vet with USAID to affirm how we ascribe “success.” • Review “Findings of Learning Network Review” document and Rwanda Meeting “Initial Thoughts” brainstorming to pull out relevant themes for Learning about Learning • Vet “Success” definition and learning about learning themes with GROOVE members to validate relevance and “capturability”, i.e. we capture stories/anecdotal evidence of the validity of the themes through the interviews, survey and AARs. • Vet Learning Product format for Learning about Learning with GROOVE members, i.e. Case study with “caselettes” • Identify GROOVE members who want to work on this topic and/or what role each org will play in the Learning about Learning • Integrate final themes into Interview Guide, Survey and AAR format

3. Definition of “Success” (SCOTT: I have put some thoughts on paper here about what a successful learning network is. Please edit/add/delete/rewrite/affirm as you see necessary. We will vet this with USAID once we are in agreement. Thanks.) As successful learning network: • Leverages subject matter curiosity of its members and existing learning aspirations/organizational change agendas of member organizations for a greater change and/or discovery purpose (e.g. sector wide or industry wide contribution) • Creates a platform for members and the organizations they represent to explore and resolve questions they are already working on and are committed to resolving or furthering the discourse on. • Produces knowledge products that are audience specific, tailored and timely. • Opens communication channels among peers of different organizations (e.g. at appropriate levels within each organization and among comparable subject matter experts) for ongoing learning and exchange.

4. Model for LN “Culture” SCOTT: In conversations with Lane, she has mentioned an interest in the issue of “culture “ of the GROOVE and the role it plays in making a LN successful. Anya pulled some examples of trying to capture/define the elements of culture so we could then look at the culture of GROOVE and how it has helped learning. Below is a list of conditions which facilitate effective collaboration or co-operation, in public, private or mixed sector contexts. It has been compiled through a review of I/O psych literature. Research suggests that when the following factors are present, successful collaboration is more likely to be achieved: • Participants involved share a common: * sense of mission and strategy; * set of values; * ability to manage change; • Participants involved share: * power among those involved; * decisions about how to manage the collaboration; * the resources themselves; • Participants involved agree over: * the legitimacy of participants to be involved in the collaboration including the convener; * perceived stakeholder inter-dependence; * the values of collaboration per se; * the importance of the issue over which collaboration is to occur (or, if not, find an inducement for others to collaborate); • The organizations involved reflect:* through their different roles and values, the complexity of the issue; the participants are geographically proximate;  • There is supportive communication and evocative leadership to promote: * good interpersonal relationships between individuals involved; * high awareness of each organization's goals, services and resources; * mutual trust; there is an external mandate for collaboration. These factors are present in the GROOVE culture. Several Brainstorming thoughts from Rwanda affirm these points as shown on the next slide.

5. GROOVE Culture related “Initial Thoughts” on Learning from Rwanda Meeting SCOTT: Here I have aligned some of the “Initial Thoughts” from the Rwanda discussion on learning about learning with the factors of culture to demonstrate how those factors are present in the GROOVE A. Participants involved share a common: * sense of mission and strategy; * set of values; * ability to manage change; • From the very beginning, it was clear that we had the “right” people in GROOVE. Everyone came in wanting to translate theory into action. This was an interest of each organization, and each organization was looking for ways to accelerate that. • Everyone walked into the room the first day with a bias for sharing. Commitment and belief in learning. Wanted to learn from each other. B. Participants involved share: * power among those involved; * decisions about how to manage the collaboration; * the resources themselves; • At a certain point, realized that there was an expectation of us as a learning network beyond the work that we would do in our own organizations. That turned out to be okay. • “Let the user complete the product” e.g., how Tracy handled ownership issues for the mentor program – it was made clear that this was a GROOVE product, not CARE’s. C. Participants involved agree over: * the legitimacy of participants to be involved in the collaboration including the convener; * perceived stakeholder inter-dependence; * the values of collaboration per se; * the importance of the issue over which collaboration is to occur (or, if not, find an inducement for others to collaborate); • When our organizational leadership questions why the network is important, we’ve taken the time to help one another find messages. • The fact that USAID is behind this effort and that it is valued by AID is seen as important by our leadership. At the same time, AID leadership sees value in the network and the involvement of AID staff because of the organizations who are a part of GROOVE. • We made clear the rules on acknowledgement/recognition of contribution. • We were given the opportunity to engage by making small commitments which grew. D. The organizations involved reflect:* through their different roles and values, the complexity of the issue; the participants are geographically proximate;  • Virtual interaction is important, but really appreciate the value of substantial, focused time to crystallize ideas. • The diversity of organizations helped us avoid group think. • Network allows us step back, look at issues more broadly, focus on bigger issues. Supports this kind of thoughtfulness in the midst of daily demands. E. There is supportive communication and evocative leadership to promote: * good interpersonal relationships between individuals involved; * high awareness of each organization's goals, services and resources; * mutual trust; there is an external mandate for collaboration. • Our relationship with our own managers and how we engaged with them made a difference in our own ability to participate • There already were some positive personal connections, shared experiences. • Sometimes there was a need to formalize the relationship (especially when we consider the needs of our individual organizations). The trust was there before we had to do that. • .

6. 3. Summary of Review:Critical Aspects of LN and Findings These are the Critical Aspects of LN that were identified in the original LN review. I have highlighted in PURPLE the aspects that contained themes that I thought most relevant to Learning about Learning and what we can provide USAID feedback on to validate or refute to help raise the chances for LN success. Under each Aspect I have also include bullets in GREEN from the Rwanda Meeting discussion on Learning about Learning. These statements show insights we have gleaned from our experience under GROOVE. The insights may align, and/or expand the Aspect. Integration of Learning into the Grant Program Management of Grants and LN Role of Donor Budgeting for Learning and LN Activities Selection of Grantees Technology and Platform for Sharing/Collaboration Facilitation Planning and Dissemination of K-Products Development of Key Performance Indicators Post-LN Strategy

7. 3. Summary of Review:Critical Aspects of LN and Findings Integration of Learning into the Grant Program Findings: • Conceptually, all grants had learning as a focus, following the LN model. • In practice, the extent to which learning was integrated into grant objectives, budgets and activities was very uneven and had direct impact on performance of LN. • LN under PLP (and GUC) overall more successful than IGP because: • Learning was central focus, i.e. core activities, of program (based on robust LN model) • Learning clearly emphasized at RFA/RFP level (as objectives, required activities) • Funds used for implementation at the individual project level had a strong learning focus that benefited grantees individually and also fed into the group learning SCOTT: I would posit that we look at the how the Learning Focus went beyond the parameters of the grant and was actually part of what each organization was attempting to do already. The grant supplemented existing aspirations of the organization and provided a critical boost in status, breadth and scale of the institutional change each organization was trying to achieve. The theme here would be Integration of Learning Grant into Organization’s Change/Learning Agenda. GROOVE Rwanda Discussion Thoughts: • Network allows us step back, look at issues more broadly, focus on bigger issues. Supports this kind of thoughtfulness in the midst of daily demands. • The fact that USAID is behind this effort and that it is valued by AID is seen as important by our leadership. At the same time, AID leadership sees value in the network and the involvement of AID staff because of the organizations who are a part of GROOVE. QUESTION: What was a peak experience in how this grant impacted learning within your organization? What was the most important organizational change the grant helped bring about?

8. 3. Summary of Review:Critical Aspects of LN and Findings Integration of Learning into the Grant Program (cont’d) Findings: • LN under IGP were more problematic because: • Program implemented more as grant with LN component “added-on” rather than integrated as learning program. • Grantees focused on implementing individual projects rather than learning at group level or capturing the learning throughout implementation process. • Weak integration of learning into core objectives and requirements of grants made it difficult for facilitator to keep members engaged and see the LN as priority.

9. 3. Summary of Review:Critical Aspects of LN and Findings 2. Management of the Grants and LN Findings: • Management of grants and LN had direct impact on performance of LN. • If management of grant and LN consolidated under one focal point (PLP), LN more successful because: • Roles and responsibilities clearer • Fewer coordination and communication challenges between manager & donor • Allows for better influence and implementation control over activities and results

10. 3. Summary of Review:Critical Aspects of LN and Findings 2. Management of the Grants and LN Findings: • On the other hand, if management of grant and LN split into two separate functions overseen by two separate focal points in different organizations (IGP), LN were more problematic because: • Coordination and communication more challenging • Donor led RFA/RFP process so LN manager had no input on selection of grantees • LN did not manage grants so could not influence amount allocated to learning • LN manager only responsible for LN component and budget so had little leverage GROOVE Rwanda Discussion Thoughts: • GROOVE is a small amount of money squared. Every dollar invested in the individuals sitting around the table gets multiplied across their organization (SCOTT: This is no small point. Any way to quantify this in addition to the cost share we all have accumulated?) • Belief that this will allow “me” to do my job better. Don’t have this opportunity in my own organization to share these kinds of ideas. • Periodicity of learning journals and the kinds of questions that were asked was helpful along with the social pressure of showing that things were moving forward. • INHIBOTOR: The network should pay earlier attention (waiting 15 months for the first face to face meeting after the launch was too long) to: 1)Talking about how best to work with AID – to get clarity around expectations as needed; 2) Building collective awareness of group dynamics and individual roles in dealing with that. 3) Build in resources ahead of time to support meetings like this one, support for facilitator, etc. • INHIBITOR: All the channels of communication – between and among KDMD, the GROOVE, the facilitator, AID need to be defined and managed. SCOTT: I did not think we had much to add here except comment on how we think the management of our grants went.

11. 3. Summary of Review:Critical Aspects of LN and Findings 3. Role of the Donor (USAID) Findings: • Donor participated in all LN (both in person and on-line). • Effect of donor participation differed across LN, depending on who individual was but positive on the whole. • Even split between those who felt donor should or shouldn’t be visibly present online or at in-person event. • Those against donor participation felt their presence at in-person events made participants feel intimidated and shy about being candid because afraid would be judged by donor—affect ability to get future grants.

12. 3. Summary of Review:Critical Aspects of LN and Findings 3. Role of the Donor (USAID)—cont’d Findings: • Many felt role of donor should be the following: • Focus on effective integration of learning into grant program at RFA/RFP level; • Focus on ensuring management of grant and LN consolidated and streamlined; • Help identify industry gaps and types of products needed; • Review drafts of products for appropriate focus and content; • Serve as technical sounding board ONLY when needed; • Attend ONLY select sessions at in-person events as technical advisor/resource person; • Help connect to relevant networks and other work of USAID; • Help to disseminate and promote group’s work.

13. 3. Summary of Review:Critical Aspects of LN and Findings 3. Role of the Donor (USAID)—cont’d Findings: • Many felt that if donors want to participate, they should: • Refrain from making critical remarks at in-person events or online; • Refrain from judging or scrutinizing the group’s ideas or work in person or online; • Refrain from communicating directly with grantees—better to communicate through facilitator; • At in-person event, integrate with the group (sit at same table, be low key, and not dominate airtime); • Support the group’s agenda rather than drive it. (GROOVE Rwanda Discussion: • The flexibility over the life of the network provided by the framework and the donor to deal with things when we’re ready. • Initial engagement with AID was relaxed, engaged, experimental. Pushed here and there, but not directive. • The way AID facilitated helped us be successful – supported the idea of a learning organization. • The fact that USAID is behind this effort and that it is valued by AID is seen as important by our leadership. At the same time, AID leadership sees value in the network and the involvement of AID staff because of the organizations who are a part of GROOVE. • At a certain point, realized that there was an expectation of us as a learning network beyond the work that we would do in our own organizations. That turned out to be okay. • Being able to access KDMD resources, to be able to engage and imagine possibilities. SCOTT: Here I think we could validate the insights about donor involvement as listed here. I think that we have had effective involvement of USAID in the workings of GROOVE, that is, there has been space given for members to work yet. USAID/QED has been available for consultation and guidance at any tme. QUESTION: What was a peak experience in interacting with USAID under this grant? Did it conttribute to your learning? If yes, how? What was a “valley” experience in interacting with USAID under this grant? Why? How did it impact learning?

14. 3. Summary of Review:Critical Aspects of LN and Findings 4. Budgeting for Learning • Findings: • In most LN, LOE and resources for k-products seriously underestimated. Result: scaled down effort or additional burdens on facilitators. • Contingency planning and budgeting was absent in all LN--problematic in cases where LN members not at the same technical level. No resources to hire required expert. • One of grant budgets slashed midstream. Result: immediate diversion of resources from learning to project implementation activities. Participants resisted producing case studies to capture learning. • SCOTT: I think that this is a very critical issue. It has been challenging to estimate the cost of learning products and the time /LOE it takes to produce them. Also, knowledge products evolve over the course of the project and as a result expand and contract (but more consistently expand!) in level of effort. It might be very helpful if we can provide insights on how to more effectively budget for participating in a learning network and to ensure learning products of consequence. • QUESTION: If you had more money under grant, what would you have done differently? Less money, what you absolutely retain/do again? What was useful in how you structured your grant? What is not useful?

15. 3. Summary of Review:Critical Aspects of LN and Findings 5. Selection of Grantees Findings: • Selection process needs to include both grant and LN managers OR clear selection criteria to ensure selection of real cohort. • LN without real cohort did not function as well as those with peers at same level (expertise, org capacity) . • Most PLP LN had clear cohorts where members shared more in common than differences and willing to learn because: • Amount of grant too small to attract those not really interested in learning • Tight focus of grant coupled with language in RFA/RFP (on learning) helped to attract grantees with similar areas of interest or work • Selection made by grant and LN manager (same organization) • Larger number of grantees than IGP • Selection process under IGP handled by donor with no input from LN manager. Result: grantees with little in common (weak cohort), in turn impact performance. GROOVE Rwanda Discussion Thoughts: • From the very beginning, it was clear that we had the “right” people in GROOVE. Everyone came in wanting to translate theory into action. This was an interest of each organization, and each organization was looking for ways to accelerate that. • The diversity of organizations helped us avoid group think. • Everyone walked into the room the first day with a bias for sharing. Commitment and belief in learning. Wanted to learn from each other. • There already were some positive personal connections, shared experiences. • Recognition of the other person was always there, e.g., Lucho and MaFi and what that has brought. The different perspective that CI brings. • Our relationship with our own managers and how we engaged with them made a difference in our own ability to participate. • Believing that my shared ideas will influence other GROOVErs. SCOTT: Select of Grantees – The selection of grantees seems to have worked for the GROOVE. We may have something to say about this topic but it would also have to include QED/USAID input because they ultimately chose the grantees. So interviewing/AAR may have to be expanded to QED/USAID. I think what is missing here is how the selection process fed into the culture of the GROOVE and how that culture then contributed to learning. QUESTION: What were the top three criteria you took into considertion when selection the grantees? Do they still hold true, that is, do you still think they are the most important three criteria? Were you accurate in your assessment of each organization and how they met the criteria? How would you change them?

16. 3. Summary of Review:Critical Aspects of LN and Findings 6. Technology and Platform Findings: • For PLP, technology solution provided by same organization responsible for managing all aspects of the grant and LN. Site already familiar to most grantees who are already SEEP members. • For IGP, grantees had to access and sign in on two platforms (SEEP and microLINKS) for different info needs. Created sense of competition between the two sites as well as frustration among grantees who prefer the one stop shop approach. • Technology can hinder activity of LN if members can’t access forum. Need to fully test any platform before rolling out (example of global value chain LN). • Most field practitioners preferred email listservs over online forums but do want online space to hold and access useful resources. • Use of Skype and webinars to connect grantees resulted in more engagement. SCOTT: I did not think we had much to add to these points on technology and how it contributed to learning beyond what these already say. That said we could simply validate what is here through the question below. QUESTION: What was a peak experience of when technology employed in GROOVEs/grants facilitated learning at an individual/org/industry level?

17. 3. Summary of Review:Critical Aspects of LN and Findings 7. Facilitation Findings: • Facilitation very uneven across LN but PLP seemed to have better facilitators than IGP, which had very direct impact on performance of LN. • Criteria for selecting facilitators emphasized technical expertise over softer skills (communication, facilitation). Lack of facilitation skills resulted in poor performing LN. • High performing LNs had facilitators who could: • connect with people (in person or virtually) • translate conceptual into practical & operational • synthesize a lot of material or distil in useful ways • guide discussions and efforts effectively • find synergies (keep focus on commonalities over differences) • demonstrate sound knowledge of subject matter

18. 3. Summary of Review:Critical Aspects of LN and Findings 7. Facilitation (cont’d) • High performing LN had facilitators who: • Made learning a priority and helped members to see value in learning by doing and learning as a group. • Invested time and effort to build trust by connecting with members and helping members to connect as peers. • Leveled playing field to ensure all members can equally participate, contribute and gain from the LN (mentor, coach, advise). • Actively engaged and encouraged members to contribute and share knowledge (thought-provoking questions, activate silent members) • Created a safe space for members to exchange lessons learned (ground rules, regular reassurance)

19. 3. Summary of Review:Critical Aspects of LN and Findings 7. Facilitation (cont’d) • Discussed with members and agreed on how often they should participate on the LN, at a minimum. • Responded in a timely manner to emails and requests for help. • Avoided information overload and making heavy demands on members’ time (brief message, directed reading, summarize discussions). • Helped members to develop a common learning agenda, with goals and indicators, that was both relevant and realistically achievable over the period of the grant.

20. 3. Summary of Review:Critical Aspects of LN and Findings • Unsuccessful or low performing LN had facilitators who: • Was top-down and militant; • Was driven by own personal agenda/interests; • Was indifferent or unresponsive to submissions, requests, emails, etc; • Showed up at in-person events late or unprepared; • Created a hierarchy; • Pointed the finger when things didn’t go well; • Was patronizing or aggressive; • Could not communicate clearly; • Could not provide clear directions; • Was not technically credible • GROOVE Rwanda Discussion Thoughts: • Virtual interaction is important, but really appreciate the value of substantial, focused time to crystallize ideas. • At a certain point, realized that there was an expectation of us as a learning network beyond the work that we would do in our own organizations. That turned out to be okay. • Flexibility – when had the need to choose between our organizational approach and the group approach, went with group approach because of the belief that it would be better. • We were given the opportunity to engage by making small commitments which grew. • INHIBITOR: Needed clearer TOR for facilitator. Alignment of group culture and facilitation didn’t work. Overly focused on workplan document. Wasn’t ever clear about the facilitator’s TOR or the resources they had. Needed more expectation sharing around the role of the facilitator. GROOVE should have taken initiative to have reviewed facilitator TOR early on. Making this match is an issue – how could anyone know the LN culture early on? Maybe should have used a different model at the beginning. Review and make change as needed after first six months. • SCOTT: We have had three facilitators in as many years. We could have something to contribute here such as concrete examples of what to do and not to do as a facilitator and/or participant being “facilitated.” • QUESTION: What was a peak experience of when the facilitator or facilitation process helped you or the members learn, that is, pushed the involvement of members to a higher level and greater learning? 7. Facilitation (cont’d)

21. 3. Summary of Review:Critical Aspects of LN and Findings 8. Planning and Dissemination of Knowledge Products Findings: • LOE and resources required to develop knowledge products consistently underestimated, even in well planned out and implemented LN. • Absence of well planned dissemination and communication strategies in all LN to magnify learning (standard communication channel –SEEP, microLINKS or USAID websites). • Other less used but effective channels: workshops and conferences, universities, online gateways, Speaker’s Corner, newsletters, bulletins, etc. • Most k-products took form of report. Limited creativity (exception BELO wiki). GROOVE Rwanda Discussion Thoughts: • Able to jump over barriers and move from thinking about individual products to the group product. • “Let the user complete the product” e.g., how Tracy handled ownership issues for the mentor program – it was made clear that this was a GROOVE product, not CARE’s. • Sometimes there was a need to formalize the relationship (especially when we consider the needs of our individual organizations). The trust was there before we had to do that. The MOU process and how we handled it a good example. • We made clear the rules on acknowledgement/recognition of contribution. • INHIBITOR: Productivization of knowledge – we needed a clearer model of what a knowledge product looked like, what was expected • INHIBITOR: Needed dialogue with KDMD and although we asked for it, it was not easy to get. We should have been able to have had an earlier conversation – here’s where we are, let’s talk about how to collaborate and move forward toward emerging objectives. KDMD relied more on providing a “menu” of what could be done. Totally supply side driven. Not ever sure what a fair expectation of KDMD was and it didn’t get set. Need to have a clear idea -- what resources are available, how we’re going to work together to get the product that we all want, how get “the synergy of being part of KDMD.”  SCOTT: As mentioned earlier, the first bullet above is relevant. If we have some guidance to give on how to budget and/or plan for knowledge products, that would be helpful. In general, giving some concrete examples of dissemination mechanisms and/or knowledge products that were most effective with explanations of how and why we know they were effective would be useful to others. We also need to define “effective” for this question. QUESTION: What knowledge products are you most proud of? Why? What was the process you underwent to produce them? To disseminate them? On a scale of 1-10, how would you rank the effectiveness of the knowledge product? Why?

22. 3. Summary of Review:Critical Aspects of LN and Findings 9. Key Performance Indicators Findings: • Absence of short- and long-term systems to monitor and measure the success/impact of the LN and the knowledge shared and produced. • No strategy to capture stories, cases where relationships extended beyond grant and resulted in successful partnerships or joint initiatives. These are good motivators for new LN cohorts. • No exit or longitudinal survey to benchmark and track impact over longer period. SCOTT: We have something to say here, especially regarding the relationships beyond the grant parameters. We have also tried to back our way into an impact assessment by doing the retrospective baseline and now the final assessment. The exit or longitudinal survey of impact beyond the end of the grant is a open question for USAID/QED. I think we could generate stories of “impact” once we agree on the definition of “success”. QUESTION: Looking at the definition of a successful LN and the relationship to learning for individuals/organizations/industry, what evidence can you give for success of the GROOVE network? What evidence can you give of what you would do differently under GROOVE to make it more successful?

23. 3. Summary of Review:Critical Aspects of LN and Findings 10. Post-LN Strategy Findings: • Some felt weakest point was absence of multiplier effect. Grants have clear end (no strategy or planning beyond to build on what was achieved). • Some thought, effort, and possibly resources, should be put into developing a post-LN strategy to scale up, replicate, connect with other networks, etc. Scott: We are now looking at this question and asking what should be continued after the grant. I am not sure if there are any conclusions we can share with USAID. It seems like it would be better handled as a separate discussion among the GROOVE/QED/USAID than as part of the impact assessment.

24. 4. Key Recommendations • Preserve, promote (and enforce) LN model. Proven to be robust and unique. Don’t need a new LN model. • Fix critical components of LN to make them more effective, focusing on: • Integration issue (at RFA level) • Grantee and facilitator selection criteria • Technology challenge • Training and supporting facilitator • More creative k-products and better dissemination • Developing smart KPI • Post LN (replicate/scale up) SCOTT: For this list of Key Recommendations, it would be helpful to know which ones USAID/QED applied to the GROOVE so we could give direct feedback on on whether the recommendation seemed to make a difference in promoting learning.

25. 5. Putting Our Learning to Action • Capturing and applying knowledge to our practice of building more effective LN

26. 5. Putting Our Learning to Action • Capturing and applying knowledge to our practice of building more effective LN