Evaluation of the BEST Initiative: A Report on Year 2 Findings for Cohort 2

Evaluation of the BEST Initiative: A Report on Year 2 Findings for Cohort 2 July 2007

Evaluation Overview: Cohort Model • Cohort 1: • 11 organizations* • Data include pre-assessment, 1st year post assessment and 2nd year post-post assessment (10 organizations), plus interviews • Cohort 2: • 7 organizations* • Data include pre-assessment and interviews *Each cohort had one organization not included in the data analysis.

Evaluation Overview: Data Collection • Surveys: • Post-test re-administered to Cohort 1 (14 responses from 11 organizations) • Pre-test administered to Cohort 2 (14 responses from 7 organizations) • Interviews: • 17 with Executive Directors and/or Board Chairs • 13 with Consultants • 5 with Funders • 2 BEST Staff • Secondary Data: Best Survey Report 2006 from Center for Nonprofit Management (CNM)* * This report was prepared by Debb Wilcox at CNM for the BEST Project. The report has generously been shared with TCC to supplement the overall evaluation.

Overview • Similar demographics in Cohort 2 as compared to Cohort 1 • Assessment process valuable for organizations • Program continues to be implemented well and organizations show evidence of improvement • Training component continues to struggle with buy-in/value for participants

Organizational BackgroundOverview of Organizations’ Demographics

Organizational Background: Leadership • Cohort 2 organizations have existed as 501(c)3s for longer and have slightly larger budgets • Cohort 2 organizations have executive directors with shorter tenures with the organization Organizational Budget (as reported by organizations)

Organizational Background: Leadership—C2 • Similar to when C1 started, no C2 organization indicated having an executive transition plan • 29% have a formal plan regarding board member terms (compared to 64% for C1) • Have very active boards, as seen in chart below, representing a strong leadership foundation

Organizational Background: Adaptive Capacity • C2 and C1 show similar collaboration patterns in most areas • C2 organizations are slightly less likely to share information with other nonprofit leaders in the community than are C1 organizations

Organizational Background:Special Note • Additional organizational background information, especially quantitative graphs, for Cohort 2 is included in several of the graphs in the outcomes section. Since the Cohort 2 data are pre-assessment the data in those graphs are all about organizational background, whereas it represents outcomes progress for Cohort 1.

Assessment ProcessCohort 2 Experience

Assessment Process “The BEST assessment process gave the staff a sense of ‘emotional readiness’ to do this tough work and created a vested staff.”—BEST Consultant • There continued to be high levels of engagement in the assessment process across the organizations, including board, executive directors and other staff • General agreement that the assessment process was a critical component of BEST, as it forced organizations to look at their organizations holistically and provided a test to assumptions about what was working and what wasn’t • The assessment process, followed by the planning process, helped organizations think about change in a structured and long-term fashion

Assessment Process, cont. • C2 organizations are actively using the assessment findings, even beyond developing the CB plan. The rates of use are much higher than C1, despite C2 scoring lower on the usefulness of the report. This is a very positive development, and likely reflects a concerted effort on the part of the program to emphasize broader use of assessment results. Also, consultants might have a better understanding of the reports and therefore be more effective at helping organizations use the reports, though there is less evidence of that conclusion

Assessment Process, cont. “The assessment part is extremely useful because usually these organizations are moving so fast on a day to day [basis] and keeping their heads above ground they don’t have time to evaluate themselves. BEST says in the beginning, ‘everybody slow down here and let take a look at what’s important and what’s not.’ That part is extremely beneficial.”—BEST Consultant • Board involvement is crucial to buy-in and change, and the assessment process did an excellent job of integrating boards • As with Cohort 1, Cohort 2 leaders appear to be underestimating the intensity/time commitment associated with a CB consultant • The creation of a CB plan is an important step to integrate directly into the process because it moves the capacity building work from theory to practice

Assessment Process, cont. • C2 was slightly less likely than C1 to establish a task force, though more than 85% of organizations in both cohorts did so; the task force (internal team) continued to play an important role in helping organizations deliberately involve multiple people in the capacity building in an organized and productive way • The flexibility of the process and overall timeline continue to be important characteristics of BEST • C2 organizations were less likely to feel the process was tailored to their needs. This may be the result of a more polished program that has a greater sense of clear operations

Assessment Process, cont. Narrative on the graphs can be found on the following slide. Narrative on the graphs can be found on the following slide.

Assessment Process, cont. • Quality of consulting during the assessment process continues to be very high on most indicators • C2 organizations were significantly less likely to feel that key leaders were able to get acquainted with the consultant conducting the assessment. Again, it is not clear why this is the case for C2. Experience shows that leader buy-in to CB is critical for long-term success, though it is not clear how much involvement with the consultant is necessary

Assessment Process, cont. • The technology audit, new for Cohort 2, received high process ratings, but qualitative findings indicate there may be a need to better integrate the audits with general organizational needs and ensure that it doesn’t stand out as a totally separate assessment

Evaluation of the BEST Organizational Assessment Process • The Assessment Process continued to get high marks for understandability and addressing essential issues • The Assessment Process helps about one quarter of organizations identify priority areas not previously considered by the organizations. This is not uncommon, as the value of an assessment process tends to be more around how to compartmentalize information, get everyone on the same page and think about taking action • The process did push people’s assumptions about what was and wasn’t working, giving them a broader perspective on their organization “Rather than continue replacing one computer at a time, our organization used the assessment to prioritize our entire technology needs, leading to a long-term plan.”—Board Chair

BEST StrategiesQuality Indicators

BEST Strategies • Organizations are very happy with their consultant-pair experience and the consultants appear to be utilizing ‘best practices’ in their work • The training program component continues to struggle with buy-in, and organizations desire more interaction with peers, indicating that the ‘cohort’ approach may not be as effective as possible (though the Executive Director’s Roundtable appears highly successful) • BEST staff are perceived to be highly accessible and clear

BEST Strategies, cont.Consulting • Consultants were most satisfied when they worked with an organization throughout the entire BEST process – assessment through implementation. This led to stronger performance, as the consultant didn’t need to repeatedly go through a learning curve • Executive Directors appreciated the large consultant pool and that they had a large role in selecting the consultant • Consultants were concerned that the pool may be too big, with some indicating they were not getting enough work. The tension made some less interested in the project, though it is not clear that this had an actual negative impact on the project • Consultants may benefit from additional formal, structured opportunities to get together, learn from each other, and share ideas in order to enhance their own work “[The consultants] did a superb job. They targeted our needs. We had a lot of input into [the process]. They leveraged our strength and were very candid about telling us what our weaknesses were. They brought us together with other organizations, shared input, even from people outside the state, on how we can better improve ourselves.”—Executive Director

BEST Strategies, cont.Professional Development “This kind of professional development isn’t just about individual capacity building, but a kind of peer learning which is of very important value.”—Executive Director • Participants generally appreciated the BEST sponsored professional development workshops, noting that “By BEST placing a priority on it, it becomes important conceptually” • Some interviewees felt that the majority of speakers were not as good as they could be and felt there may be opportunities to tap local knowledge of those in the program, though they did indicate that it has improved since last year • BEST includes so many facets of capacity building that it appeared difficult to find topics that were of specific interest to the majority of agencies and capacity builders • Participants valued the workshops mostly for their value in bringing the BEST group together. It seems like there are significant additional opportunities for the participants to share information and experiences, that may be as beneficial as professional workshops “The professional development opportunities are mostly worthwhile, even if just to be in the same room and share information and experiences.” —BEST Consultant

BEST Strategies, cont.Professional Development This chart provides additional evidence for the previous slide on Professional Development

BEST Strategies, cont.Website • The website is generally perceived to be of high quality • Several interviewees indicated they had looked at case studies of change • Interest in greater interaction via website among participants

BEST Strategies, cont. BEST Overall Process • Interviewees were very grateful that the BEST project was as flexible and robust as it was • Sustainability continues to be a frequently raised issue by all stakeholders (grantees, consultants and funders) • There are some questions about the ‘cohort’ model and the extent to which this approach is valuable and/or being adequately capitalized on • BEST staff are highly professional, skilled and respected by grantees and consultants “We were pleasantly surprised how open [the funders] were to our needs and how generous they were to give us what we actually needed.” — Executive Director “If you had a line from 1-5, I would have a 5 for highest level of trust and flexibility [for BEST staff]. They are always responsive and very supportive.” —Executive Director

BEST Strategies, cont.BEST Funders Collaborative • Grantees continue to express the uniqueness and value of funders working together—they value the effort • Funders feeling a sense of common ownership of the program, a sign of a balanced and functioning collaborative • Funders continue to feel they are learning more about the sector and their grantees “There is a strong sense that we will be better together than we will separately in years to come as we try to help the sector.” –Funder

BEST OutcomesFour Core Capacities

Outcomes:Core Capacities* Outcomes are presented in each of four core capacity areas: Leadership, Adaptive, Management and Technical, as well as outcomes in Program Quality. The chart below shows information for three of four capacities and Program Quality** *All data from pre-assessment on graphs are in regard to Non-BEST capacity building improvements in the year preceding BEST involvement. As a result, pre scores may reflect either the relative amount of focus on a particular area or the relative quality of the capacity building. **Program Quality is a one-item assessment of the amount of perceived improvement in program quality as a result of the BEST capacity building activities.

Outcomes:Core Capacities • Organizations are showing strong immediate improvements in leadership and program capacity, with improvements in adaptive and management capacity coming over time. By the post-post, all four capacities show improvements to about the same level. Two reasons likely account for the difference in the improvement trends: • The types of projects undertaken often reflected specific leadership needs, like board development, and were framed around the work they do: program capacity • The assessment process and resulting capacity building plan is a way for leaders to tackle organizational issues head on, which is a way to build leadership credibility and visibility

Outcomes:Core Capacities • Over the long-term, adaptive and management improvements evidenced greater stability, while leadership and program capacities had slight declines. Again, two reasons likely account for the difference: • The overall gains from the baseline year were greater for leadership and program. Just as the focus on these areas during capacity building led to their increases, there is a logical “settling” process that occurs when the focus on those areas is reduced • Adaptive and management improvements often are the result of changes in the way that things are done and establishing new systems and processes. Change associated with these tends to be harder to establish, but once in place has greater staying power, whereas leadership and program capacities are more exposed to environmental changes

Outcomes: Leadership • The process of being selected as a BEST participant is, in itself, a boost in confidence for organizations • Several organizations showed improvements in areas of mission, vision and strategic direction “As a result of BEST, our board…spends more time on strategic issues and long-term vision. It makes board meetings more interesting and efficient.”—Executive Director “We worked on a sustainability plan and did a succession plan. Our ED just left so this plan is working beautifully.”—Board Chair

Outcomes: Leadership • Leadership sustainability and reliance on one leader has significantly improved for C1 (C2 has remarkably good sustainability indicators already)

Outcomes: Leadership • Boards are being greatly strengthened in some areas, as they become more involved, trained and committed to the organization. For example, one ED reported that his board went from being 40% engaged to 75-80% engaged after BEST

Outcomes: Leadership • Boards seem to be getting worse at being able to reach out and inspire. This might be the result of a focus on more mundane board responsibilities, but still presents a significant concern that organizations might begin to lose good board members • The focus on board development has led to several persons dropping off boards in ways that have made the board stronger “[As a result of BEST diversity workshop], our board has become more cohesive. We now have a process for board nominations that is making the board more diversified.”—Board Chair “It galvanized our board, which is fairly new. We never had a strategic planning process that involved the board, union reps and administration. The BEST process brought all of them to the table. This wouldn’t have happened as easily without having an umbrella of BEST supporting us.”—Executive Director

Outcomes: Leadership • C1 is showing indications of utilizing strategic plans, both in leading to program improvements and regularly reviewing progress. This shows that effective development of a strategic plan can have longer-term impacts on program improvements

Outcomes: Adaptive • The process of going through the assessment phase itself is having a positive impact on the organization’s adaptive capacity • Entire organizations, from boards to staff members, are becoming more vested in the capacity building process and strategic change. In one case it was noted by a consultant that the staff’s “whole attitude has changed.”

Outcomes: Adaptive • Strategic planning is being done on a much more regular basis and this has been able to be sustained over time • Evaluation is more on the radar screens of the organizations, and many organizations are using evaluation findings to inform decision-making • There has not been a significant change in ability to identify new funding sources/opportunities, though revenue diversification has improved (see management section)

Outcomes: Adaptive • Collaboration levels remain fairly high, though the post-post showed decreases in some areas. While it is difficult to attribute the change to BEST, the lower levels might reflect a decrease in intensity after the program period, indicating that the convenings were a stronger motivator for collaboration than the development of a cohort. The collaboration changes might also reflect organizations looking more internally to address issues, such as management, and therefore less energy to look externally for collaborations

Outcomes: Management • Management capacity shows steady increases over time

Outcomes: Management “Our budget shows that we have extended from a four to a five million dollar organization. This has to do with careful planning and budgeting and new leadership roles.”—Executive Director • Organizations have improved their financial stability. While organizations have not identified new funders (see adaptive), the improvement is likely the result of several things: • Better resource development plans • More efficient operations, which have been improving steadily • Organizations have also improved the effectiveness of operations, likely due to: • Much better use of technology to support work • Improved recruitment, development and retention of staff

Outcomes: Technical • Organizations did not focus much on technical capacities (capacities specifically related to implementing organizational and programmatic functions), so limited outcomes in this area • One area with improvements is around fundraising skills, including the hiring of new development staff • Marketing and technology were two other areas where organizations saw some improvements. It will be interesting to see the effect that the technology audits have on this area for C2 “We have an annual fund (which we didn’t have before) that we have established and an annual fund plan, which includes a 100% board giving goal and recently raised $150,000.”—Executive Director “We have been able to improve our programs. Because of the fact that we are able to market better our program has grown double the size.”—Executive Director

Outcomes: Program • Organizations are much more focused on their overall mission (see leadership capacity) and as a result of that focus are viewing programs through the mission lens

BEST OrganizationsPerceived Capacity Building Needs and Sustainability

Perceived CB Needs • Strategic planning is much less of a perceived need, reflecting the focus by many organizations in C1 (this is also confirmed in the CNM data, where BEST organizations were significantly less likely to report needing SP CB) • Resource development/fundraising continues to be the highest perceived need, followed by operational management, evaluation and leadership development (CNM data revealed development/fundraising items as top organization-reported needs, followed by marketing plans. Evaluation was rated low and the other two items were not part of the CNM survey)

Perceived CB Needs

Sustainability • Sustainability continues to be a large concern. There is evidence that organizations are integrating capacity building into the way they do business, but it is not clear if that is sustainable over the long-term • While organizations are showing evidence of sustaining existing capacity building, there is some concern that they are not engaging in new bigger projects, due to lack of resources

BEST Recommendations

RecommendationsAssessment • Help new cohorts understand how time consuming good assessment and capacity building can be • Ensure that organizations have broad internal participation during the assessment process • Consider an assessment that can be administered over time to help keep an organization on track

RecommendationsStrategies • Work to maximize the use of the cohort model by: • Continuing facilitation of interaction • Using convenings for organizations to exchange information • Perhaps provide other incentives for interaction, like small planning grants (this could even include more prodding from consultants or the funders) • Use life-cycle to help organizations self-identify with cohorts • Continue to refine the workshop model, focusing on where organizations are in their life-cycle so that they derive greater benefit from the workshops

Recommendations Strategies, cont. • Start to consider potential ways to allow organizations to access capacity building funding on an ongoing basis so that needed changes are not put aside and momentum can be sustained • Focus more deliberately on consultant development—perhaps through specific workshops targeted directly at them

Evaluation of the BEST Initiative: A Report on Year 2 Findings for Cohort 2