What SAIE does

1. Measuring enterprise development impact in complex, resource constrained environments in South Africa – adapting the EvaluLEAD Guide from leadership development to enterprise development

2. What SAIE does • Entrepreneurship & Enterprise Development • BEST Game business simulation used in over 60 countries (ILO/GTZ) & a range of adult training programmes • Agriplanner and related Ag programmes • Wings of Welcome tourism programme • Business VENTURES in schools gr 2-12 • Training for Bankers in assessing loan applications • Career guidance programmes

3. Some of SAIE’s products

4. Land Restitution Farmers

5. Mixed results with quasi-experimental methods, 1 • Worked with University of Cape Town Graduate School of Business staff linked to the Global Entrepreneurship Monitor (GEM) • Pilot tested methodology in 5 schools with control groups, pre-post testing, random selection & external Q administration • Business VENTURES had a significant positive effect on learners in a range of entrepreneurial knowledge, skills and attitude areas • Level of understanding of basic business concepts (business concepts) • Capability in financial arithmetic (financial arithmetic) • Confidence in ability to start a business (entrepreneurial confidence)

6. Mixed results with quasi-experimental methods, 2 • Rolled out approach to 41 schools • Positive impact results, but also some confusing results (like regressions in financial arithmetic scores) • Led us to a critical review of the method • Led to a review of entrepreneurship teaching practices in SA schools (new to curriculum) • Led us to visit many of the schools to interview the teachers (training dosage, other issues) • Looked at disciplines like Math Education & Health Education

7. Study on Instructional Improvement (SII) at University of Michigan • A large scale, multi-method, longitudinal study that examines the effects of instruction on student achievement at 120 schools, including intensive qualitative interviews at 12 of these schools. • Future entrepreneurship research studies will need better controls for extraneous factors including (but not limited to): • teacher qualifications • teacher experience • teacher knowledge of entrepreneurship • teacher motivation and attitudes • classroom implementation of entrepreneurship curriculum • how the various materials were used • interaction of teachers, students, content and environments.

8. Tailoires HIV/AIDS criteria for assessing strength of evidence • The level of evidence: informed judgment, associations, plausibility evidence, or probability evidence from randomized controlled trials. • The quality of the intervention: what are constant and variable components of the intervention, is it feasible, are there identified mechanisms of action, is there an experiential base, and has there been careful pilot testing? • The quality of the outcome measures: the robustness and validity of the outcome measures. • The process evaluation of the intervention: the reception by target group and practitioners,factors facilitating and inhibiting implementation, and the quality of implementation, in terms of intensity, duration and completeness. • The context in which the intervention is delivered: similarities between target groups, the sociocultural setting, the environment and infrastructure, and historical factors such as the phase of the epidemic.

9. Searching led us to EvaluLEAD • Team concluded: need to develop richer, more sophisticated ways of measuring impact which also takes into account the rich anecdotal information not captured in quantitative evaluation methodologies • Stories from SAIE participants were often encouraging, compellingly powerful & showed developmental impact • Complexities of leadership development in EvaluLEAD were appealing because of complexity in entrepreneurship development

10. How EvaluLEAD evolved • In 2001 the Population Leadership Program (PLP) of the Public Health Institute (PHI), a project funded by the United States Agency for International Development (USAID), was wrestling with the question “What form of evaluation would best help us determine if and how leadership development activities make a difference?” • At the same time the W.K. Kellogg Foundation was reviewing 55 varied leadership development programmes for good evaluation practices. The two groups collaborated in developing the PLP’s promising emerging framework into what became known as EvaluLEAD. • In early 2004, after some further concept development work, 17 organisations running a wide range of leadership development programmes were trained to use the EvaluLEAD methodology and field test it in their organisations throughout 2004, with external expert assistance throughout the year. The EvaluLEAD Guide is the result of the learnings and developments from these groups.

11. Domains of Impact • Individual Domain • Organisational Domain • Agencies, departments, programmes, teams, alliances, other • Societal/Community Domain • SAIE added Facilitators in Organisations as key individuals, so we use two levels of Individuals – Participants and Facilitators

12. Forms of inquiry • Evidential • Attempts to capture and represent facts regarding what is happening to people • Descriptive, numeric and physical evidence of programme impact – “hard evidence” • Evocative • Seeks viewpoints and sentiments of those influenced by the programme • Stories, views, opinions, discourse • Richness and human dimension of what is happening

13. Result Types • Episodic (Immediate) • Cause and effect; time bound; somewhat predictable • Developmental • Can occur across time; forward progress, stalls, setbacks • Open-ended results that are less controllable and predictable • Transformative • Fundamental shifts; regenerative moments; ‘prize’ to which programmes aspire; crossroads or unanticipated new roads

14. Programme Results Map

15. Implementation process • Clarify your vision of change – desired programme impacts • Define the context • Describe your evaluation context • Define desired result types • Define domains of impact • Create an initial programme results map • Prioritize results • Develop data collection strategies • Refine evaluation plans

16. Unique challenges for SAIE • Different programmes & desirable to have one evaluation framework • Range of target groups - economically poor communities, schools, universities, large NGO youth leadership • Dependence on facilitators from other orgs • Great materials but mixed facilitator knowledge and skill levels • ‘Training dosage’ varies

17. SAIE’s process • Involve a Reference Team • Clarify domains of influence – added Facilitators • Developed key indicators as main discussion and focus point • Discussions led to need to clarify theories of change • Shift from measuring attitudes to measuring behaviour – application of what is learned • Led to a rich ‘practice to theory to practice’ discussion and helped clarify the organisation’s core business as enterprise development • Key theorists and the conceptual areas they represent • Key indicators • Clarifying most helpful data collection methods • One result of this whole process was putting organisational dialogue and learning front and centre as a key strategic issue – (and this needs to be extended now to other constituents)

18. Theory and Theorist Summary, 1 • How to influence behaviour • Icek Ajzen, 1987 – Theory of Planned Behavior • Bandura, 1977, 1986 – Social Cognitive Theory • Resiliency, self-efficacy, self-awareness – Bandura, & others: Linquanti, Grunstein, Garmezy, etc. • Influence of culture on risk behaviours • Wildavsky and Douglas, 1982 – Cultural theory of risk

19. Theory and Theorist Summary, 2 • Enterprising competencies • Kolshorn and Tomecko of CEFE, 1995 – Strengthen enterprising competencies • Wood, 2008 - Contribution-based mindset (or value added thinking); • Kawasaki – ‘making meaning,’ adding value, helping lots of people, doing what’s right, paying back society; • Malloch – strong motivating principles about benefitting society as the reason for existing - that go beyond profit • Australian government – Enterprise Education, 2004

20. Theory and Theorist Summary, 3 • Entrepreneurship development • Sull, 2004 – Disciplined Entrepreneurship - Cognitive, iterative process of small experiments & improving • Lévi-Strauss, 1967 – Entrepreneurial bricolage – making do with what is at hand • Networking, connecting & gaining access to critical resources – Mike Morris on critical resources and access to them • Necessity; dissonance; pain; survival as key drivers • Teaching and learning • Career development

21. SAIE’s work aims to help individuals demonstrate:

22. SAIE’s work aims to help Facilitators demonstrate:

23. Organisational indicators

24. Societal / Community Indicators

25. Data collection methods • Pre-post questionnaires for adult programmes • Key informant interviews and sampling of focus groups for adult programmes • Sampling of focus groups for schools programmes • Case studies with external assistance • Exploring tape recordings with transcripts of interviews and videotaping interviews

26. Questions and discussion • What would you like to ask about, discuss, comment on or share? • Thanks for your attention and interest!

27. Collingwood Place, 11 Drake Street, Observatory 7925 South Africa • PO Box 13805, Mowbray 7705 • Phone 27 21 447 2023 • Fax 27 21 447 9911 www.entrepreneurship.co.za • Email: ventures@entrepreneurship.co.za Don Shay Don Shay & Associates Small business research and development consultants donshay@mweb.co.za +27 (021) 532-2433 tel/fax www.partnersforimpact.co.za