Suki Goodman

1. Suki Goodman School of Management Studies University of Cape Town

2. Evaluating Programme Theory Evaluating Programme Theory: a case illustration of the value and challenges in conducting a theory evaluation

3. Outline of Presentation • Aim • A little about theory evaluations: what we hope to achieve • Brief introduction to programme • The logic model/causal logic • Brief examples of attempts at theory evaluation • Practical problems and challenges

4. Aim • The aim of the presentation is to give snapshot of some the processes, results, relevance and challenges of attempting to conduct a theory evaluation of a judicial education programme

5. TheoryEvaluationsRossi, Lipsey & Freeman, 2004 • Evaluation at this level of the evaluation hierarchy assesses if the causal logic implicit in the programme is practically realistic and theoretically sound (Donaldson, 2003) • The programme theory is analysed to assess the feasibilityof the stated objectives of the intervention

6. Evaluating the programme theory entails addressing a variety of evaluative questions such as: • Are programme goals and objectives well defined? • Is there alignment between the programme theory and a documented social need? • How well does the programme theory compare with research and practice • Is the programme logic feasible and plausible? (Rossi et al., 2004)

7. There are a number of steps in a theory evaluation • evaluator has to extract a logic model of the training programme • assess if it represents the key stakeholders’ understanding of the underlying causal processes • outline historical conflicts/problems in the conceptualising of the programme which might be reflected in the final product

8. Brief introduction to programme • Law, Race and Gender (LRG) Unit’s programme of social context training. • The LRG Unit housed at the University of Cape Town’s Law Faculty • Unit has been conducting social context training with South African magistrates since the mid 1990s

9. Law, Race & Gender Unit • Established 1994 as a research and training unit • Social context training: • Provide different views and experiences of the world that people have • Explore ways of understanding different attitudes and social realities • Enable participants to recognise and deal more sensitively and fairly with cases

10. LRG Approach • Judicial education in three dimensions to be effective: • Content – substantive law • Craft – Skills enhancement • Social Context

11. Fig. 1 Programme Logic Model Proximal outcomes Distal outcomes: Individual Distal Outcomes: Organisation Input Change in knowledge, awareness, attitudes and understanding Change in work behaviour: Using new knowledge on the job Improved administration and delivery of fair and equal justice appropriate Workshop/ training intervention

12. Is the programme logic feasible and plausible? • The feasibility and plausibility of the sequence linking or causal logic implied in the programme will be analysed by • (1) a critical discussion of how key concepts are operationalised in the training and • (2) interrogating the causal relationships implied in the programme with social psychological knowledge about the attitude behaviour relationship

13. TheOperational Definitions of Key Concepts • The programme’s documentation gives scant attention to how racism or prejudice is defined or understood bar a relatively crude explanation related to : • redressing past inequality and prejudicial treatment • discriminatory laws associated with the apartheid regime

14. What we know about modern-day racism is that it operates on two levels: • traditional blatant racism • more subtle negative feelings of which the individual is unaware(Akrami, Ekehammar & Araya, 2000; Dovidio, 2001, McConahay, 1986; Meertens & Pettigrew, 2001; Sears, 1998)  

15. Latter that causes theory problems for the programme • Training attempts to raise magistrates’ levels of consciousness about their stereotypes, biases and discriminatory practices • Not structured around, or designed to penetrate, seemingly intractable deep-seated implicit attitudes

16. The attitude-behaviour relationship • The causal logic of the theory suggests: training will bring about change in magistrate’s prejudicial attitudes which will in turnbring about changes in their behaviour

17. The relationship between attitude and behaviour is complex and nuanced(Ajzen, 1985,1987, 1988, 1991; Ajzen & Fishbein, 1980; Albarracin & Wyer, 2000; Bagozzi, 1992, Bohner & Wanke, 2002; Eagly & Chaiken, 1993; Eiser, 1994; Eiser & van der Pligt, 1988; Foster & Nel, 1991, Kraus, 1995). • Historically there has been ongoing controversy about the casual links in this relationship

18. The Causal Debate • A contemporary conceptualisation of the attitude-behaviour relationship posits that it can best be understood as a “substantive relationship of interest” (Kraus, 1995, p.71) • A relationship which will vary in strength according to context and is influenced by a variety of variables

19. Modern theorists agree that there is no straightforward answer to the theoretically, and practically, complicated question, does attitude predict future behaviour?(Bohner & Wanke, 2004; Eagly, 2004)

20. Using theory to evaluate the programme • Theory of Reasoned Action (Ajzen & Fishbein, 1980; Ajzen & Fishbein, 1975) • Theory of Planned Behaviour(Ajzen, 1985,1987, 1988, 1991, Ajzen & Madden, 1986)

21. Attitude Subjective norm Intention Behaviour Perceived behavioural control Fig. 2 Theory of Planned Behaviour (Version 1 without broken arrow, Version 2 with broken arrow)

22. Challenges • Evaluability assessments • Eliciting programme theory: • Defining boundaries • Identifying causal logic • Academic pursuit • Practicality for “real” world evaluations • Implications of flawed logic for programme staff