Evaluating widening participation a series of unfortunate events

1. Evaluating widening participation �a series of unfortunate events� A case for a multi-functional approach to the evaluation of widening participation Chris Howard Institute for Access Studies c.d.howard@staffs.ac.uk

2. Overview Evaluation theory uncovered Evaluation of widening participation A Multi-functional approach to evaluation Evaluating widening participation using a multi-functional approach to evaluation.

3. Evaluation Theory There is no clear account of evaluation (Shaw, 2000; Thomas, 2000). The discourse on evaluation contains contradictions and overlaps on the description, terminology and uses of evaluation (Howard, 2004; Shulha and Cousins, 1997). In order to develop a coherent account of evaluation it is necessary to focus on both the theoretical and practical dimensions of evaluation.

4. Evaluation Theory: Theoretical dimension Evaluation comprises of four functions (Howard 2004). Impact function addresses the question �what difference has occurred as a result of the project?� Improvement function addresses the question �how can a project be improved to become more effective?� The adherence function addresses the question �Did a project do what it was supposed to do?� Thus adherence focuses on the comparison between the implementation of a project as it was observed with the intended delivery. The generalised knowledge production function addresses questions centred on �What knowledge have we gained from the project?� This function has predominantly addressed the specific question �What knowledge have we gained about the social issue/problem a project has addressed?�.

5. Evaluation Theory: Theoretical dimension Theoretical paradigms conceptualise evaluation. There are four theoretical paradigms, which are positivism, critical realism, pragmatism and constructivism (Milne et al. 2004; Pawson and Tilley, 1997). Paradigms are distinguished by their underlying epistemological and ontological assumptions (Ovretveit, 2002). Paradigms conceive evaluation based on incorporating functions that correspond with their own assumptions about knowledge and the social world.

6. Positivism Assumes natural world is synonymous with the social world in that they both consist of objective and quantifiable truths (Morss,1996; Walsh, 1998). Therefore the methods of inquiry used in the natural world should be applied to the social sciences (Bryman, 1988). Ontology- social reality is objective Epistemology- there are objective truths Evaluative Functions- impact, adherence, generalised knowledge production.

7. Critical Realism Assumes the experience of reality is a social construct but claims that there is a �real world� with underlying structures and mechanisms (Hughes and Sharrock, 1997). Ontology- a real world that is stratified into different domains. Epistemology- there are underlying causal mechanisms that connect structures and events. Evaluative Functions- impact, improvement, adherence and generalised knowledge production.

8. Constructivism Assumes reality is a product of social activity (Kukla, 2000) where individuals create meaning through interaction with each other and their environment (Gredler, 1997; Prat and Floden, 1994). Ontology � reality is subjective. Epistemology � existence of multiple truths with equal validity. Evaluative Functions � improvement.

9. Pragmatism Assumes that evaluation should meet the needs of policy makers and those in a position of power (Patton, 1986, 1997; Weiss, 1987). Ontology � reality regulated and constructed by those in power. Epistemology- knowledge is what is considered pragmatically acceptable by those in power. Evaluative Functions- not fixed.

10. Evaluation Theory: Practical Dimension Evaluation approaches Translate the theory espoused by theoretical paradigms and apply this in a practical setting (Hall and Hall, 2004; Shaw, 2000). Specific to theoretical paradigms. Share the underlying epistemological and ontological assumptions of theoretical paradigms. Advocate a set of research methods that compliment the assumptions of theoretical paradigms. Do not necessarily adopt all the functions espoused by theoretical paradigms

11. Widening Participation Widening participation has become a national concern since the publication of the Dearing report (1997) and Kennedy report (1997). Previous strategies intended to diversify the Higher Education (HE) student population have been of limited value (Lumby and Wilson, 2003). The statistical evidence for widening participation: Participation rates for low socio-economic groups have increased by 8 percent during 1990-2000 in comparison to an increase of 11 percent for high socio-economic groups during the same period (DfES, 2003).

12. Widening Participation The national response has been to focus on a structured national strategy (HEFCE 06/04). This has taken the form of Aimhigher, which is an aspiration raising project implemented regionally using a range of activities (DfEs, 2003). The Widening Participation and Fair Access Research Strategy consultation paper (WPFARS) (HEFCE 06 2004), acknowledges that it is crucial to evaluate this widening participation strategy.

13. Widening Participation The suggested evaluation outlined in the WPFARS and the National Aimhigher evaluation seminar (May 2004), is based on meeting measurable targets e.g. 50 people attend a taster day. The evaluation of Higher Education Institutions (HEI) widening participation activities is based on meeting targets to determine the distribution of HE funds (HEFC 59/2003). This corresponds with the target of 50 per cent of 18-30 year olds in HE by 2010 (DfES, 2003). Therefore the evaluation approach is derived from pragmatism and adopts an impact function based on meeting measurable targets.

14. Limitations of current evaluation Can not identify how or why widening participation works. Can not identify aspects of current widening participation that may need improving. Can not identify the unintended impact of widening participation. Outputs accounted for may not actually relate to widening participation.

15. Multi-Functional Approach to Evaluation Using a critical realist framework, we can develop an evaluation approach that adopts all four evaluative functions. The four functions are not exclusive or in competition with each other. The four functions can be incorporated within an evaluation approach to inform and triangulate with each other.

16. Multi-Functional Approach to the Evaluation of Widening Participation Adopting a multi-functional approach to evaluation would: Impact function: would establish what widening participation drivers work and in what contexts. Identifying the impact of a project also enables areas to be identified for improvement (Scriven, 1993) (improvement function). Generates knowledge about the mechanisms that produce widening participation into HE (Pawson and Tilley, 1997) (Generalised Knowledge Function).

17. Multi-Functional Approach to the Evaluation of Widening Participation Improvement function: challenges the appropriateness of current widening participation and identifies aspects for improvement. Identifying and implementing improvements to widening participation will relate to a positive effect on project impact (Sanders, 1999) (Impact function). Identifying areas for improvement contributes to producing new knowledge about widening participation (Lipsey, 1993).

18. Multi-Functional Approach to the Evaluation of Widening Participation Adherence function: Measures whether the widening participation strategy is being delivered. Monitoring the delivery of widening participation is necessary when measuring it�s impact (Weiss, 1998) (impact function). If widening participation is implemented incorrectly this can by improved by ensuring that is delivered correctly (Mark et al. 2000) (Improvement function). The means of monitoring the delivery of the strategy informs how to monitor future and/or strategies in other areas (Generalised knowledge production function).

19. Multi-Functional Approach to the Evaluation of Widening Participation Generalised knowledge production is informed by the impact, improvement and adherence functions. Impact provides data about what mechanisms work for widening participation, which can be used to design future strategies. Improving widening participation provides knowledge when designing future strategies. Monitoring widening participation provides knowledge about the means of monitoring the delivery of future widening participation strategies. Finally, the function can be used to indicate how a multi-functional approach operated in the context of widening participation and how it can be applied across other settings.