EVALUATION: A PROCESS SUPPORTING SKILLS AND INTERVENTION DEVELOPMENT

1. EVALUATION: A PROCESS SUPPORTING SKILLS AND INTERVENTION DEVELOPMENT Réginald Savard, Ph.D. Université de Sherbrooke

2. Evaluation: The Research • Do you know of many studies aimed at determining… • How interventions have been useful to clients? • How interventions have contributed to compensating for their difficulties? • How interventions have helped to meet their expectations and needs? • How interventions have improved their future or social trajectory?

3. Evaluation

4. Evaluate for whom?

5. Evaluate for whom? Organizations • Evaluate skills and performance • Evaluate learnings post training Clients • Evaluate effects of interventions (on the basis of expectations and needs) • Evaluate level of satisfaction Counsellors • Evaluate effectiveness of interventions • Evaluate factors during the intervention process

6. Evaluate by whom? The actors concerned, their motivations, their interests…

7. Evaluate by whom? The actors concerned, their motivations, their interests…

8. Evaluate by whom? The actors concerned, their motivations, their interests…

9. Evaluate by whom? The actors concerned, their motivations, their interests…

10. Evaluation for CounsellorsA FEW ISSUES • Satisfaction with service offered • Job enrichment • Work satisfaction • Professional skills • Self-esteem, humiliation, shame • Feeling of personal efficacy • Etc.

11. Who evaluates? Who evaluates whom? External evaluator • 3 perspectives • Internal (self-evaluation) • Counsellors and clients • Organizational • External

12. Why evaluate? • A phrase from the Viennese philosopher Ludwig Wittgenstein. • What he meant, so far as I understand him, was that any thinking person sees the world like a fly looking out of his bottle. • That's to say that we see the world clearly, but we're limited by confines beyond our control: in the fly's case, the bottle. • In the human's case, our singular experience.

13. Why evaluate?

14. Why evaluate?

15. Why evaluate? • To grasp the reality of individuals in constant interaction with their environment and to understand the relationships within that environment. • To grasp the reality of individuals seeking to meet their needs and the ability of their environment to meet those needs. • To evaluate factors related to the individual and those related to the environment. • To evaluate the individual’s ability to meet his/her needs in his/her environment.

16. Why evaluate? • Through evaluation, counsellors (or the organization) learn about themselves (or the client), their functioning, the meaning of their results, their processes (Bouvier, 1998). • Evaluation practices encourage counsellors to seek to gain better self-knowledge, particularly through results analysis, by engaging in reflection in order to better understand the processes implemented. • Evaluation makes it possible to understand what they have done, what they are in the process of doing, what they plan to do, and what they will do.

17. Why evaluate? • To situate the person in his/her developmental trajectory in relation to the different spheres that help to explain his/her problem and adaptation, as well to identify his/her needs (Pronovost and Moreau, 2004). • Evaluation is necessary to gain a better knowledge and understanding of the problem situation and to make better decisions about the best intervention (Alfoldi, 1999).

18. Why evaluate? • Evaluation reveals the person’s potential for change and ability to adapt (Pronovost and Moreau, 2004). • Evaluation may help to activate a change process with clients • Become aware of their abilities • Become aware of their difficulties • Formulate new ideas • Propose methods never before contemplated • Identify worthwhile projects

19. Why evaluate? • Evaluation makes it possible to step back, which affords some protection for counsellors who are very involved in the problem situation (Alfoldi, 1999). • Evaluation makes it possible to clarify the roles and responsibilities of each person, as well as the intervention targets (Alfoldi, 1999). • According to Alfoldi (1999), evaluation makes it possible to • Have a different understanding of the person’s functioning (needs, abilities, etc.) and sources of obstacles or dysfunction • Plan the intervention • Develop an action plan. • According to Pronovost and Moreau (2004), • Determine the service to be offered.

20. Why evaluate? • Evaluation enables the actors to clarify and adjust objectives, verify practices, improve action, and help with decision making • Regulation of intervention practices • Development of human resources • Management of training plan • Support for, or piloting of, quality initiatives • Definition of an organization’s strategic directions

21. Why evaluate? • Evaluation makes it possible to identify the client’s initial functioning and determine whether that functioning improved following the intervention. • Evaluation is a way of taking a step back from a client’s problem that sometimes makes counsellors emotionally vulnerable.

22. Evaluation: a process Why evaluate? • This process should make it possible to plan, carry out, and re-evaluate the intervention (Hayes, Nelson and Jarret, 1987).

23. Evaluate what? • Evaluation is a (dynamic) process through which we prepare (identify), observe, and collect useful information (obtain) and synthesize it in the form of advice or judgements (reviews) for the purpose of disseminating it to others (counsellors, employers, decision makers, etc.). Orientedtowards a specific goal (or goals), making it possible to judge possible decisions. Stufflebeamet al. 80

24. Evaluate what? • Process (1) through which we define (2), obtain (3) and provide (4) information (5) that is useful (6) and enables us to judge possible decisions (7) • process = ongoing activity • define = identify pertinent information • obtain = collect, analyze, measure data • provide = communicate those data • information = facts to be interpreted • useful information = that satisfies relevance criteria • possible decisions = orientation actions, etc. Stufflebeamet al. 80

25. Evaluate what? • Evaluating the quality of the product delivered means asking questions about the criterion that defines that quality. • Quality is meeting client expectations and needs. • Evaluating the quality of the product delivered is based on the idea of querying resources about their ability to meet client expectations and needs.

26. Evaluate what? • Relevance of interventions in relation to the client’s expectations and needs and the organization’s priorities. • Effectiveness, i.e., the degree to which interventions were (or should be) carried out and their impacts • Impact evaluation is a systematic identification of the positive or negatives effects, intentional or not, on different elements generated by an activity, in this case a program or project. • Efficiency, i.e, the degree to which interventions were (or should be) carried out given the resources available. World Development Bank

27. Evaluate what? • Evaluation requires selecting the subject of the evaluation and the most appropriate tool. • There are a wealth of tools available, and that should make it possible to conduct an analysis that is as realistic and dynamic as possible of the person in terms of his/her career path (Birk and Cassereau, 2003). • Wealth of kaleidoscopic pictures (changing and moving). • Evaluation must be able to be argued, justified, and consequently based on a certain number of parameters or variables that can be identified (Steffatre and Pachoud, 2003).

28. Evaluate what? • Combine the subjective and the objective to avoid a distancing from the actual experience of the clients and the counsellor (Lecomte and Savard, 2003). • According to Patterson (2002), • Evaluation forces us to specify the topic to be discussed to avoid confusion. • Evaluation requires systematic logic in the face of reality, corroboration of intuition through observation. • To intuition and logical discovery, add experience and knowledge through empirical demonstration.

29. Evaluate what? • Combine the subjective and the objective to avoid a distancing from the actual experience of the clients and the counsellor (Lecomte and Savard, 2003). • Evaluation forces us to specify the topic to be discussed to avoid confusion. • Evaluation requires systematic logic in the face of reality, corroboration of intuition through observation. • What are the factors that help to explain the results obtained? • Contribution of counsellor’s characteristics • Contribution of client’s characteristics • Contribution of techniques Heppner and Heppner, 2000; Swanson, 2002

30. Evaluate what? • Specific elements:8% (Techniques) Specific interventions • General factors:92% (Common factors) • Counsellor variables • Client variables • Working alliance Wampold, 2001 General

31. Evaluate what?

32. Evaluate when? INTERVENTION

33. Evaluate how? • Qualitative approach: evaluation through a combination of data collection and information processing techniques to determine significance • Quantitative approach: evaluation using standardized tools • Systematic observation grids • Questionnaires • Standardized tests

34. Evaluate how? • Standarized tools would be required to guide the intervention effectively. • The criteria for selecting an evaluation instrument are as follows: • It must be flexible. • It must not affect the relationship between the counsellor and the client. • It must take into account several factors in order to reflect the complexity of the intervention. Beutlerand Martin, 2002

35. Evaluate for whom, by whom, why, what, when, and how? Intermediate results

36. Perhaps the essence of research can be summed up in this quotation from Schopenhauer: “The task is not so much to see what no one has yet seen but to think what nobody has yet thought about that which everybody sees.”

37. Reginald.Savard@USherbrooke.ca