Project Evaluation for Gerd Beidernikl Center for Education and Economy

1. TCA VENet|project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September 2005, Cyprus

2. the goals of this session • To introduce myself and the ZBW • To develop a common understanding WHY we are evaluating our work and HOW we are going to do it • To present the (preliminary) evaluation concept and to get feedback on it • To make sure the words and concepts used are familiar to everybody • To develop a common understanding of how to develop criteria that could be used as criteria for success • Overview on the upcoming steps

3. Introduction

4. my background • Profession: Sociology, further training in the field of evaluation and quality management • Researcher in the „Center for Education and Economy“, Graz, Austria • 5 full time employees • Public and private clients • My main working areas: labour market research, research on vocational education and training, evaluation, statistics and surveys • Ca. 20 evaluation projects in the last 4 years

5. project references on EU level • Evaluation of the ongoing Leonardo project „inclusive training for disability care workers“ • Self-evaluation of the Equal project „girls crack IT“ • Evaluation of the Styrian Territorial Employment Pacts • Evaluation of the articleVI project „GO BEST“ • Project management of an ongoing Leonardo project „online consulting tool for IT professions“ • Member of a proposal development group in the 6th framework programme, priority 7 • Reasearch activities for the CEDEFOP: reports on the Austrian training system, comperative country studies, administrating the ERO database (European Research Overview)

6. workflow of this presentation Concept Log Frame Elements of tender Gender Mainstreaming Feedback Evaluation What does it mean? What can it do for us? What do we need? Feedback Next steps Timeline At this meeting After the meeting Defining the Log Frame Indicators Terms of Reference

7. Evaluation >>There are as many different definitions of „evaluation“ as there are evaluators.<<

8. what is “evaluation” • >>Evaluation is the systematic investigation of the merit or worth of an object or process for the purpose of reducing uncertainty in decision making.<<(Mertens, 1998) • 3 types of evaluation – regarding the point in time • Ex-ante or predictive evaluation  concept • Acompanying or formative evaluation  processes • Ex-post or summary evaluation  results • Different context, different focus • Generating and securing knowledge • Controlling • Legitimizing actions • Dialogue with stakeholders, etc…..

9. why evaluation? what’s the benefit? • An evaluation offers a systematic way of approaching a topic  benefits for project planning • Gives neutral feedback for all project partners • Informs about performance, efficiancy, effectiveness and quality of the project • Makes the project results visible  marketing, proof • Possibility to learn in the runtime of a project • Possibility to counteract in case of anticipated problems • Secure Knowledge • Identify best practice and transferable models

10. what this evaluation should be • Support for the transnational cooperation • Answering the question: Do we deliver what we are supposed to deliver? • Neutral reflexion of what is going on • Suggestions for changes and improvement • Ensuring positive project development by giving periodic input • Fostering the coperation and communication

11. what an / this evaluation should not be • A judgement of good and bad, of right or wrong • Evaluators are no project outsiders, they are project partners • Evaluators are no pitbull terriers, lurking for any mistake • Evaluation ≠ project controlling • No analysis of cost efficiancy • No evaluation of the benefits for the target groups

12. What are your expectations?

13. Evaluation concept

14. logical framework • One commonly used apporach: logical framework = LogFrame • The LogFrame is the starting point of the evaluation • Displays the main logic of the evaluation subject • A skeleton of the project that has to be evaluated • Represents the causal relations between different levels of objectives, actions and results

15. LogFrame Concept Indikators sustainability Overall objectives Impact Specific objectives Performance Outcome Expected Products Output Structure Ressources / Input Processes / Mechanisms Effects / results Needs Goals Effectiveness Efficiency Relevance Secondary effects - unexpected Context effects

16. preconditions • There are commonly shared, defined and agreed goals • Goals have to be measureable • There are defined ressources • There are defined processes • The skeleton of the log frame has to be filled based on information in the kick off stage of the project. • Should be kept in mind in every working session of this meeting. • Will be worked out by the evaluation team after the meeting in cooperation with the project management

17. call for tenders • The call for tender asked for • Evaluation of the agreed procedures, tasks and responsibilities  formal evaluation • Evaluation of the results and the processes to reach these results  content evaluation • Participation at every meeting • Monitoring of the Gender Mainstreaming Process • Periodically reporting

18. main elements of the tender Evaluation TCA VENet Formal Evaluation Content Evaluation Concept Evaluation Process Evaluation Structural Evaluation Result Evaluation Gender Mainstreaming VENet Model European Mainsteaming Spport System Internet Platform Final Activity Gender Mainstreaming – cross-sectional topic

19. formal evaluation - concept • Concept analysis • Assessment of the codified project structure and plan • Assessment of management tools • Assessment of communication tools • Data • project documents (e.g. proposal) • meetings • protocols (e.g. steering group) • interviews

20. formal evaluation - structure • Structural analysis • Focus: transnational cooperation als system of interacting organizations resp. individuals • Assessment of flow of communication • Assessment of the cooperation patterns • Data • Hard facts: contribution on formal level (reports, website,..) • Soft facts: • Social network survey (x2, email) • Satisfaction survey (x2, email) • Event questionnaire (x5, at every project meeting) • Interviews • Documents, Internet Platform (?),….

21. Social network analysis (SNA) • SNA – research method that focuses on the relation (communication, cooperation, exchange of information,…) between any given set of actors. • A SNA survey asks the partners to rate their own network behaviour according to different criterias (frequeny of interaction, channels of interaction, topics of interaction,…). • The answers on these questions are producing a picture of the overall network. All individual answers are displayed on the network level. • Benefit • You get SNA plots of the network structure • Key fiugures on well and not well established elements

22. network plots

23. Gender Mainstreaming • GM are the efforts we make to ensure that gender aspects and gender equality is considered in all parts and stages of the project (working plan, objectives, actions,…). • A cross sectional topic • GM aspects are considered in every part of the evaluation • and the evaluation itself • 3 main GM approaches • Equal opportunities approach  neutral treatment • Diversity approach  awareness of the different needs • Affirmative action approach  focus on women

24. Gender Mainstreaming • Data: protocols, interviews, survey questions, products,…

25. Content evaluation • Two levels • Processes • Results • Main questions • Have the intended results been reached (quantity and quality)? • Have the processes to reach these results been carried out as intended? • Did the partners contribute as scheduled? • Are there any positive or negative deviations? If so, why? • Which deductions for the following project work can be made?

26. Content evaluation • For each of the five main products • VENet model, internet platform,….. • Data • Documents • Interviews • The products themselves • Check lists resp. progress reports for the project (in cooperation with the project management) • Questions in the email survey

27. deliverables of the evaluation • Short reports before each meeting • Distributed electronicly (internet platform) • Participation at every transnational meeting (1 w.d.) • Presentation of evaluation results and workshop on the findings (1-2,5h) • 1 mid-term report • 1 final report

28. Feedback on the evaluation model What is your impression?Does the evaluation provide the information we need?GM in our project?How do we take the evaluation results up?

29. Defining the LogFrame

30. Defintion • By now the LogFrame is empty • Will be based on the transnational project plan with defined goals, milestones and responsibilities • Next Step: Filling the LogFrame with information • Results from this meeting (e.g. working plan,…) • Documents • In cooperation with transnational coordination • The rough LogFrame will increase in its density as the project goes on

31. Definition • The defined aims of the project are crucial for the evaluation! • Important: If you set up any aims, goals and objectives: • be SMART • S…specific: general aims must be broken down • M…measurable: no untouchable aims • A…acceptance: agreement among all partners • R…realistic: no trivial aims, no unrealistic visions • T…timely fixed and with fixed responsibilities

32. Definition • The LogFrame is the catalogue of the evaluation. • Beneath this we will set up the so called Terms of Reference • Kind of evaluation handbook, the practice version of the tender • Puts down the procedures of the evaluation • Commonly shared evaluation principles • Will beset up by the ZBW

33. Next Steps

34. At this meeting • Consider the LogFrame allready when working out objectives (SMART), products and processes • Evaluation grows as the project grows. • Some short interviews on the 2nd resp. 3rd day of the meeting – transnational coordinators • A concluding event questionaire in the last session of the meeting • Define one main contact person regarding the evaluation in each partner organization

35. After the meeting • ZBW will work out the LogFrame in cooperation with the transnational coordination • Set up the Terms of Reference • Set up dates for the next evaluation steps • Inform the partners about the evaluation

36. Timeline

37. Your responsibilities! • To commit and take active part in the evaluation process • To fill in questionnaires carefully • To submit documents and information when asked for • To return you contribution to the evaluation in time – respect deadlines • To evaluate the evaluation! Feedback is welcome!!

38. Contact information Thank you! Let‘s get it on! Mag. Gerd Beidernikl Zentrum für Bildung und Wirtschaft Joanneumring 5/4, 8010 Graz, Austria (valid 1st October 2005) Mail: gerd.beidernikl@zbw.at Phone: +43 / 316 / 72 17 44 -13 Fax: +43 / 316 / 72 17 44 -21 Mobile: +32 / 650 / 46 21 061