Presentation purpose

1. Indicators for Reflection (Evaluation?) of Learning Processes in Sustainability Oriented Regional Networks of Cooperation – Explorative StudyCharles University Environment Center 3-LENSUS indicators

2. Presentation purpose • overview of possibilities for evaluation of the learning processes within L4SD learning networks represented by RCE activities and experiences derived from the 3-LENSUS project to promote of regional and European learning networks L4SD • reflect the main principles and advantages of regional cooperation, with specific emphasis on the role of universities • analysis is based on the experience of promoting 3-LENSUS project goals: assist in the development and further use of the database, select best practices for wider dissemination; and how to practically implement project goals and decisions in the core activities of regional cooperation partners • indicators are a first attempt to design tools for evaluating regional cooperation which could support development beyond 3-LENSUS • main outcome of the analysis was design of an evaluation tool for selection procedure of Best Practices, and foundation to build upon for future evaluation of regional cooperation 3-LENSUS indicators

3. Context • Regional learning for sustainability - overall aim of L4SD is to empower citizens to act for positive environmental and social change. Regional Centres of Expertise (RCEs) have been developed on the principle to create a local/regional knowledge base to support L4SD actors, and promote major goals of L4SD in a resource-effective manner. • RCEs were proposed by the UNU in 2004 as a “network of formal, non-formal and informal education and learning-related institutions that addresses need for equitable partnerships between the combined expertise of communities, professions, NGOs and governments and create an innovative platform for multi-stakeholder dialogue that enables diverse groups to interact, learn collaboratively and take collective decisions and actions towards SD”. • main aim of the regional network is to support joint projects and activities and better coordination, sharing information, knowledge, expertise and experience between actors 3-LENSUS indicators

4. Five principles of regional processes towards sustainability (Graz Model for Integrative Development Processes): • leadership, social network, participation, learning and research. • principles are interrelated e.g. to achieve a supportive social network, a shared vision needs to be built up through the participation of people. • principles are suitable for the evaluation of processes; e.g. RCEs evaluate themselves on the basis of this tool. • Education v learning: educational outcome versus process oriented approach • RCE process maybe the most significant global scale experiment in social learning yet created, i.e. not a transmissive and outcome oriented approach relating to instruction and teaching, but rather learning as a concept stressing the process through which information is integrated. • Learning possesses reflexive character, it has to encounter an active, participative (cooperative) and communicative aspect and leads to a change. 3-LENSUS indicators

5. The principle of education and learning in regional cooperation • both focused on the same goal: developing competencies and capacities. • integrative development processes towards sustainability in regions incorporate a transformative dimension and hence the role of the learning environment is important as individuals, organizations, social groups, citizens and diverse stakeholders learn from each other and external parties. • sustainability oriented educational methodologies are also interdisciplinary, are value oriented and feature a local and global troubleshooting orientation. In this context, education and learning are strongly integrative with the other principles of the Graz Model and imply sustainability as a learning objective. 3-LENSUS indicators

6. The role of HE in regional cooperation • RCEs represent an effort to reassess the links of universities with the outside world • By including universities within RCEs, they become agents in social learning processes, they invite people to learn cooperatively, over time, from each other • As a result, theory and praxis are integrated within RCEs to bridge the gap between knowledge, pro-SD values and unsustainable actions, practices and policies • The problem • What is the most appropriate way of engagement in the process? • feedback about transformation in terms of innovative practices, environmental improvements and social change that occurs within a region and within universities themselves is the focus • but most RCEs (and other forms of regional cooperation) are not yet in the stage that some “real outcomes” are available. • So, we should concentrate on characteristics of the learning process in the region (that includes potential for transformative regional development); information about learners involved, their characteristics and relations would be valuable, too. 3-LENSUS indicators

7. Indicators as a tool for reflection on sustainability oriented (learning) processes • are a normative tool created to be milestones on the path towards some intended state. They capture important features of a transformation process, express the information in a simple way and communicate it to those who will use it for policy making purposes. • provide important new information about an issue, process or condition, reflect it, and are used as an information resource for practical actions to undertake change. • They show how well a system is working, document success of some project or strategy, provide information about its progress, stages and best strategies to achieve the goal. They assist in planning processes, improving them, indicate progress achieved • for 3-LENSUS purposes, they provide feedback that could improve desired (sustainability oriented) development, help to deal with uncertainty and change and also illustrate the complex nature of sustainable development and L4SD. 3-LENSUS indicators

8. Nature of indicators • Indicators are varied, but there are certain characteristics that effective indicators have in common: • they should be relevant, • easy to understand and interpret, • representative and reliable. • they need to be obtainable from reliable sources and available for a feasible cost. • methodologically they must be based on the available data, and be constructed only as fine-scaled as the (evaluation) question requires, and the simplest thing that can be measured and which provides an adequate response to the question should be used • indicator concept is not an abstract one: they should ultimately be used to justify policymaking 3-LENSUS indicators

9. Types of indicators • Many different types distinguished by authors for strategy implementation: • Checklist indicators that provide information on initial measures taken in order to implement the concept • Input indicators for a broader spectrum of activities taking place (e.g. amount of money invested in the materials) • Output indicators provide information on the results of these activities (e.g. performance of trained teachers, number of businesses involved in ESD projects, teaching manuals). • Outcome indicators capture the possible impact resulting from the implementation of the particular strategy, especially its qualitative aspect in terms of values, attitudes and choices in favour of SD (e.g. learning outcomes resulting from ESD partnerships). • Baseline indicators: identify the status of the overall picture of the issue under examination • Context indicators: identify the existence of support systems • Process indicators: identify the existence of substantial processes and activities • Learning indicators: promote learning and reflection on the issue • Impact indicators: assess impacts that result from the efforts • Performance indicators: assess the change in the status of the overall picture related to the issue in a region or country 3-LENSUS indicators

10. Types of indicators cont. • from the 3-LENSUS point of view, a process oriented approach to assess the role of universities in a region is more appropriate for analysis than an output or outcome oriented approach, which nevertheless could bring some benefits to regional partners as well. • Regarding the long term vision of SD (in this context – potential or real change or innovation based on social transition), we should observe social learning processes started by an activity or project as they could lead to a final intended state (while the impact of the projects or activities might still be difficult to observe). 3-LENSUS indicators

11. Selecting 3-LENSUS project indicators • proposed LENSUS task was to develop qualitative indicators to measure quality of the educational activities offered through VCSE and 3-LENSUS, such as competence development for critical thinking and transdisciplinary approach to SD themes. • in practice, the efforts to develop an indicator set were more closely connected to the development of the L4SD project (and regional cooperation) database and were aimed at evaluating or rather comparing the entries. Thus, our focus shifted towards learning processes within regional cooperation and selection of its best practices. 3-LENSUS indicators

12. Indicators for evaluation of the database entries were selected in three stages: • L4SD indicators. • A comprehensive set of questions on RCE performance was developed to evaluate an educational system’s performance to “compare national performances; measure effectiveness of innovation (programs) interventions; assist self-evaluation of educational institutions and educational measures”. • In the beginning, usability of the existing ESD indicator set was explored - some of its definitions were found useful but many differences between the ESD concept and learning processes taking place on the regional level were identified. Our priority was not evaluation of the educational system but rather the social processes promoted by learning. • Conclusion: ESD indicators derived with regard to the potential transition of the educational system (national system, school system etc.) towards sustainability and existing indicator set does not sufficiently take into account non-systemic features : social learning processes, regional SD oriented cooperation of universities, change or innovation outside educational systems, research as a driving force. 3-LENSUS indicators

13. 2. Outcome indicators – first database • Second attempt and research design concentrated on the role of universities in RCEs and in multi-stakeholderorganisations. Their traditional fields of activities were explored: expertise and training capacities, and know-how transfer towards local stakeholders. • research design was based first on a broad internet review. Institutions/projects with the most significant outputs were chosen – contacted and asked for clarifications. • The categories below were created and projects classified according to output: • ACTIVITIES of the university in the region (description – open question) • CONTRIBUTION of university to RCE (Yes/No): • Source of Technical Expertise (Research) • Changes in University Practices and Acting as a Model of Sustainable Practices • Coordination, Promoting and Enhancement of the Engagement of Local Authorities • Transformative Education (Training) • Community Awareness (Seminars - Conferences) • WEBSITE info and CONTACT completed the survey and could serve to verify the results. • 42 entries made. However, very few ticked the option “Changes in university practices and acting as a model of sustainable practices” which is the most interesting one (comprising an internal process that has potential for long term impact). Generally, all of the criteria were not particularly specific and needed to be elaborated further, which was undertaken in stage 3. 3-LENSUS indicators

14. 3. Process and output indicators – database created by Graz team • experiences from the first two were used and combined with the effort to build the database 3-LENSUS indicators

15. Development of indicators • When constructing indicators, at least four stages have to be undertaken: • first the issue to be observed and its most important aspects should be ascertained, • evaluation question formulated, • then the indicator (or indicator system so that assessment would be more complex) should be derived and justified, • and finally results (evaluation, reflection) used for change in the system observed. • It’s always important to plan well. Below is an overview of indicators’ construction from the stage of planning and qualitative description of the actual state of play to the final outcome of the certified indicator. 3-LENSUS indicators

16. 3-LENSUS indicators

17. 3-LENSUS indicators

18. 3-LENSUS indicators

19. 3-LENSUS project database • Key 3-LENSUS project outcome: creation of an electronic database to act as a virtual warehouse of Europe-wide regional learning projects related to Learning for Sustainable Development (L4SD). • Projects had to exhibit the following characteristics: • have a multi-stakeholder orientation, i.e. one university/research institution and at least two other regional actors involved • display an element of regional sustainable development • be focused on lifelong learning • Indicators embedded in database form to assess all projects registered by a given date to provide a transparent basis for selecting the best examples of L4SD practice. • A simple value scale was designed for each individual indicator that was transparent and could be easily applied by any partner to produce virtually the same result. 3-LENSUS indicators

20. Design • Indicators were: • of both a qualitative and a quantitative character, • values created were either numbers that corresponded to numerical answers in the database form, or a simple 1-3 Likert scale to determine the quality and extent to which the pertinent question in the database form was answered. • Draft design was submitted for peer review, resulting in recalibrations to ensure the qualitative indicator values were of equal weighting and the range of quantitative indicator values were more comparable to each other. 3-LENSUS indicators

21. Process and output indicators – database created by Graz team experiences from the first two was used and combined with the effort to build the database 3-LENSUS indicators

22. List of indicators 3-LENSUS indicators

23. Social network characteristics 3-LENSUS indicators

24. Learning indicators – L4SD aspects 3-LENSUS indicators

25. Learning indicators – e-learning network characteristics focused on their social functions 3-LENSUS indicators

26. Research indicators – reflection of learning, social and other innovative processes 3-LENSUS indicators

27. Sustainable development indicators 3-LENSUS indicators

28. Work with database entries • a total of 26 projects were registered by the start of June 2010 • all 17 indicators to all registered database projects • other members of the 3-LENSUS project team tested the results on a sample of the projects - found to correspond overall • discussion over how to differentiate between projects that answered the optional questions regarding eLearning and research and those that did not - they remained optional questions in recognition that not all projects were likely to have both elements. Final analysis showed no difference to overall project rankings whether points for the optional questions were included or excluded (but they provided examples of best practice in eLearning and research for L4SD). 3-LENSUS indicators

29. Evaluation • maximum points = 75 • final selection of best practices not based solely on point rankings • projects registered by 3-LENSUS partners not considered • respondents asked to self-select as an example of one of five categories: • regional development • multi-oriented learning approaches • multi-stakeholder involvement • empowerment • transdisciplinarity • examples of eLearning and research • Result: six examples of best practice selected • Plus: two further case studies added to achieve regional balance among 3-LENSUS partners 3-LENSUS indicators

30. Conclusion • first steps taken towards reflecting on forms of regional cooperation among diverse stakeholders (with special focus on universities) aimed at sustainable development innovations and change • some characteristics of intended regional SD objectives were identified: learning processes with the potential to cause change related to (reflective) research practices and which occur within networks of cooperation of diverse stakeholders (thus demonstrating a strong social dimension) • the task of documenting these processes and features and distinguishing their particular qualities (and scaling them) was undertaken in order to select best practices and make them publicly available for those who want to be inspired. • an objective set of indicators describing (the hidden attributes of) the most important characteristics was developed, tested and used for evaluation of database entries 3-LENSUS indicators

31. Constraints • a small number of characteristics were used to describe a very complex issue– it was not feasible to require respondents to complete an overly extensive database questionnaire. Some important features were necessarily missing. • only a relatively small number of entries was analysed – this has an impact on validity • the desired state was described as a process, not an outcome – some of the questions were not concrete enough 3-LENSUS indicators

32. THE END! 3-LENSUS indicators