Evaluation – some ideas and suggestions

1. University of Bielefeld Evaluation – some ideas and suggestions Elke Wild • Educational psychologist, • focus on motivation, self-regulated learning and designing powerful problem-oriented learning environments • member of the national counseling group in PISA 2004-> assessment of students‘ attitudes and perceptions of family and school environment • I‘m in charge of the university, responsible for the organizational development

2. Evaluation - Definition • Evaluation is often seen as unnecessary or threatening because it‘s used to justify allocations of ressouces, responsibilities etc. • The actual purpose of evaluation, however, is to • assess the effects of your interventions (observation) and • to value them in terms of standards (by referring to objective criterias, temporal and/or social comparisons; most common are social comparisons by referring to matched control groups) • Thus, evaluation is a means to • obtain or preserve (attributed or self-defined) standards of quality („Qualitätssicherung“) • to get feedback how to improve quality / desired goals or standards.

3. „traditional“ vs. „innovative“ kinds of evaluation • researcher-designed field studies aiming at the documentation of success or failure (empirical support of theoretical assumptions) • „collaborative innovation“ as an alternative (and probably superior) form of evaluation; it means that practicians contribute to • the definition of goals (theoretical constructs) and criterias (empirical indicators) • the development of conceptual features (including scripts, materials, assessment instruments) • the implementation of measures (i.e., exchanges in the course of teacher education) • the kind of evaluation (decisions concerning the relative importance of goals, the dedication of time and money)

4. proposal held out the promise of two kinds of evaluation • formative evaluation means an ongoing assessment; emphasis is on the continous and qualitative feedback concerning the process -> evaluation as a kind of coaching, counseling designed to optimize interventions/projects • final evaluation („summative Evaluation“) means to assess goals attaiment at the end of an intervention in order to come to definite conclusions concerning the benefit of the project („product oriented evaluation“)

5. issues to clarify • frequency / times of measurements: before projects starts (expectations)? after the first run (problems, challenges)? after each run (progress, new goals?) and/ or at the end (lasting effects in terms of implemantation, transfer, dissemination)? • relevance of content areas/dimensions: knowledge (repertory of instructional strategies in different domains, languages etc.), attitudes (awareness of the „european dimension“; sensitivity for social and political issues), changes in (professional) self-concepts and motivation to continue exchange, intra- and interdisciplinary cooperation, use information technologies (internet, email) • frequency of feedback: third and sixth meeting (see proposal)? • interviews in two selected sites / countries: most „typical“ institutions/sites? most divergent institutional / organizational conditions (restrictions, resources, experiences, ....) to consider a variety of potential problems?

6. issues to clarify: target (groups of) persons • members of the organizing committee /coordinators, • mentors, • fellow worker, • guest families, • students (Refrendare), • students (Schüler), • teachers / teacher training institutions who need informations concerning the establishment of intercultural exchange-projects; strategies to distribute CD-ROMs (Nachhaltigkeit und Verbreitung – see expert comments!)

7. materials / strategies: • a) to facilitate participants‘ problem solving • b) to improve the communication within INCLUDEME-members: utility of includeme-domain, homepages, INTERWISE conference system; • c) to ensure dissemination: utility of handouts / reports / „survival kids“; instruments for ongoing self-evaluation; marketing • d) to influence probability of acceptance and transfer: informations concerning „critical issues“ (requirements that are necessary or desirable; factors that may moderate success / effects),

8. formative evaluation– methodological suggestions • questionnaires (questionnaires, online scales, diaries/portfolio as a means to foster self-monitoring) in ordera) to assess informations concerning similarities and differences between locations/countries (issue of generalizability) b) to examine effects on participants (vs. comparative students, teachers etc.)c) to get some transfer information (changes in attitudes of staff members not participating directly) • interviews (semi-structured, with selected individuals that represent participation students, mentors and coordinators) in order to identifiy challenging issues and moderating factors • expert ratings (professionals for multimedia learning environments, teachers interested in european exchanges) to assess the utility of handouts / reports / procedures / instruments used for evaluation • additional methods: tests (competencies)? observations (variance in videotyped instructional strategies)? records / files (representativenes of participants)

9. some final remarks... • result of the first evaluation (on base of the application for grant) of two experts • critical feedback: only few ideas to improve long-lasting effects and dissemination within institutions and beyond • ideas how to inform and motivate colleagues, students, classmates... • idea how to distribute CD-ROM