"Evaluation of Postgraduate Studies at ETH Zurich: Goals, Methodology and Experiences"

1. "Evaluation of Postgraduate Studies at ETH Zurich: Goals, Methodology and Experiences" ADL-Working Group: Quality of ADL Zurich, 4.10.2001 Hansruedi Frey Center for Continuing Education at ETHZ.

2. Content • Continuing Education at the Swiss Federal Institute of Technology, Zurich (ETHZ) • The Center of Continuing Education (CCE) • Quality Concept and Evaluation Methodology (Principles, Evaluation Cycle and Methods ) • Lessons to be Learned for the Evaluation of Online Post-Graduate Studies

3. Products: Organizers: Support Center: 11 Post-Graduate Studies with 300 Students ( 600 hours) • Policy Issues • Student Administration • Information and PR • Institut. & Fin. Support • Market Studies • Quality Control, Evaluation • Network for EducationalTechnology • Online Education Portal Dep. 9 Post-Graduate Courses with 150 Students ( 200 hours) Inst. Prof. 120 Further Education Courses with 5000 Participants ( 1-5 days) 1. Continuing Education at ETHZ

4. Particularitiesof Continuing Education at ETHZ • Continuing Education is Part of the Mission • Postgraduate Studies obtain High Subsidies • Target Groups are Well Educated People (Post-Diploma / Post-Master Level) • The Offer is Decentralized and Heterogeneous • Ownership and Responsibility are Decentralized • Individual PG-Studies compete in Different Education Markets • Different Client Groups have Different Career Expectations • Flexible Response to Needs / Limited Synergies

5. 2. Types of Evaluation and Responsibilities Responsibility of the Center for CE: • Market Studies for New PG-Studies and Courses • Periodical Evaluations of PG-Studies and Courses • Methodological Evaluation Support for Organizers of Further Education Courses Responsibility of the Organizers: • Monitoring of Ongoing PG-Studies and Courses • Evaluation of Further Education Courses

6. Market Studies serve to identify... Clients Client Needs Client Motivation They serve to select Content and Form. Quality Evaluations focus on... Transfer Effectiveness Efficiency They serve to improve Content and Form. Objectives of Evaluation

7. Professional and Educational Environment (Labor and Education Markets, Trends) Planning, Concept (Goals, Objectives, Concepts, Contents, Target Groups) Outcome, Utility (Transfer, Long-Term Benefit, Career, Value of the Degree) Evaluation in Context Start End Organization (Admission, Selection, Human and Material Resources) Output, Effects (Marks, Skills, Satisfaction, Efficiency, Effectiveness) Market Studies Quality Control CCE-Quality Cycle: 6 Core Elements Implementation (Didactic, Exams, Processes, Client Relations, Professional and Teaching Competence)

8. Motives for Education and for the Choice of ETHZ (Cost, Image, Content) Congruency between Required and Acquired Skills Environment Personal Knowledge Increase and Trans-fer to Work Place Age, Sex, Education and Nationality Planning Outcome Evaluation Organization Output Implementation Material Conditions: Money and Labor Input, Employer Support Employment Situation before, during and after the Studies Timing, Organizational and Infrastructural Quality Core Elements and Survey Topics

9. Planning and Environment: Market Heterogeneity

10. Outcome: Coherence of Training with the Requirements of the Work Place (View of Graduates)

11. Organization and Implementation: Client Satisfaction

12. Lessons to be Learned for ALD Strategic In-Depth Evaluation of Post-Graduate Program Ex-Ante Evaluation to clarify Needs and Potentials Continuous Monitoring to fine tune Ongoing Courses Periodic In-Depth Evaluation of Individual PG Studies Annual Self-Evaluation to improve Performance

13. Traditional Lectures: • Direct Observation/Feedback • Potential Drop-Outs take Part • Intuition of Teacher • Collective Feedback • High Potential for Immediate Teacher Reaction • Distance Learning: • e-Observation/Feedback • Potential Drop-Outs are Lost • Quantitative Behavior Data, • Individual Feedback • Limited Potential for Immediate Teacher Reacti0n Continuous Monitoring to fine tune Ongoing Courses • Goal: Immediate Adaptation of Ongoing Courses to Client Needs • Actors: Teacher, Students • Content: Course Implementation • Methods: Feedback, Simple Ratings (Smiley Model)

14. Traditional PG-Studies: • Teacher Observations • Qualitative and Quantitative Feedback • Improvement depends on Client Participation andAna-lytical Capacities of Teachers • Distance Learning: • Large Data Base on Behavior • Emphasis on Quantitative Data • Improvement depends on Systematic Data Analysis and Curriculum Development Annual Self-Evaluation to improve Performance • Goal: Improvement of Future Courses • Actors: PGS-Coordinator, Teachers, Students • Content: Output, Organization, Implementation • Methods: Exams, Surveys, Feedback-Rounds, Observations

15. Traditional PG-Studies: • Tracer and Market Studies • High Client Identification and Response • Weak Employer Response • Individual and General View • Local Market Needs • Distance Learning: • Market and Delphi-Studies • Weak Client Identification and Response • No Employer Response • Only General View • Global Market Needs Periodic In-Depth Evaluation of IndividualPost-Graduate Studies (Product Review) • Goal: Assessment of Outcome and Environmental Changes • Actors: PGS-Coordinator, Graduates, Employers, Center for C.E. • Content: Outcome, Transfer, Market Needs, Advancement • Methods: Benchmarking, Surveys, Delphi-Rounds, Controlling

16. Traditional PG-Studies: • Limited Market with known Patterns of Competition • Local Needs, Language and Culture • Government Orders and Subsidies facilitate Pricing, Matriculation and Exams • Distance Learning: • Global Market with unknown Patterns of Competition • International Needs, Language and Culture • Difficult Tutoring, Pricing, Matriculation and Examination Procedures Ex-Ante Evaluation to clarify Needs and Potentials • Goal: Product Acceptance on the Market Place • Actors: PGS-Coordinator, Potential Clients/Employers, C.f.C.E. • Content: Market Trends, Institutional Strengths, Market Strategy • Methods: Market and Delphi Studies, Surveys, SWOT-Analysis

17. Traditional PG-Studies: • Decentralized Planning and Implementation • Limited Markets • Flexible Reaction to Highly Fragmented Local Markets • International Alliances between Individual Institutes • Distance Learning: • High Initial Investment • New Technologies • Global Competition • Strategic Program Planning • Review of Organization Structure (Centralization?) • Cost Accounting Strategic In-Depth Evaluation of an Entire Post-Graduate Program (Institutional Level) • Goal: Strategic Positioning and Synergies at Program Level • Actors: University Council, Politics and Industry, Center for C.E. • Content: Strategy, Competence, Coherence, Cost, Market Position • Methods: Portfolio, SWOT, Scenario, Change M., Cost-Analysis

18. Survey Methods and Response • Surveys among Graduates (35 - 50 Questions) adapted to the Special Needs and Conditions of the Courses • Good Response, Significant Data • Generally High Appraisal • Surveys among Employers (20 - 30 Questions) adapted to the Special Needs and Conditions of the Courses • Weak Response, Insignificant Data • (in some Cases due to Job Change of Participant after Graduation) • Methodological Support for Further Education Courses (Different Types of Questionnaires) • Little Interest of Organizing Institution for Instruments and Methodological Support

19. Methodological Skills Interdisciplinary Knowledge Scientific-analytical Skills Integrative, conceptual Skills Process Orientation Presentation Techniques Writing Skills Computer Skills Personal Skills Time Management Professional Skills Information Management Marketing Process Management Technology Management Information Technology Social Skills Leadership Cooperation Skills Personal Contacts Relevant Categories for Planning

20. Course Documentation Course Infrastructure Duration Full Time, Part Time Modular/Continuous Set-Up Administration Tutoring New Learning Technologies Theory and Case Studies Active Participation Relevant Categories for Organization, Infrastructure and Implementation