How do we design Programmes at Third Level using Learning Outcomes within the Bologna Framework?

1. How do we design Programmes at Third Level using Learning Outcomes within the Bologna Framework? Presentation 2 20 May 2016 Don State Technical University Rostov-on-Don, Russia. Dr Declan Kennedy, Department of Education, University College Cork, Ireland. 1

2. What are Programme Learning Outcomes? How do I write Programme Learning Outcomes? How do I link Programme Learning Outcomes to Module Learning Outcomes. How do I design programmes using Learning Outcomes within the Bologna Framework? 2

3. Writing Programme Learning Outcomes Programme learning outcomes are learning outcomes that describe the essential knowledge, skills and attitudes that it is intended that graduates of the programme will be able to demonstrate. The rules for writing learning outcomes for programmes are the same as those for writing learning outcomes for modules. The general guidance in the literature is that there should be 5 – 10 learning outcomes for a programme and that only the minimum number of outcomes considered to be essential be included. 3

4. Two types of Programme Learning Outcomes The first type of learning outcome refers to those learning outcomes that can be assessed during the programme, i.e. within the various modules. “Aspirational” or “desirable” learning outcomes indicate what a good quality student would be expected to achieve by the end of the programme. This type of learning outcome may not be assessed at all but gives an indication to employers and other agencies the type of standard of practical performance that graduates of the programme will display at the end of the programme. 4

5. Example of Programme Learning Outcomes [BSc(Ed)] On successful completion of this programme, students should be able to: • Recognise and apply the basic principles of classroom management and discipline. • Identify the key characteristics of excellent teaching in science. • Develop comprehensive portfolios of lesson plans that are relevant to the science curricula in schools. • Evaluate the various theories of Teaching and Learning and apply these theories to assist in the creation of effective and inspiring science lessons. • Critically evaluate the effectiveness of their teaching of science in the second-level school system. • Display a willingness to co-operate with members of the teaching staff in their assigned school. • Foster an interest in science and a sense of enthusiasm for science subjects in their pupils. • Synthesise the key components of laboratory organisation and management and perform laboratory work in a safe and efficient manner. • Communicate effectively with the school community and with society at large in the area of science education.

6. Further Example of Programme Learning Outcomes[Undergraduate Engineering Degree] On successful completion of this programme, students should be able to: Derive and apply solutions from knowledge of sciences, engineering sciences, technology and mathematics. Identify, formulate, analyse and solve engineering problems. Design a system, component or process to meet specified needs and to design and conduct experiments to analyse and interpret data. Work effectively as an individual, in teams and in multi-disciplinary settings together with the capacity to undertake lifelong learning. Communicate effectively with the engineering community and with society at large. 6

7. Further Example of Programme Learning Outcomes On successful completion of this programme, students should be able to: Perform problem solving in academic and industrial environments. Use, manipulate and create large computational systems. Work effectively as a team member. Organise and pursue a scientific or industrial research project. Write theses and reports to a professional standard, equivalent in presentational qualities to that of publishable papers. Prepare and present seminars to a professional standard. Perform independent and efficient time management. Use a full range of IT skills and display a high standard of computer literacy. [Postgrad Comp Sc degree] 7

8. Skills in broad sense Cognitive Note the overlap! Affective

10. Beware of this Tuning Publication! The authors of this publication: • Have invented their own incorrect definition of learning outcomes and give incorrect information on how to write Learning Outcomes • Use a definition of Competences which is in conflict with the definition of Competence used in the European Qualifications Framework. • Are confused about the relationship between Competence and Learning Outcomes and Incorrectly links learning outcomes to competences • Confuses the concept of learning outcomes with the assessment of learning outcomes. The authors do not appear to realise that the extent to which a learning is achieved is measured by the assessment of the learning outcome. • Incorrectly link learning outcomes to particular levels in degree programmes. • Many of the exemplars given of Learning Outcomes are incorrectly written. • Give misleading statements implying that in order to be compliant with Bologna you must be compliant with Tuning. The authors fail to point out that the Bologna Process does not require any compliance with Tuning project. • Fail to point out the problem with Competences and Quality Assurance Systems. • A dreadful publication!

11. Gibbs, A, Kennedy D and Vickers A (2012) Learning Outcomes, Degree Profiles, Tuning Project and Competences. Journal of the European Higher Education Area 15 (5) 71 – 87

12. The relationship between competences, objectives and learning outcomes is discussed by Hartel and Foegeding (2004) in area of Food Engineering. In this paper they define competence as “a general statement detailing the desired knowledge and skills of students graduating from our course or program”. Relating competences, objectives and learning outcomes

13. Competence: • The student should be able to use the mass and energy balances for a given food process. Objectives: • Understand scope of mass balances in food processing systems. • Understand appropriate use of mole fractions and mass fractions in mass balances Learning outcomes: • Describe the general principles of mass balances in steady state systems. • Draw and use process flow diagrams with labels on flow streams for mass balance problems. • Solve mass balance problems associated with food processing operations. • Design and solve mass balances for complex process flow systems, including batch mixing problems, multiple stage flow problems, problems with multiple inflows and outflows, recycle streams and multiple components, and processes where chemical reactions take place. Hartel and Foegeding (2004)

14. Van der Klink and Boon (2002) describe competence as a “fuzzy concept” On the positive side they state it is a “useful term, bridging the gap between education and job requirements”. Competence – a “fuzzy” concept(Van der Klink and Boon)

15. Van der Klink and Boon (2002) attempt to trace the different interpretations of the concept of competence within the educational systems of various countries: There is considerable confusion about what competency actually means… First, differences can be observed between nations along the lines of different national educational policies and different types of relations between education and the labour market, many of which have an historic origin. In the British approach it refers to the ability to meet the performance standards for functions and professions such as those developed for National Vocational Qualifications (NVQs) in the UK. In the USA, competencies refer to the skills, knowledge and characteristics of persons, that is traits, motives and self-concept, which contribute to performance excellence. ….. More than in the UK or the USA, the German perspective stresses a holistic view of competency. It is not just a random collection of skills and knowledge. Competencies are defined as integrated action programmes that enable individuals to perform adequately in various job contexts within a specific profession (Van der Klink and Boon, 2002)

17. Conclusions re Competence • There is no single definition of the term competence. Descriptions of the term competence range from that of a broad overarching attribute to that of a very specific task. This is in contrast with the clear definition of the concept of a learning outcome found in the literature . • One of the big problems encountered when using competences is that there does not appear to be any clear guidelines on how they should be written. In contrast to this, the guidelines for writing learning outcomes are very clearly laid out in the literature. • In general, if someone achieves a Learning Outcome they reach a level of competence. Competence may be viewed as a result of achieving a set of Learning Outcomes in the workplace. • Achieving of Learning Outcomes is a stage on a way to becoming competent, i.e. Learning Outcomes and Competences can complement each other but we must be careful how we define competences.

18. “One of the reasons for the debate about the usefulness of managerial competence may be the soft focus and blurred edges of the term ‘competence’. Social science has the habit of taking a word from our common vocabulary and altering the meaning by it adoption as a technical or academic term. This process is still happening to ‘competence’ and a common consensus has yet to be established as to what the word should mean when used in management applications. (Brown, 1994)

19. “The fact that the concept of competencies serves as a remedy for solving rather different problems probably has to do with its diffuse nature. It is actually an ill-defined concept with no clear content, thus allowing ample interpretations. This major vagueness is partly caused by the application of the concept in various countries, different settings and for different purposes. Its vagueness is probably at the same time the explanation for its prominent status today but it makes it difficult to use the concept as a sound cornerstone for designing HRD [Human Resource Development] and educational practices.” (Van der Klink and Boon, 2003)

20. Recommendations • It is obvious from the literature that within certain professions, the term competence has a shared meaning. Hence, there is no problem with using the concept of competence since there is a common understanding of its meaning among the members of that profession. • The problem arises when the term competence is used in a general context without defining what is meant by the term. • Given the considerable confusion in the literature, if the term competence must be used, then its meaning needs to be clearly defined for the context in which it is being used.

21. Therefore, in order to avoid confusion it is recommended that when using the term competence, the following guidelines should be followed: • State the definition of competence that is being used in the particular context. • To ensure clarity of meaning, write competences using the vocabulary of learning outcomes, i.e. express the required competence in terms of the students achieving specific programme learning outcomes or module learning outcomes.

22. Since there is not a common understanding of the term competence, learning outcomes have become more commonly used than competences when describing what students are expected to know, understand and/or be able to demonstrate at the end of a module or programme. The “fuzziness” of competences disappears in the clarity of learning outcomes! In short, use Learning Outcomes to clarify what is meant by a statement of Competence. In short, use Learning Outcomes to clarify what is meant by a statement of Competence. In short, use Learning Outcomes to clarify what is meant by a statement of Competence.

23. Examples of Language used when writing Programme Learning Outcomes Knowledge • Discuss a wide variety of…. • Outline a broad range of fundamental concepts….. • Describe the theories and concepts in the field of…. • Identify a range of processes used in…. • Discuss relationships between the various areas of….. • Examine current theory in the area of… • Critique modern theories in the area of …. • Examine and evaluate current problems in the area of ….. etc.

24. Skills (in broad sense – as in European Qualifications Framework) • Apply a range of techniques to solve… • Modify techniques in the area of ….. to solve…. • Link theory with practice in order to… • Analyse data to facilitate decision making in the area of… • Utilise appropriate methods, skills and techniques to solve…. • Exhibit proficiency in using a broad range of routine laboratory techniques in the field of….. • Recognise limitations in the areas of… • Link relevant theories to the development of a design to…. • Utilise appropriate models and techniques in the area of …. to ……… • Select and apply the most suitable techniques to solve problems in the areas of…..

25. Skills (continued) • Apply appropriate decision making to achieve high standards of performance in the area of…. • Identify appropriate solutions to plan future developments in the area of… • Select appropriate instrumental methods to… • Utilise existing strategies to design…… • Evaluate existing problems in the area of ….in order to…. • Initiate research ides and evaluate research related publications in the area of…. • Implement work objectives and exercise leadership in …..

26. Skills (continued) • Combine technical skills to define a problem in the area of ….. and implement suggested solutions to…. • Apply technical knowledge in the area of … to solve problems related to… • Recognise existing strategies to facilitate solutions in the area of….. • Formulate options and solutions to… • Diagnose problems and suggest solutions in the area of…. • Transfer methodologies to new applications in the area of….. • Integrate a range of acquired transferable skills such as …….

27. Skills (continued) • Develop your personal capabilities in order to …. • Engage with new developments and practices in order to… • Recognise the need for life-long learning and professional development in the area of…. • Identify and address continuing requirements for professional development in the area of • Contribute to the future development of the field of……. • Interpret relevant regulations in the area of .. • Recognise the relationship between science, technology and society in the area of…. • Critically appraise research in the area of…. and evaluate the work of peers. etc.

28. Attitudes • Display an appropriate standard of professional practice in the area of… • Embrace responsibility for the welfare of others…. • Display personal ethical standards in the area of… • Articulate and defend the need for personal responsibility and ethical considerations in the workplace for….. • Work ethically and professionally as part of a team….. • Act appropriately in unfamiliar situations in the area of…. • Apply appropriate ethical considerations when … • Work as a member of a team to manage…. • Accept accountability for achieving… • Work autonomously or as a member of a team in order to….. etc.

29. HOW DO I MAP PROGRAMME LEARNING OUTCOMES TO MODULE LEARNING OUTCOMES?

31. Example: Programme Learning Outcomes for MSc in Toxicology • PLO1 Discuss the interdisciplinary relationship between the various specialised areas of Toxicology in providing informed scientific debate of current, topical issues in Toxicology. • PLO2 Evaluate current scientific problems which are at the forefront of Toxicology, Pharmacology, Biotechnology, Molecular Biology, Environmental Science, Nanomaterials and Food Science. • PLO 3 Exhibit proficiency in a broad range of routine laboratory experimental and advanced specialised research techniques in the field of Toxicology. • PLO 4 Select and apply the most suitable experimental techniques necessary to solve a Toxicological problem and develop new skills to a high level in emerging experimental techniques in the field.

32. PLO 5 Integrate the range of acquired generic, transferable skills (e.g. business skills, communication skills, numerical and statistical skills, ITC, problem- solving, decision making, management, team-work, innovation and entrepreneurship) necessary for graduates working as a Professional Toxicologists. • PLO 6 Initiate research proposals, interpret and critically evaluate research related publications in the domain of Toxicology and demonstrate leadership skills as part of an interdisciplinary scientific research/industrial or management group. • PLO 7 Contribute professionally to the future development of the field of Toxicology through applied study or further research. • PLO 8 Critically appraise scientific research and apply integrated approaches to accurately assess and critically evaluate the work of scientific peers.

33. Example of Mapping of Programme Learning Outcomes on to Module Learning Outcomes Programme Learning Outcomes Module Learning Outcomes

34. What are the benefits and potential problems of Learning Outcomes? 34

35. The benefits of Learning Outcomes • Help to explain more clearly to students what is expected of them and thus help to guide them in their studies – motivation and sense of purpose • Help teachers to focus more clearly on what exactly they want students to achieve in terms of knowledge and skills. • Help teachers to clarify their thinking about what they want to achieve and the common language of learning outcomes helps to facilitates discussion with colleagues. • Helps to define the assessment criteria more effectively. • Help to provide guidance to employers about the knowledge and understanding possessed by graduates of programmes, i.e. show the value of the programme in terms of programme learning outcomes and module learning outcomes. • Help to start discussion on Teaching and Learning in third level institutions. 35

36. Transnational Implications of Learning Outcomes Learning Outcomes have applications at three levels: • Local level – individual third level institutions for describing modules and programmes. • National level – within each country for describing National Qualification Frameworks and systems for Quality Assurance. • International Level – facilitate clarity and transparency of qualifications and mutual recognition of qualifications. Learning outcomes provide the common language in the clear description of programmes and modules. The ECTS system provides the common currency.

37. International Recognition and Mobility “Learning outcomes are important for recognition, since the basis for recognition procedures is in the process of shifting from quantitative criteria such as the length and type of courses studied, to the outcomes reached and competencies obtained during these studies. The principal question asked of the student or the graduate will therefore no longer be “What did you do to obtain your degree?” but rather “What can you do now you have obtained your degree?”. This approach is of more relevance to the labour market and is certainly more flexible when taking into account issues of lifelong learning, non-traditional learning and other forms of non-formal educational experiences” Council of Europe, 2002.

38. Potential problems with Learning Outcomes • Could limit learning if learning outcomes written within a very narrow framework – lack of intellectual challenge to learners. • Learning outcomes should not be reductionist but rather expansive and intended to promote the higher order thinking skills. • Danger of assessment-driven curriculum if learning outcomes too confined. • Could give rise to confusion among students and staff if guidelines not adhered to when drawing up learning outcomes, etc. 38

39. Having designed modules and programmes in terms of learning outcomes, we must now find out if our students have achieved these intended learning outcomes. How will I know if my students have achieved the desired learning outcomes? How will I measure the extent to which they have achieved these learning outcomes? Therefore, we must consider how to match the method of assessment to the different kinds of learning outcomes e.g. a Learning Outcome such as “Demonstrate good presentation skills” could be assessed by the requirement that each student makes a presentation to their peers. When writing learning outcomes the verb is often a good clue to the assessment technique. Assessment of Learning Outcomes

40. Formative Assessment • Assessment FOR learning – gives feedback to students and teachers to help modify teaching and learning activities, i.e. helps inform teachers and students on progress being made. • Assessment is integrated into the teaching and learning process. • Clear and rich feedback helps improve performance of students (Black and Williams, 1998). • Usually carried out at beginning or during a programme, e.g. coursework which gives feedback to students. • Can be used as part of continuous assessment, but some argue that it should not be part of grading process (Donnelly and Fitzmaurice, 2005) 40

41. Summative Assessment • Assessment that summarises student learning at end of module or programme – Assessment OF Learning. • Sums up achievement – no other use. • Generates a grade or mark. • Usually involves assessment using the traditional examination. • Only a sample of the Learning Outcomes are assessed – cannot assess all the Learning Outcomes. 41

42. Continuous Assessment • A combination of summative and formative assessment. • Usually involves repeated summative assessments. • Marks recorded. • Little or no feedback given. 42

43. Assessing learning outcomes: points to consider • Learning outcomes: “statements of what a student will know, understand, and/or be able to do at the end of a learning experience”. • Having described your courses in terms of learning outcomes, you now want to find out whether students have achieved them • Specify the types of student performance that will provide evidence of learning

44. Paper/thesis Project Product development Performance Exhibition Case study. Clinical evaluation Oral exam Interview Research assignment Portfolio Others?? Common assessment techniques in Higher Education

45. One of the big challenges is to move away from assessment based solely on terminal exams – not intelligence fair, forcing all kinds of learning to fit into the paper and pencil test straight jacket. • Purposes of Assessment: feedback, diagnosis, motivation, guidance, learning support, selection, grading, certification, progression, professional recognition, gate-keeping…..

46. Learning outcomes Demonstrate good presentation skills. Formulate food product Identify an area for research Identify signs and symptoms of MS in a patient Assessment? Multiple choice questions Prepare a 1000-word research proposal Lab-based project Make a presentation to peers Example of Matching the Assessment to the Learning Outcome

47. Assessing your assessment – is it doing the job you want it to do? Is it comprehensive?

48. To what extent has each Learning Outcome been achieved? • Not a question of “yes” or “no” to achievement of Learning Outcomes. • Rubric: A grading tool used to describe the criteria which are used in grading the performance of students. • Rubric provides a clear guide as to how students’ work will be assessed. • A rubric consists of a set of criteria and marks or grade associated with these criteria.

49. Linking learning outcomes and assessment criteria.

50. Important to ensure that there is alignment between teaching methods, learning outcomes and assessment criteria. Clear expectations on the part of students of what is required of them are a vitally important part of students’ effective learning (Ramsden, 2003) This correlation between teaching, learning outcomes and assessment helps to make the overall learning experience more transparent and meaningful for students. For the good teacher, learning outcomes do not involve a “paradigm shift”. Teaching for understanding Learning outcomes There is a dynamic equilibrium between teaching strategies and Learning Outcomes. 50 50