Evaluations & Assessment 2003

1. Assessing to Support New Learning Outcomes: Assessing for Graduate CapabilityAcquisition Evaluations & Assessment 2003 Sally Kift, Assistant Dean QUT Law Faculty

2. Overview • Acknowledge input of large team on Large Grant. • What’s the assessment issue? • What are the areas on which we are focussing and why? • Developing an assessment framework • Later session on this today • Look briefly at the areas of concern. • What we have achieved/still to do. • Discussion.

3. What’s it all about? • In embedding graduate capabilities, questions arose re • Validity and reliability of traditional assessment practices and their transferability to the new imperatives of assessing authentic learning tasks. • How can we assure the quality of our assessment of students’ capability development? • The (tentative) answer is that - we should be able to assure the quality of assessment if it satisfies certain criteria (as distilled from the educational literature)

4. In four project areas: - • Identified 4 areas of capability development that have proved extremely challenging to assess – • Embedding Indigenous content and perspectives; • Development of oral communication (esp negotiation, advocacy, tutorial participation & interviewing for internal and external cohorts); • Infusion of ethical values and knowledge; • Teamwork in large classes (internal and external cohorts)

5. What does that mean? Session later today • Developed assessment framework to provide theoretical basis for identifying models of best practice • Found assessment practices in the project areas • both w/i and o/s QUT. • THEN – evaluate these assessment tasks against draft frameworkto distill relative strengths and weaknesses(some egs of evaluation on website) • This with a view to developing best practice models for trialling in 2ndstage • Re Indigenous area, content reviewed as well

6. Eg: Oral communication • 5 specific communication skills being addressed: • Interviewing • Tutorial performance • Negotiation • Advocacy • OK • Oral presentations • New criteria sheet; and • Instructions to tutors & students (in core 1st yr) Improved negotiation exercise in core 2nd yr • New assessment criteria (core 1st yr); • ‘Student Assessment Handbook” for tutors & instructions for students; • Student video (o/s grant)

7. Eg: Teamwork • Best practice models being developed for – • In the classroom • Outside the classroom (live interactive groups) • In the virtual classroom (predominately electronic contact) • Consider whether assessing process or product • If process models could include: • Formative or summative; optional or compulsory • Self, peer or teacher assessment • Allocating individual or group marks

8. Eg: Ethics – • Very challenging • Connections with Indigenous project area • Legal ed currently utilises Bloom taxonomy to measure cognitive domain achievements (ie “professional” aspect of ethical knowledge) • Audit revealed only one unit in Law (cf Justice) as having structured form of assessment that includes both cognitive and affective components. • Have had to devise a matrix of criteria and standards …2/

9. Ethics – Reloading the Matrix • The matrix of criteria and standards – referenced to observable characteristics or qualities in assessment materials which are themselves coupled to accepted taxonomies of achievement • Attempt to maximise the dependability and reliability of assessment. • Have used Facione & Facione (1994) and Bloom et al (1964)

10. WarnerBros The Matrix CRITERIA Criteriamapped to increasing levels of achievement (Bloom 1 to 6) - ie leftmost criteria reflect “low-level” Bloom achievement (eg knowedge, comprehension), while rightmost criteria reflect highest level of Bloom achievement (synthesis, evaluation) Standards mapped to increasing levels of achievement from Facione & Facione 1 –4 & referenced to observable & objective descriptors reflecting differences across Facione– eg “locates ALL rules’”(L4); “locates MOST rules” (L3); “locates SOME rules” (L2) & “does not locate rules” (L1) STANDARDS

11. Re Indigenous Project Area(With ors across QUT) • What is an Indigenous perspective (cf content): Access understanding through – • the cultural construction of whiteness (eg explore own racial position) & • consultation with Indigenous persons (latter to move beyond problematising and essentialising) • Assessment: eg reflective assessment to chart attitudinal change informed by knowledge development and incorporating higher order analysis. • Also Indigenous employment and staff development

12. Anything else? • Grant websitea major dissemination strategyhttps://olt.qut.edu.au/law/ASSESSMENT/ • Ethics approval for the research undertaken by Project team members • Staff survey and student focus group • Pleasing no of pubs: eg 4xE1, 4xC1,15xATN • Including Casual staff (2 RAs and training) • Links across Faculty and across University • Positively received 2003 Law Discipline Review

13. Still TO DO: • A LOT!!! • Trial the new materials • Survey their success with students and staff and analyse • Refine assessment framework • Produce some training videos to assist in both teaching oral communication skills and in modelling application of assessment criteria.

14. The Pretty Website –

15. Assessing to Support New Learning Outcomes: Assessing for Graduate CapabilityAcquisition Evaluations & Assessment 2003 Sally Kift, Assistant Dean QUT Law Faculty