Implementing the AAC&U Value Rubrics in Sakai

1. Implementing theAAC&U Value Rubricsin Sakai Sean Keesler and Janice Smith Three Canoes June 15, 2010

2. Presenters Sean Keesler, Three Canoes Janice Smith, Ph.D., Three Canoes http://threecanoes.com Three Canoes Mission Three Canoes Consulting wants your institution to become self-sufficient in using Sakai and the Open Source Portfolio (OSP) for teaching, learning, and assessment. We offer on-site and virtual services for assessment planning, ePortfolio design, OSP customization, and documentation and training for your use of Sakai. We are fellow educators with years of experience working with higher ed and K-12 institutions and have been a part of the Sakai and OSP communities since their inception.

3. Session Outline The AAC&U VALUE Rubrics Implementing the AAC&U VALUE Rubrics in Sakai Three Canoes OSP Contribution Technical Specifications Discussion

4. The AAC&U VALUE Rubrics AAC&U Association of American Colleges and Universities VALUE Valid Assessment of Learning in Undergraduate Education

5. Association of American Colleges and UniversitiesVALUE Project Part of AAC&U’s Liberal Education and America’s Promise (LEAP) initiative Project Dates: May 2007 through April 2010 Staffed by Sakai’s own Wende Morgaine, January 2008 to January 2010

6. Association of American Colleges and UniversitiesVALUE Project Project involved teams of faculty and other academic and student affairs professionals. Iterative process took place over eighteen months. Teams gathered, analyzed, synthesized, and drafted institutional level rubrics. Rubrics address 15 essential learning outcomes. Each VALUE rubric contains the most common criteria for judging student work in for the outcome. The VALUE rubrics reflect U. S. faculty expectations for essential learning regardless of institutional type, mission, size or location.

7. VALUES Project Assumptions To achieve a high-quality education for all students, valid assessmentdata are needed to guide planning, teaching, and improvement; Colleges and universities seek to foster and assess numerous essentiallearning outcomes beyond those addressed by currently availablestandardized tests; Learning develops over time and should become more complex andsophisticated as students move through their curricular and cocurriculareducational pathways toward a degree; Good practice in assessment requires multiple assessments, over time;well-planned electronic portfolios provide opportunities to collect datafrom multiple assessments across a broad range of learning outcomeswhile guiding student learning and building self-assessment capabilities; ePortfolios and assessment of work in them can inform programs andinstitutions on progress in achieving expected goals. From AAC&U VALUE Website: http://www.aacu.org/value/index.cfm

8. AAC&U Essential Learning Outcomes Intellectual and Practical Skills Inquiry and analysis Critical and creative thinking Written and oral communication Quantitative literacy Information literacy Teamwork and problem solving Personal and Social Responsibility Civic knowledge and engagement—local and global Intercultural knowledge and competence Ethical reasoning and action Foundations and skills for lifelong learning Integrative and Applied Learning Synthesis and advanced accomplishment across general and specialized studies

9. Inquiry and Analysis VALUE Rubric

10. Creative Thinking Rubric

11. Implementing the AAC&U VALUE Rubrics In Sakai Sakai doesn’t come with a default portfolio experience. There is well researched prior art. You can springboard your implementation. Use the AAC&U VALUE rubrics to start the conversation at your institution about portfolio purpose and assessment.

12. Possible Sakai Portfolio Workflow Students Choose associate learning artifacts with matrix cells according to essential learning outcomes that the artifacts address. Use a reflection form to evaluate themselves according to the relevant VALUE Rubric. Faculty Use an evaluation form containing the same VALUE Rubric to evaluate student artifacts in each matrix cell. Students Compare their self-reflection with evaluation by their faculty Use a portfolio template to share the content of the matrix with others as an indication of their learning 11th Sakai Conference - June 15-17, 2010 12

13. Inquiry and Analysis Evaluation Form

14. Completed Evaluation Form

15. Student View of Evaluation Form

16. Calculation of Average Rating

17. Values Matrix

18. VALUE Portfolio Template Ideas Create a “matrix”portfolio to harvest data from the VALUES matrix Ask faculty evaluators to sign off on each VALUE indicating student mastery VALUEs that have been mastered would appear on the portfolio with checkmarks Provide themes for students to select or alternatively encourage students to creatively illustrate their portfolios with images and captions

19. Live Demo of VALUE Rubrics

20. Three Canoes OSP Contribution Matrix and 15 VALUE Rubric forms Go to Open Ed Practices http://openedpractices.org Search for Three Canoes Or http://openedpractices.org/resource/aacu-essential-learning-outcomes-matrix-and-value-rubrics http://openedpractices.org/practice/assessing-student-learning-using-aacu-value-rubrics

21. Technical Specifications Rubric renderers Four non-zero choices Criteria: Ennumerated string values Keys Weighted averages Standard Fields

22. Rubric Renderer “Standard” form renderers don’t produce a form that looks like a rubric. They are useful starters for developers to modify Your own rubrics can be implemented as Sakai forms with a similar renderer. The renderer has some limitations and assumptions incorporated into the design. 11th Sakai Conference - June 15-17, 2010 22

23. Four Non-Zero Choices • This renderer • Creates a table with four columns • Can be modified • If an evaluator doesn’t choose one of the four choices, the score is a ZERO for that criteria. • You can make the renderer capable of handling any size table of values. 11th Sakai Conference - June 15-17, 2010 23

24. Criteria - Enumerated String Values This renderer expects each field with enumerated string values to be a row of the rubric. There may be a need for other “non-rubric” fields on the form. 11th Sakai Conference - June 15-17, 2010 24

25. Keys • Each criteria is assigned a key (sakai.key). • Keys: • Match criteria rows in the rubric table with the text input for that criteria. • Match a criteria to it’s rubric weight. • Determine the sort order of the table rows and text input fields for each criteria 11th Sakai Conference - June 15-17, 2010 25

26. Weighted Averages The result of completing the rubric form is a weighted average of the criteria scores. Setting the sakai.rubricWeight for each criteria element will allow different emphasis on each criteria. Inter-rater reliability can be achieved by automatically calculating the final rating. 11th Sakai Conference - June 15-17, 2010 26

27. Standard Field Names The final calculated score is called “rating.” The evaluator comments are called “comments.” 11th Sakai Conference - June 15-17, 2010 27

28. Our Questions for You How would you use the AAC&U VALUE Rubrics in your institution’s instance of Sakai? Do you have ideas for improving the rubric forms or the VALUES matrix? What characteristics should the VALUEs portfolio template have? Do you have any ideas about how to move the VALUE Rubrics toward portability across systems?