Assessment Matrix Workshop

1. Katherine Perez Jacqueline Peña Office of Planning and Institutional Effectiveness January 2009 Assessment Matrix Workshop

2. SLO matrix • SLOs and Corresponding Assessment Methods • Workshop: Revising SLOs and Assessment Methods • Reporting Results and Use of Results • Workshop: Writing the Results and Use of Results • PO matrix • POs and Corresponding Assessment Methods • Reporting Results and Use of Results • Workshop: Revising POs and Assessment Methods + Writing Results and Use of Results • Additional individualized assistance Overview of workshop

3. SACS Accreditation • Key SACS Deadlines: • September 10, 2009—Report due • March 8-12, 2010—Onsite Review • Key IE Deadlines: • February 1, 2009—Fall 2008 SLO matrices due • June 1, 2009—Spring 2009 SLO and PO matrices due • Continuous improvement • Institutional Effectiveness • Sound, research-based and assessment-based decisions Why Focus on Assessment?

4. Blank SLO Matrix 08-09 Link to the Mission:

5. Criteria: • Can be observed and measured • Relates to student learning towards the end of the program • Reflects an important concept • Formula: Who + Action Verb + What • Students will identify and apply the key components of a learning system. • Studentswill identify and apply the five key components of a learning system. Writing the slo

6. Principle 3.4.12 • The institution’s use of technology enhances student learning and is appropriate for meeting the objectives of its programs. Students have access to and training in the use of technology. (Technology use) • Required for all undergraduate programs • Optional for 2007-2008 SLO matrices • Required for 2008-2009 SLO matrices Technology SLO

7. General (not appropriate for SACS): • Students will use information technology to gather and disseminate information. • More Specific: • Graduates of the program will demonstrate the ability to use appropriate computer technology, software, and the Internet to complete their capstone research papers. • Much more Specific (appropriate for SACS): • Students will be able to effectively demonstrate information technology skills by locating and retrieving information on economic topics and issues, published research in Economics and related fields, and/or by finding information about the generation, construction, and meaning of economic data for their final research projects. • Students will write and present a capstone project that requires the use of Word, Excel, PowerPoint, and information technology. Sample Technology outcomes

8. Action Verbs [Who + Action Verb + What] Note: This table reflects the original Bloom’s Taxonomy Table. The modified version places evaluation as the fifth column and synthesis (creating) as the sixth column.

9. Artifact • Collection of the artifact • Where, when, and how • Census versus sampling • Criteria • Minimum standards on a rubric/scale or the percentage of correct items • Evaluation of the artifact • Faculty panel or external evaluators (reliability) • Rubrics, embedded questions, etc. Assessing the SLO

10. Direct and Indirect Measures

11. Sample: SLO + Assessment

12. Summary of Results • Direct measures • Test items • Performance as determined by rubrics • Indirect measures • Surveys and questionnaires • Interview and focus group data • Format • Narrative • Tables or charts • Analysis of results • If pertinent, explain results in a narrative form by interpreting results or using qualitative analysis of the data. • Every student learning outcome must have at least: • One set of results • One student learning improvement strategy (use of results) Reporting the results

13. All the students passed the assessment. 75% of the students met the criteria for success. Our students passed the dissertation defense on the first attempt. 75% of the students (n=15) achieved a 3 or better on all the rubric categories for the capstone course research paper. Results Activity

14. Master’s Thesis – Results Table Criteria: Students will achieve a 3 or better on a 4-point rubric on all five sections of the master’s level thesis.

15. Sample: SLO + Assessment + Results

16. Explaining your use of results

17. Target met. Will continue to monitor. • The faculty will meet and revise the three introductory courses. • A larger sample will be obtained. • We will revise the rubric and have a calibration session with the faculty prior to evaluating the student papers each semester. • A capstone course will be created that emphasizes research and thesis writing methods. • A student resource center will be created with computer equipment, writing tutors, and statisticians who can assist our students with course- and thesis-related writing and research. Use of Results Activity

18. Sample: Completed SLO matrix

19. Sample: Completed SLO matrix

20. Overview and Comparisons: • Give an overview or summary of all the outcomes together • Discuss trends that you have seen over the years • Explanations • Provide qualitative explanations for poor results or exceptionally high results • Notes and documentation affecting results • Response rate (e.g. Only 50% of the students completed the project.) • Inter-rater reliability (e.g. 2 faculty members reviewed the artifacts and the inter-rater reliability was only 60%.) • Assessment Commentary • Revise or create instrument (e.g. artifact, rubric) • Modify assessment methods (e.g. data collection, sampling, criteria, evaluation process) The summary page

21. Write or revise your Results and Use of Results. • Results: • Clearly state the results of each PO assessment (table or narrative). • Use of Results: • Explicitly state how you will use the results to improve the program during the following assessment cycle/year. • Summary Page: • Overview of results and use of results (looking at the big picture) • Trends over the last year or several years • Comparison to national or state trends or data • Explanations for poor or unexpected results • Notes or documentation that could explain the results • Assessment improvement plans Activity: Results and Use of Results

22. END OF THE SLO PORTION What Is Next? • 10-minute overview of Program Outcomes • PO writing assistance • Continued SLO writing assistance

23. Program-level outcomes • Focus on student success (not student learning) • Formula = Who + Action Verb + What • Examples: Graduatesseekingemployment in the field will find such employment within 6 months of graduation. Graduates will be satisfied with advising and mentoring services. Faculty will publish at least one journal article or book chapter every two years. Writing the PO

24. Sample: Completed po matrix

25. Take 15 minutes to review or write at least one program outcome with corresponding assessment methods, results, and use of results. • PO Formula: Who + Action Verb + What • Assessment Methods Formula: • State the instrument/method/tool (e.g. survey, national exam results). • Explain the data collection steps (e.g. when, where, how). • State the success criteria (minimum standards for success). • Results: • Clearly state the results of each PO assessment (table or narrative). • Use of Results: • Explicitly state how you will use the results to improve the program during the following assessment cycle/year. Activity: Program outcomes

26. The Ie Team • Marta Perez • Director • perezma@fiu.edu • 305-348-2733 • Maria Corrales • Coordinator • corrales@fiu.edu • 305-348-0459 • Katherine Perez • Coordinator • katherine.perez@fiu.edu • 305-348-1418 • Jacqueline Peña • Coordinator • jpena@fiu.edu. • 305-348-1367 • Mayelin Felipe • Computer Specialist • mfelip01@fiu.edu • 305-348-0115 • Karla Felipe • Computer Specialist • kgarcia@fiu.edu • 305-348-0115 • Amanda Berhaupt • Graduate Assistant • aberh001@fiu.edu • 305-348-2731 • RandhirKaur • Graduate Assistant • rkaur001@fiu.edu • 305-348-2731