Direct Measures of Assessment

1. Direct Measures of Assessment Office of Institutional Effectiveness University of Houston-Victoria

2. What is a Direct Measure of Assessment? A direct measure of assessment directly evaluates the work being done. In academic units, this work is student learning.

3. Is an assignment or course grade a direct measure? A grade may tell us if a student earned a B or a C, but it doesn’t tell us why. For program assessment, we need more specific information: students were strong in A and B, but struggled with C and D. Additionally, grades often reflect things not related to the student learning outcome being measured: Attendance, class participation, grammar and punctuation, etc.

4. Two common types of Direct Measures • Standardized Tests: tests that administered to all test takers under identical conditions. • Classroom Work: papers, projects, computer programs, case analyses, field work completed as a requirement of a specific course or program of study.

5. Advantages of Standardized Tests as Direct Measures • they offer scores that can be compared to national samples; • they test students against national standards of performance.

6. Disadvantages of Standardized Tests as Direct Measures • If the test does not count toward regular coursework, students may not do their best; • Finding a meaningful sample of students to take the test can be difficult; • There may not be a direct relationship between what is taught in the program and the content of the test; • Tests are expensive.

7. Is an instructor-made or textbook-provided test a Direct Measure:? • It is not often that instructors give an entire exam over one Student Learning Outcome. It is more appropriate to identify specific questions related to the Student Learning Outcome and use them as a Direct Measure.

8. Advantages of Classroom Work as a Direct Measure • It is already being examined by faculty as part of the grading process, so using it in program assessment is time-efficient and inexpensive; • A grade is attached, so students are more motivated than is typical with standardized tests; • Classroom work reflects what is actually taught at the institution; • Faculty may be more invested in their own analysis of students’ work.

9. Examples of Classroom Work Commonly Used as Direct Measures • Penultimate work from a course toward the end of the course of study: a research paper or an internship. Used to answer questions like, “What are students’ strengths and weaknesses at the end of our program?” • Pre-Post: Sample of student work at the beginning of their course of study and at the end. Used to answer questions like, “What is the value added for students in our program?” • Portfolios of student work. Used to answer questions like, “How do our students develop?”

10. Specific Examples of Direct Measures • Ratings of student skills by their field experience supervisors • Scores and passrates on appropriate licensure or certification exams • Capstone experiences such as research projects, presentations, theses, dissertations, oral defenses, exhibitions, and performances, each scored with a rubric • Other written work, performances, and presentations, scored with a rubric

11. More Examples • Portfolios of student work • Score gains between entry and exit on writing samples • Observations of student behavior, undertaken systematically and with notes recorded systematically • Summaries and assessments of electronic clas discussion threads • Think-alouds • Exercises using classroom response systems • Feedback from computer-simulated tasks

12. Gathering Student Work In very large programs, samples of student work may be used for program assessment. Recommended sample sizes are provided from Suskie(2009):

13. Choose a Direct Measure for the following student learning outcomes… • Describe and analyze math concepts graphically and numerically • Identify the author's thesis and the main evidence supporting it • Exemplify ethical, legal, and professional standards for advanced nursing practice • Demonstrate necessary pedagogical knowledge for teaching in the classrooms • Analyze the economic impact of a development project

14. Using a Rubric to Assess Student Work When you assess student work, you need a set of criteria. A rubric provides a format for expressing criteria and standards. Rubrics disaggregate various aspects of student work so that the instructor can see what is most and least challenging for students in the assignment.

15. Conclusion Direct measures can be as simple as including a question on an exam or as complicated as a program portfolio. Faculty should consider practical constraints and select strategies that are realistic, focused, and manageable.

16. Bibliography • Allen, M. J. (2004). Assessing academic programs in higher education. Bolton, MA: Anker. • Suskie, L. (2009). Assessing student learning: A common sense guide (2nd ed.). San Francisco: Jossey-Bass. • Walvoord, B. E. ( 2010). Assessment clear and simple. San Francisco: Jossey-Bass.