What I Learned About Assessment From the AP Program Dan Kennedy Baylor School Houston AP Teachers Meeting November 11,

1. What I Learned About Assessment From the AP Program Dan Kennedy Baylor School Houston AP Teachers Meeting November 11, 2006

2. True Confession of a Veteran Mathematics Teacher: For many years I never thought much about assessment. I graded my students in what I thought was an appropriate variety of ways: Tests … Quizzes … and Homework. This model had stood the test of time.

3. In 1986 I was invited to become a member of the AP Calculus Test Development Committee. From 1990 to 1994 I would serve as chair. My experience with this group changed my views of assessment forever.

4. I had already learned one important fact about classroom assessment merely by teaching an AP course: It changes the entire classroom dynamic when the teacher honestly does not know what will be on the test. The teacher has no other option but to teach the students how to think for themselves!

5. Why students don’t think on tests: • Thinking takes time. • Thinking is only necessary when you cannot do something “without thinking.” • If you can do something without thinking, you can do it very well. • Students who can do something very well have been well-prepared. • Therefore, if you prepare them well, your students will proceed through your tests without thinking!

6. Were AP Calculus exams predictable? 1987 BC Exam: 1. Differential equation2. Implicit Differentiation3. Area/volume4. Series5. Particle problem6. Theory problem (stretch) Of course, this was just one exam. But there were others like it.

7. Will this be on the test? But if we tried to change anything, teachers would notice. Then, in AP workshops all over the country, teachers would find themselves uttering to AP consultants the words they dreaded most when spoken by their students:

8. And why should teachers NOT ask that question? • It is how the game is played. • We show the students how to do math. • We let them practice at it for a while. • Then we give them a test to see how well they can mimic what we did. • The game is won and lost for BOTH of us on test day.

9. This was just another example of the educational paradigm that was leading my student not to think on tests! But how can teachers change the game if we want our students to succeed?

10. Teachers have one secret weapon: We define what it means to succeed. We control the grade!

11. Something I learned about assessment from the AP program: It is perfectly OK to scale grades! 75% = 5

12. At our school, 75% is not a good grade. In fact, 65% is a minimal pass. • Is this reasonable? Think about it. • The all-time NBA record for field goal percentage in a season is 72.7%. • The all-time record batting average for major league baseball is .440 (44%). • A salesperson who makes a sale on 75% of first contacts is a genius. • So how can we expect 75% success from someone who is just learning?

13. If the AP exam were constructed so that the low-to-average student could get 75% of the maximum points, • it wouldn’t be much of a test, and • the distribution would be skewed rather than normal.

14. 99 • • 92 • 82 71 • 30 • 20 75 93

15. An Important Disclaimer: Scaling grades is not about building self-esteem. Scaling grades is about teaching mathematics. Assessment should support your efforts to teach your students mathematics. It should not get in the way.

16. Scaling grades on the TI-84 Plus

20. Some things that ETS worried about that I didn’t: • r-biserial • Content validity • Speededness • True score • Grading rubrics

21. r-biserial (r-bis) “A correlation coefficient relating performance on a test question and performance on the measure used as a criterion. It is an index of discrimination measuring the extent to which examinees who score high on the measure used as the criterion tend to get the question right and those who score low tend to get it wrong.”

22. 1969 Multiple-choice question #26: The answer is (C).

23. AB Stats: A 3% B 57% C 7% D 3% E 20% BC Stats: A 1% B 70% C 11% D 2% E 9% Projected Chimpanzee Stats: A 20% B 20%C 20% D 20% E 20%

24. Correct responses to problem #26: AB 7% BC 11% Chimps 20%

25. Content Validity “Validity is the extent to which a test measures what it is supposed to measure. The content validity of an (AP) test is the extent to which the content of the test represents a balanced and adequate sampling of the universe of content in which the test is intended to measure achievement.”

26. The AP Calculator Experiment (1983-84) In 1983 the AP Calculus Committee decided to allow (but not require) the use of scientific calculators on the AP Calculus examinations. This was not to be a very happy debut for technology on the AP stage.

27. AP readers found that students were losing points on the free-response section because of calculator misuse. The calculators affected the scores. But calculators were not being tested! This compromised the content validity. The committee had two choices: 1. Forbid calculators and test as usual; 2. Require calculators and alter the test. They chose to forbid the calculators.

28. One of my Precalculus tests from 1990: Note the emphasis on computation. Note that there is nothing here to suggest that any of this stuff is worth knowing!

29. A recent test on the same functions: Still not perfect, but a better test.

30. Speededness “The appropriateness of a test in terms of the length of time allotted. For most purposes, a good test will make full use of the examination period but not be so speeded that an examinee’s rate of work will have an undue influence on the score he receives.”

31. Allowing for speededness

32. True Score “A score entirely free of errors of measurement. True scores are hypothetical values never obtained in actual testing. A true score is sometimes defined as the average score that would result from an infinite series of measurements with the same or exactly equivalent tests, assuming no practice effect or change in the examinee during the testings.”

33. Why teachers don’t need to worry about true score: We can assess our students all year long! The more often the better. Sorry, kids. Yessss!

34. AP Calculus Grading Rubrics If the AP readers can give partial credit fairly to 250,000 students, I ought to be able to do it for my own students. In AP Calculus, I can even use the AP rubrics to do it.

35. AP Calculus Exams are: • Designed to test knowledge • Designed to test cleverness • Scaled reasonably • Not made up by the teacher • Open assessments • Comprehensive assessments (valid) • Honest about technology

36. Two Fundamental Principles: • Assess what you value. • Value what you assess.

37. Some problems with traditional tests: • They assess only a fraction of what we value. • They depend too much on luck. • There is often no feedback (as with final exams). • They are usually taken alone. (Is this what we value?) • They are usually timed. (Is this a good model for quality work?) • They are frequently taken under artificial, stressful conditions. • They are dependent on teacher stimulus. • They are often devoid of creativity (if students are “prepared”). • They favor one narrow kind of student performance. • Success is usually short-term and non-transferable. • The emphasis in the end is what the student can NOT do. • They can inhibit further learning.

38. Some assessment strategies I like: • Assess what you value and value what you assess! • Assess often, with different kinds of assessments. • Give meaningful and prompt feedback. • Give partial credit for partially correct work. • Explain all your expectations to your students from the start. • Test diligence, knowledge, and cleverness in focused ways. • Encourage creativity through your assessments. • Scale grades to control the standard deviation. • Only fail students who are failures. Keep everyone in the game. • Encourage collaboration in class and on homework. • Assess diligence. Find a way to grade homework frequently. • Try portfolios. • Remember: This is not about self-esteem. It’s about teaching mathematics to all your students!

39. Rebecca Flake’s Portfolio Entry Rebecca Flake

40. Students need to hand in a portfolio of items of their own choosing. The main point of this assessment is that they are not responding to a stimulus from me (as in a test or a quiz). My primary directive for student portfolio entries is this: Give me evidence of your learning that I otherwise would not have!

41. This was my first year to be a peer tutor, and I enjoyed helping the girls in the dorm a lot. Last night, though, I finally saw the importance of my peer tutoring. My roommate came in at 10:00 extremely upset over her Precalculus test that was the next day. I calmed her down and told her that I would help her if I could. Carrie, who had been in the play, had gotten behind in her work, so she didn’t understand what they were doing. She showed me the problem. I knew the answer, but I wasn’t sure how to explain it to her in a way that was not confusing. I thought about it for a while, and I ended up trying several approaches (with Clara’s help) that I had learned in Calculus, until I finally got through to her. Then I made her work a few problems for me, and she did them perfectly. She understood! I was so happy to be able to help her that I had forgotten I was supposed to be studying for my own Calculus test. She was so happy she understood that she began to cry. She really began to cry. It’s great to be able to use the things you have learned to help other people learn too.

42. A happy footnote: Carrie really did understand. She scored 93 on the Precalculus test the following day – a personal best for her, and a full 9 points above the class average. Actress Carrie

43. E-mail me at: dkennedy@baylor.chattanooga.net

44. Or visit the Baylor School web site at www.baylorschool.org. Click on me under Faculty and link to my home page.