Creating A Multiple Measures Placement System

1. Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber

2. The Concept • Although test scores may predict failure, they do not necessarily predict success. • Research shows that students’ backgrounds, environments, and personal habits may have more influence on their potential academic success than their residual academic skills. • Find a way to factor that information into the placement decision at testing time.

3. Decision Process • Involve Faculty • Select Questions • How many • What variety • Select Weight Values • Use negative weights? • How much possible total weight

4. Select Questions • Use enough questions that any one does not have undue influence • Must be manageable • Each weighted answer choice requires an additional line in the placement rule • Weights must be repeated in each rule segment • Questions must, in some way, relate to student success.

5. Limits • Limit total weight so that background information does not allow students to skip a course level • Limit number of questions to a manageable number • More questions adds to testing time • Placement rules can become unmanageable • Answer choices must be mutually exclusive and all inclusive

6. Example • (If Arithmetic, plus all weighted choices is >= 75 OR Algebra, plus all weighted choices is >= 48)AND (Algebra, plus all weighted choices is < 65 OR Algebra Not Taken) AND (CLM, plus all weighted choices is <62 OR CLM Not Taken) Then Placement is Elementary Algebra

7. Example, Continued • If this rule had 5 questions with 4 weighted answer choices each, there would be 80 lines just for weights. • With too many questions, or too many choices per question, rules can become unmanageable

8. Conditional Weights • High school accomplishments have limited shelf life • How much does it matter that a 25-year-old student had 2 years of high school algebra? • Does it matter that the same 25-year-old student works for a surveyor and uses algebra daily?

9. Conditional Weights • Young students have not had time to build skill usage experience, but their high school accomplishments are relevant • What older students accomplished in high school is less relevant than what they do now.

10. Example • How long has it been since you were enrolled in high school or other formal educational process? • Less than 2 years or still enrolled • 2 to 5 years • More than 5 but less than 7 years • 7 years or more • Use high school data for up to 5 years, experience for more than 5 years.

11. Assigning Weights • Total possible weight should not move student more than one level in either direction • Set maximum possible weight so a student who scores near or above the midpoint of a placement range could move up, but one who scores below the midpoint could not. • Use faculty to select BGQ and assign weight • Guide them

12. Multiple Measures Movement Model

13. Sample Question With Weights • Which choice below best describes you when you read textbooks or other complex information? • I usually need to read material several times before I understand it well -.01 • Sometimes I can understand what I read the first time, but often I must reread it .00 • I usually understand what I read if I take notes or highlight passages. +.01 • I always understand what I read the first time through +.02

14. Preparing to Build the System • Assign numeric codes to course names • Determine which tests will be used for each course in each discipline • Create cut score Table • Create a BGQ weight Matrix

15. Building the System • Create Background questions • Assign BGQ to groups • Create branching profiles • Create course groups • Create courses and assign to groups List • Create majors if used • Create placement rules Edit1Edit2

16. Verify • Write most complex rule first • Run verify function in branching profile • Use several BGQ and score combinations to test the placement rule • Compute weighted score for each run • Try to hit cut scores to test for bad weight or answer choice selections

17. Common Errors • Unequal weights between rules • E.G. A response has .01 weight in one rule and -.01 weight in the next rule • Misplaced Parentheses • The multiple measures weights make the rule larger and more difficult to visualize • Misuse of AND/OR • Misuse of arithmetic operators • Wrong answer choice in rule line

18. Troubleshooting • From the score report, determine what the student’s weight should be from the BGQ responses • Using the weight, compute the weighted score • Determine what the placement should be • Examine the appropriate rule for errors

19. Computing the Weight 1

20. Computing the Weight 2

21. Computation • Score is multiplied by 1 plus the accumulated weight. • 85 * (1+.04) = 88.4 • Placement will be based on a score of 88. • Example 2 • 85 * (1+ [-.03]) = 82.45 • Placement will be based on a score of 82

22. Creating a Multiple Measures Placement System An exercise with Ron Gordon & Armand Brunhoeber Thank you for not throwing things at the presenters