Download
1 / 44
440 likes | 601 Views
ASSESSING STUDENT LEARNING AT THE COURSE LEVEL IN SCIENCE COURSES CAN WE RECONCILE LARGE CLASSES, MULTIPLE CHOICE TESTS AND TRUE LEARNING ASSESSMENT?. A. Anil Kumar, Ph.D. Head, Department of Physics & Professor, Electrical and Computer Engineering & Fa-Chung (Fred) Wang, Ph.D.
E N D
ASSESSING STUDENT LEARNING AT THE COURSE LEVEL IN SCIENCE COURSES CAN WE RECONCILE LARGE CLASSES, MULTIPLE CHOICE TESTS AND TRUE LEARNING ASSESSMENT? A. Anil Kumar, Ph.D. Head, Department of Physics & Professor, Electrical and Computer Engineering & Fa-Chung (Fred) Wang, Ph.D. Professor, Department of Physics PRAIRIE VIEW A&M UNIVERSITY 9th Annual Texas A&M Assessment Conference The Hilton, College Station, TX February 22-24, 2009
PHYSICS (PHYSICISTS) & ASSESSMENT • Assessment is new to physicists. • No national or regional accreditations. • For physicists assessment is based almost entirely on “peer endorsement.” • In a standards based world, that does not work.
WHAT WE ARE UP TO TODAY • Our emphasis is on how to define and implement assessment in physics courses. • More importantly, how to utilize these results in improving our instruction so that the student is better positioned for learning. • In particular, we want to focus on freshman and sophomore courses, where multiple-choice tests (MCTs) are almost the norm. • So, the question is: How does one reconcile MCTs, assessment and true student learning? • In other words, how do we know that they know? • Unfortunately, assessment seems to have diverse definitions.
DEFINITION OF ASSESSMENT "Assessment is the process of gathering and discussing information from multiple and diverse sources in order to develop a deep understanding of what students know, understand, and can do with their knowledge as a result of their educational experiences; the process culminates when assessment results are used to improve subsequent learning.” Huba, M. E. and Freed, J. E. (2000). Learner-Centered Assessment on College Campuses -- Shifting the Focus from Teaching to Learning. Boston, Allyn and Bacon.
OUR PRESENTATION IS IN THREE STAGES • Discuss the environment regarding MCTs – typical posing of the questions, accepted approaches to answers, myths and practices • Preliminary results from our own assessment efforts – pre- and post tests, combining MCTs with Fill-in-the-Blanks, and other techniques • Recommendations for future MCTs based on studies of cognitive learning theories
MULTIPLE CHOICE TESTING – PROS & CONS • Pros/Advantages • Less time to administer • More comprehensive evaluation of extent of knowledge. • Increase in efficiency • Graded purely on their selections • Lower likelihood of teacher bias in the results. • Graded purely on their knowledge of the topic • Cons/Disadvantages • Ambiguity • Limited feedback, if ever • Learning reduced to “correct” answers • Process of analysis difficult to ascertain, at best • No training for faculty or students in MCTs • Acquired “tricks” from schools to answer MCTs
MCTs - IF, IF, IF, … If item writers are well trained and if items are quality assured, it can be a very effective assessment technique. If students are instructed on the way in which the item format works and myths surrounding the tests are corrected, they will perform better on the test. If there are large numbers of items on a test, and if good sampling and care over case specificity are employed, overall test reliability can be further increased. …
EXAMPLES OF “CORRECT” ANSWERS BUT WRONG PROCESS • The fraction 64 divided by 16 yields: • 8 • 2 • 4 • Other ______ • Correct Answer: 4 • Process: • The fraction 95 divided by 19 yields: • 7 • 5 • 9 • Other ______ • Correct Answer: 5 • Process:
TYPICAL INSTRUCTIONS FOR ANSWERING MCTS • Look to see if any of the answers is much longer than the others. • The person making the test wants the right answer to be completely true. • To make it completely true, they sometimes add more details to the correct answer, making it much longer than the others. • The freezing point of water is • 32 degrees Kelvin • 32 degrees centigrade • 0 degrees Fahrenheit • 0 degrees centigrade for pure water at sea level • (D) is probably the correct answer.
TYPICAL INSTRUCTIONS FOR ANSWERING MCTS • See if one of the answers sticks out as the "odd one out". • Since there is only one correct answer to a question, that answer has to be different from the other answers. • So if you see two or three answers that all mean the same thing, they must be the wrong answers. • Boy • Girl • Son • Lad • Since "Boy", "Son", and "Lad" all mean roughly the same thing, chances are the answer is "Girl", even if we do not know what the question was. This trick is dangerous, however. Sometimes the "odd one out" can fool you. Maybe the answer was "Son" because it is the only answer about family relationships. But this trick might help if you are guessing blind anyway.
THIS EXAMPLE IS FROM AN ACTUAL BOOK ON HOW TO PREPARE FOR TAKS EXAM • Which of the following is a mathematician: • (A) Ann Richards • (B) Susan B Anthony • (C) Karen Uhlenbeck • (D) Hillary Rodham Clinton
SOMETIMES, A STUDENT WHO THINKS DEEPER MAY NOT GET THE “CORRECT” ANSWER! • A raindrop on a drizzly day takes one minute from the time it leaves the cloud until it hits your umbrella, 3 miles below. The average velocity of the raindrop on its journey is • 3 miles/hr • 60 miles/hr • 180 miles/hr • Impossible to determine because the raindrop accelerates due to gravity.
EDUCATION QUANDARY: 'MULTIPLE-CHOICE QUESTIONS CAN BE HARDER IF A PUPIL READS THE QUESTION CAREFULLY OR HAS A DEEPER UNDERSTANDING. SURELY THIS IS WRONG?’ “Like so much of our exams and testing industry, the questions have far more to do with helping adults rank and sort children for their own ends than with deep learning.” http://www.independent.co.uk/news/education/schools/
Outcomes Assessment for Calculus-Based Physics I (Mechanics)
To find another form of test-format, other than Multiple-Choice Type. This other format should have the same advantages of Multiple-Choice: Easy to grade, and little ambiguity in assigning grades. This other format will offer a better measure of Students’ Quantitative and Transferrable Skills. Goal
Giving Test in two versions, one in Multiple-Choice & another in Fill-in-the-Blanks Format. Compare Students’ performance in these two different formats. Using the results and analysis to modify or improve Teaching-Methods and format for later tests. Approaches
Started in the fall of 2007, pre- and post-test were given in two types of question: Multiple-Choice type to test simple concept, Fill-in-the-Blanks type to test quantitative skills. the wording and style were similar to questions & problems in text-books. (Giancoli, Serway, etc.) Feedbacks: We were not happy. (a) MC type tends to be on the easy side & FB type tends to be harder. Can we have something in the middle? (b) Students spent a large portion of the test-time in reading 9 problems, some have two or three parts. (c) The style of Pre- and Post-tests were not the same. True Measure Of Outcome? Test Design
We started to try a new format in a Post-Test, Dec. 2008. Old: 10 to12 problems, each relates to materials covered in one chapter, or even in one section; New (Goal): 3 to 5 problems, each problem has several sub-problems. Each sub-problem tests a concept, when combined it relates to materials covered in several chapters; Two versions of test. Part (a) of problem 1 is given in MC format in version Gn, and in FB format in version Pk. Part (b) of problem 1 is in FB format in version Gn and in MC format in version Pk. The two versions are the same problem with different numbers.
PROBLEM 1A-G - Pk A 1,000-kg car accelerates uniformly from 4.00-m/s to 16.00-m/s in 4.00-s. The car’s acceleration, in m/s2 , is (a) 1.0; (b) 2.0; (c) 3.0; (d) 4.0; (e) 8.0. The car’s displacement is _________________m. The net force on the car is _________________N The car’s final momentum, in N-s, is (a) 4,000; (b) 8,000; (c) 12,000; (d) 16,000; (e) 20,000.. The car’s final kinetic energy is _____________J. The net work done on the car is _____________J. The net average power to the car is __________W.
PROBLEM 1A-G - Gn A 1,000-kg car accelerates uniformly from 4.00-m/s to 12.00-m/s in 4.00-s. The car’s acceleration is __________________m/s2 . The car’s displacement, in m, is (a) 12.0; (b) 16.0; (c) 24.0; (d) 32.0; (e) 40.0. The net force on the car, in N, is (a) 1,000; (b) 2,000; (c) 3,000; (d) 4,000; (e) 5,000. The car’s final momentum is ________________N-s. The car’s final kinetic energy is _____________J. The net work done on the car is _____________J. The net average power to the car is __________W.
There were 91 students, 43 took version Gn & 48 took version Pk. Each student worked 4 sub-problems in MC format & 4 in FB format. The 8 sub-problems are grouped into three levels: Level 1, easiest, such as “Find the net force on an object when the mass & the acceleration of the object are given”. Level 2, a little more involved, such as “ Find the acceleration or displacement when the initial velocity, final velocity & time are given”. Level 3, more involved, such as exchange of elastic potential energy & kinetic energy, or exchange of kinetic & gravitational potential energy.
Much less time will be spent by students reading into the physical situations. Physical quantities covered in different chapters are shown to be inter-connected. Advantages of the New Format
NEXT STEPS Repeat the tests and correlate student’s prior preparation in mathematics and physics with test performance. Conduct interviews with students in groups with similar performance levels. Assess the level of remediation a student needs to improve his/her learning during the semester.
EXPERT-NOVICE DIFFERENCES IN MEMORY http://research.physics.illinois.edu/PER/ In each pair, the second picture was altered from the original either in: (i) changes that did not alter the physics depicted, or (ii) changes that made a meaningful change to the physics depicted. Novices perform equally well on both types of image pairs, but experts perform significantly better on image pairs that contain a meaningful change in the physics depicted than they do on image pairs that do not contain a meaningful change to the physics depicted. Implication: Conceptual physics knowledge plays a role in the way physics experts form representations of physics pictures. Moreover, for some image pairs, novices were much more able to detect the change in the picture than were experts. This suggests that the memory representations that experts create may lack the surface-feature details that novices’ memory representations contain.
Our recommendation is that tests be prepared such that: • They offer an environment in which the student can receive partial credit; • The student be given an opportunity to explain the rationale for his/her choice; • The student’s choice must provide an insight into the depth of his/her understanding. • Knowing something about the way appropriate process was used to obtain wrong answers and inappropriate strategy was used to obtain correct responses helps to provide a more valid interpretation of test results. • Such knowledge is also essential to distinguish adequate from inadequate procedures so that suitable remedial work can be assigned. Effective remediation usually depends on having detailed knowledge of the types of errors that are being made and these have to be inferred from the alternatives selected by the student.
SPECIFICALLY • Build a degree of ambiguity into the question. • Design the distractors to reflect the dominant types of misconceptions. • Design a “nested” series of problems that allows us to assess true student learning: • Ability to model different parts of the problem and integrate into a “whole.” • Ability to categorize problems based on similarities - surface and deep. • Ability to transfer skills from one type of problem to another. • Ability to transfer skills from one subject to another.
EXAMPLE • A raindrop on a drizzly day takes one minute from the time it leaves the cloud until it hits your umbrella, 3 miles below. The average velocity of the raindrop on its journey is • 3 miles/hr • 60 miles/hr • 180 miles/hr • Impossible to determine because the raindrop accelerates due to gravity. • Provide the rationale for your choice: ____________________________________________________________ • ____________________________________________________________
EXAMPLE (continued) • A raindrop on a drizzly day takes one minute from the time it leaves the cloud until it hits your umbrella, 3 miles below. The average velocity of the raindrop on its journey is • 3 miles/hr – CLEARLY WRONG • 60 miles/hr – CLEARLY WRONG • 180 miles/hr – MOST LIKELY CORRECT • Impossible to determine because the raindrop accelerates due to gravity. – POSSIBLY CORRECT
IF CHOICE (C) Rationale: The raindrop would have acquired the terminal velocity by the time it reaches the umbrella.
IF CHOICE D Rationale: There are a few assumptions implied in this problem. The raindrop accelerates due to gravity and may experience other forces as well due to – friction upward, wind sideways, … This may cause the raindrop to traverse a longer path on its way to the umbrella. While the “vertical” displacement may be 3 miles, the actual path length may be longer. Too little information provided to make an accurate determination.
EXAMPLE • Examine the figure below. Assume a frictionless surface. Answer the following questions 1-5. • The acceleration of the two blocks is observed to be 1.8 m/s2.What is the magnitude of the tension T in the string connecting the two blocks? Assume a frictionless surface. • (a) 5.4 N • (b) 9.0 N • (c) 14.4 N • (d) Other ________ N
EXAMPLE (continued) 2. What is the magnitude of the force F pulling the 5 kg block to the right? Assume a frictionless surface. (a) 5.4 N (b) 9.0 N (c) 14.4 N (d) Other ___________ N
EXAMPLE (continued) 3. What is the magnitude of the net force on the 5 kg block? Assume a frictionless surface. (a) 9.0 N (b) 14.4 N (c) 18.8 N (d) Other ________
EXAMPLE (continued) • 4. Assume now that the friction of the surface cannot be ignored. What is the magnitude of the net force on the 5 kg block? • (a) 9.0 N • (b) 14.4 N • (c) 18.8 N • (d) Other ________ • What extra information do you need for this problem? Friction
EXAMPLE (continued) Follow-Up Questions • 5. How would your answer to each of the previous questions change if: • The masses are doubled? • The two blocks are reversed? (see figure below) • The surface is inclined to the right? • The surface is inclined to the left? • In (c) and (d) above, what more information would you need? • Can you identify a real-life situation for which this problem might be a good model? 5 kg T=? a=? 3 kg F
CONCLUSIONS • Revisit our entire approach to testing, evaluation and assessment. • Incorporate modern thinking about learning. • Employ problem-based learning approach.
THANK YOU FOR YOUR ATTENTION AND PATIENCE. ANY QUESTIONS?
