660 likes | 1.08k Views
Think-Aloud Protocols. February 5, 2010. Today’s Class. Probing Question Think-Aloud Protocols Assignments. Probing Question for Friday, February 5. Let’s say that the NSF has just given you a $100,000 grant to study student note-taking during classroom lecture
E N D
Think-Aloud Protocols February 5, 2010
Today’s Class • Probing Question • Think-Aloud Protocols • Assignments
Probing Question for Friday, February 5 • Let’s say that the NSF has just given you a $100,000 grant to study student note-taking during classroom lecture • i.e. you can assign one minion to the task full-time for a year • Which qualitative method would you prefer to use, and why? • What factors would you be especially attentive to, to ensure validity of your results?
Today’s Class • Probing Question • Think-Aloud Protocols • Assignments
I need a volunteer • A fast typist • To try to record things people say while thinking aloud • Only record the person thinking aloud
I need two more volunteers • Volunteer #1: Will think-aloud • Volunteer #2: Will run the think-aloud
What should Volunteer #2 rememberwhile running the think-aloud?
What should Volunteer #2 rememberwhile running the think-aloud? • Use appropriate instructions • Prompt the subject • Write down interesting events • Focus on what the subject does and says, not “why” • Don’t help the subject
Long division • Volunteer #1 will think-aloud • Volunteer #2 will run the think-aloud • So, Volunteer #2, give Volunteer #1 some instructions…
Long Division 7140 16, 429, 782, 600
How did it go? • Volunteer #1: • How hard was the task? • Did thinking aloud make it harder?
How did it go? • Volunteer #2: • How hard was it to get him/her to think-aloud? • Did you worry about disrupting him or her? • Audience: • Any (friendly and constructive!) criticism for Volunteer #2?
Thoughts/comments • From anyone • On why this task might have been difficult/easy for the person thinking aloud, or the experimenter • (or anything else)
Drawing a scatterplot • Volunteer #1 will think-aloud • Volunteer #2 will run the think-aloud
How did it go? • Volunteer #1: • How hard was the task? • Did thinking aloud make it harder?
How did it go? • Volunteer #2: • How hard was it to get him/her to think-aloud? • Did you worry about disrupting him or her? • Audience: • Any (friendly and constructive!) criticism for Volunteer #2?
Thoughts/comments • From anyone • On why this task might have been difficult/easy for the person thinking aloud, or the experimenter • (or anything else)
The 9-Dot Problem(Kershaw & Ohlsson, 2004) Volunteer #2 will connect all 9 points with just 4 straight lines(you don’t already knowthe answer, right?)
How did it go? • Volunteer #2: • How hard was the task? • Did thinking aloud make it harder?
How did it go? • Volunteer #1: • How hard was it to get him/her to think-aloud? • Did you worry about disrupting him or her? • Audience: • Any (friendly and constructive!) criticism for Volunteer #1?
Thoughts/comments • From anyone • On why this task might have been difficult/easy for the person thinking aloud, or the experimenter • (or anything else)
Insight Problems • The biggest problem with thinking-aloud as a technique. • When people are trying to figure something out, and it requires “insight”, thinking-aloud can be very disruptive. • Therefore, if your participant is really baffled, • Now might not be exactly the time to push them to “KEEP TALKING” • In general, it’s best to avoid using think-aloud in these situations • What else might you use to study process in these situations?
Two more volunteers! • One more time
Trogdor! • http://www.homestarrunner.com/trogdor.html
How did it go? • Volunteer #1: • How hard was the task? • Did thinking aloud make it harder?
How did it go? • Volunteer #2: • How hard was it to get him/her to think-aloud? • Did you worry about disrupting him or her? • Audience: • Any (friendly and constructive!) criticism for Volunteer #2?
Thoughts/comments • From anyone • On why this task might have been difficult/easy for the person thinking aloud, or the experimenter • (or anything else)
OK enough for now • What have we learned about think-alouds from all of this? • Bonus points for expressing your thought as a Beyer & Holtzblatt-style aphorism
What could you do? • If you want to do a think-aloud on some uninterruptable process?
Example • Take, for example, a helicopter pilot who needs to drop a ladder to someone while still flying
Task-splitting • You might want to do a think-aloud of the pilot’s processes as (s)he does each of these tasks separately: ie, a think-aloud as (s)he lowers a ladder, and a think-aloud as (s)he flies the plane • This would show you the pilot’s major sub-goals, decision-points, and evaluation criteria for each task
Retrospective Think-Aloud • Videotape the participant as they are doing the task • Take a 10 minute break • Then show the video to the participant and have them do colour-commentary. • Take notes on their colour-commentary • (Or even tape-record it)
Retrospective Think-Aloud is Also Useful… • When you are studying someone who is so expert at an action, that their actions are completely automatized. • In this case, thinking-aloud can disrupt or change someone’s behavior.
Volunteer #1 • Volunteer #1: Please tell me the cash value of the change you have in your pocket. Don’t think aloud.
Volunteer #2 • Volunteer #2: Please THINK ALOUD while you determine the cash value of the change you have in your pocket.
Volunteer #2 • Volunteer #2: Please THINK ALOUD while you determine the cash value of the change you have in your pocket. • Volunteer #1, is this the procedure you used?
Analysis • Let’s take the longest of the protocols that our noble live transcriber has produced • Can you email it to me? • I’ll put it up on the screen
Segmentation • It’s better to do segmentation from an audio recording than a transcript in general, but for this time…
Segmentation • Your thoughts on where we should segment?