Shane T. Mueller & Kejkaew Thanasuan Department of Cognitive and Learning Sciences

1. Shane T. Mueller & Kejkaew Thanasuan Department of Cognitive and Learning Sciences Michigan Technological University 1 Modeling expert memory search, knowledge access, and decision making: A model of crossword puzzle play

2. Why Study crossword? • Naturalistic approach to studying decision making and knowledge-based problem solving • Experts in this task have highly-developed memory encoding and retrieval skills • The knowledge space is well-characterized

3. Top players are 2-5x faster than good casual players, who are 10x faster than novices Crossword Expertise • 3

4. Crossword Expertise 4 Expert Novice

5. Goal • To understand crossword skill by developing a computational model of crossword solving. • Rely heavily on constraints from a natural corpus. • Understand how 'cues' provide both memory access and constraint

6. Schematic Model ofKnowledge access 6

7. Memory search and Access • Any feature in the clue (word stem or hint) will activate a set of answer words according to the relative probability. • Probabilities of multiple features can combine to form activation distribution. 7

8. Orthographic Representation • Each answer has associations from orthographic units in the clues that lead to that answer. • Currently, lexical units include: • letters • adjacent letter pairs • length units • Based on Mueller & Thanasuan (2014, JMP) model of word-stem completion. 8

9. Orthographic Corpus • Used (250K token/4M word) Ginsberg database (described later) • Each word coded for 26 letter-features + 27^2 letter-pair features • A bank of 10 (logarithmically-defined) length features. 26 729 10 250K 9

10. ---K-- SKATE KOREA OSAKA ANKLE KNEE KOALA KNEES SKEET ASKED AKRON Orthographic Search through activation space TU-K-- TUSK TURK TUSKS TUCK TUCKS TURKS TURKEY TIMBUKTU TUCKIN TUTU 0.0009366936 0.0009117806 0.0009024288 0.0008888983 0.0008640101 0.0008119471 0.0007999176 0.0007830753 0.0007498722 0.0007384729 0.05923629 0.04718652 0.03634697 0.02930575 0.025934 0.0205008 0.01378494 0.01108984 0.01044018 0.01013701 10

11. Semantic Representation • Each answer has associations from lexical units in the clues that lead to that answer. • Currently, lexical units include: • word • word pairs • Association strength increases with each experience. • No effort made to form semantic associations based on contextual semantics (e.g., LSA) or linguistic analysis 11

12. Semantic Knowledge Base 12 • Ginsberg's Crossword Clue database • 4,000,000+ clues from 50K puzzles • 250K unique answers (rows) • With stemming by Celex 2.5, 110K clue words (columns) • 550K clue word pair units (columns) • Very sparse matrix (.000024 used) 110K 550K word-pairs 250K 12

13. “Flop” EDSEL BOMB DUD EDSELS ISHTAR KER SANDAL THONG FIASCO ANEGG EARED UTURN HIT SMASH NERD ....TURKEY (25) Semantic Activations 0.00760 0.00622 0.00581 0.00551 0.00539 0.02554 0.02339 0.01639 0.00829 0.00822 0.00539 0.00539 0.00502 0.00470 0.00451 0.230 0.086 0.072 0.063 0.057 0.032 0.017 0.017 0.015 0.013 0.013 0.013 0.011 0.011 0.010 .....0.0057 • “Thanksgiving Bird” • CARVE • PIE • BASTE • EMU • TOM • TURKEY TURKEYTROT IBIS • EGRET • MEAL • PIES • WOODYWOODPECKER • ROC • EMUS • LOON 13

14. Memory Search • Any clue can provide both semantic and orthographic cues. • Top options get sampled and evaluated against orthographic constraints and (for orthographic route) semantic cues. • Semantic retrievals must get 'recovered' ala SAM, based on their strength. • Parameters control number of candidates that can be checked and recovery probability, • Hypothesis—search goes on in each domain separately and independently.

15. Single-route models

16. Crossword Experiment • One clue at a time. • Stimuli: 56 answers, clued with easy/difficult semantic and easy/difficult orthographic. • Difficult: 1 letter; Easy: 3 blanks • Clue difficulty selected subjectively • 4-11 letter words • Conducted in lab (novice) and ACPT, and on-line (experts) 16

17. Implemented in the Psychology Experiment Building Language (PEBL). See http://pebl.sourceforge.net 17

18. Semantic Model Evaluation • 43 clues • For 24, model and data agreed with pre-determined difficulty • For 7, model and data agreed, opposite of pre-specified difficulty. • For 3, model agreed with pre-specified difficulty but not data • For 9, humans followed prespecified difficulty, but model did not.

19. EASY CLUE "Good source of potassium" Swaggering show of courage" "Food storage places” "Precious stones" "Fir tree fruit" A child's sneaker usually lacks this" Some failures of Semantic Model ANSWER • BANANA • BRAVADO • FREEZERS • JEWELS • PINECONE • SHOELACE HARD CLUE • "split ingredient” • "Swagger" • "Spoilage slowers" • "Safe deposit" • “Wreath adornment" • “It goes over the tongue"

20. Experiment Results 20

21. Comparison to novice 21

22. Expert results 22

23. Single Route Models • Each uses a generate-and- check scheme • Single route models produce no difficulty effect on non-cued route 23

24. Qualitative Interpretation • For Experts, we see both semantic and orthographic difficulty. • Semantic difficulty diminished as more orthographic letters are given. • Consistent with cascading or parallel models search along each rout. • Additional RT data may be needed to understand parallel-serial architecture

25. Summary • Difficulty effects can be produced (usually) by relying SOLELY on statistics of the environment (no free parameters). • 'Cues' provide both memory access and constraint. • Experts have better semantic retrieval which gets them started in the puzzle. • Also, improved orthographic completion that eliminates semantic difficulty effects. • In expert memory, Cues provide both activation and constraint. 25