Representative design and the problem of generalization

1. Representative design and the problem of generalization Daniel Read PRELIMINARY draft: 17/06/2008

2. Essence of representative design • A study is a sample of behavior • A representative study is one that contains a representative sample of subjects, of situations, and of tasks. • The concept of representative design comes from Egon Brunswik.

3. Brunswik’s ideas were based on his theory • Humans as “intuitive statisticians” • Environment conveys unreliable and redundant cues to underlying reality • Humans learn to interpret these cues to infer that reality. • Led to lens model. *** Picture here ***

5. Brunswick design Measured “photographic” size of images correlates with “estimated bodily size” much less than with “measured bodily size”. The agent is using more information than photographic size (and maybe not even using it at all). Representative design can help us assess precisely what is going on.

6. But representative design does not need the theory • Theories/claims about human behavior are claims about people/behaviors/tasks (and how they will interact with underlying constructs). • Underlying the concept of representative design are two critiques concerning when non-representative outcomes can occur: • unfamiliar situations or combinations of situations produce atypical responses. • “selecting” situations increases likelihood of extreme or unlikely observations

7. Some “misleading” questions Questions from Piatelli-Palmarini Inevitable Illusions Tests for confidence in textbooks inevitably contain many questions like these. Correct answers are “Vegetation” and “Peru.” Adonis was the god of • (a) Love • (b) Vegetation The potato originated in • (a) Ireland • (b) Peru

8. Comparing Representative with Non-R choice of stimuli Almanac questions e.g., “Which city is farther north?” Non-representative: People asked to choose “good general knowledge items” Representative: Random items chosen from same source. Source: Juslin, 1994 Representative Non-representative

9. Why stimuli only? • Is random sampling of methods possible? • Not meaningful in many cases, but we can ask: • Does changing the method change the result? • If so, in what way? • Parameter change? Qualitative shift? • What method(s) give us the most information about the domain we care about?

10. Effect of varying method Claim: People are “hyperbolic discounters” The rate at which we devalue Method: Choose between £100 today, or £120 in one year. Return given in nominal amounts. Alternative: Choose between £100, or save the £100 for one year at 20% interest rate. 10

11. 17 713 17 713 1319 119 17 713 1319 119 1319 119 0.5 0.4 Discount rate 0.3 0.2 0.1 0.0 Interest - Interest+ Nominal - rate Amount amount Results Inferred “discount function” can take any form, and a wide range of values, depending on how you ask. Conventional theoretical interpretation arises from use of “Nominal amount” condition. Maximum borrowing rate Realistic lending rate Hypobolic Constant Hyperbolic 11

12. Lesson • We cannot generalise from one method designed to measure one construct, to other methods designed to measure the same construct. • Essence of multi-trait, multi-method matrix.

13. The key questions (a) How general are the claims do you wish to make? • How general are the results you have (or plan to have) obtained. Set the goal of making the two levels of generality equivalent.

14. What does “people are overconfident mean?” • For these people, at all times, for all questions, and for all tasks, (inferred) subjective probability is greater than objective probability? • Something in between? • For these people, at this time, for these questions, and for this task subjective probability is greater than objective probability?

15. Minimal generality • If theory T states that in situation X a person (agent) will do A, and the person in fact does B (and does so regularly enough that chance can be ruled out) then theory T is ruled out. • No need to show that the agent will do B in the “real world” • No need to show that the agent will do B in other situations, even other experimental arrangements. c.f. Mook, 1983

16. Harlow’s monkeys Harlow found that baby monkeys hung out with a cloth “mother” that did not feed them instead of with a wire “mother” that did. But this is sufficient to test the “hunger-reduction” interpretation of mother love. Babies do not lover their mothers only because she feeds them. See Mook, 1983

17. "Representative design in its full scope requires not only a basic theoretical and methodological restructuring but is a formidable task in practice as well. Ideally, it would take concerted research projects of a magnitude hitherto unheard of in experimental psychology“ • Brunswik, 1956

18. References • Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 379-387. • Juslin, P. (1994). The overconfidence phenomenon as a consequence of informal experimenter-guided selection of almanac items. Organizational Behavior and Human Decision Processes, 57, 226-246.