James Carroll INMM Taos Technical Meeting, May 31, 2012 LA-UR-12-21784

1. Amplifying the  Wisdom of the Crowd, Building and Measuring Expert Consensus for Nonproliferation Issues James Carroll INMM Taos Technical Meeting, May 31, 2012 LA-UR-12-21784

2. Abstract Scientific and cultural progress mostly takes place in the marketplace of ideas. The internet has largely changed how this marketplace functions. Although it has brought many improvements, it has also created a new set of problems. We propose a web service that can be used to crowdsource the process of knowledge acquisition and of knowledge summary, mediate disagreements, and improve discourse on controversial issues, thereby improving the efficiency of the marketplace of ideas. Our approach involves a specific technique for integrating: a forum; a wiki organized into separate camps, with camps organized into a hierarchical structure; a survey system based upon camp support; a mechanism for dynamically reorganizing the structure of the camp hierarchy while mediating user disagreements; and a customizable mechanism for determining how votes are weighted based on credentials or expert assignment. A version of this approach has been implemented, released as open source, and a beta test has gone live. Initial use seems to validate the merits of this approach. (U)

3. Outline • Epistemology and the Marketplace of Ideas • Inefficiencies in the Marketplace of Ideas (In the internet age) • A Potential Partial Solution

4. 1. Epistemology and the Marketplace of Ideas

5. Logic, the quest for truth: Socrates Plato Aristotle

6. Logic, the quest for truth: Socrates Plato Aristotle

7. The Destruction of Objectivity • David Hume (1711-1776) • The Problem of Induction • a question of how things behave when they go “beyond the present testimony of the senses, and the records of our memory” [Hume1748]. • The problem of Generalization (no-free-lunch). • Kurt Gödel (1906-1978) • for any self-consistent recursive axiomatic system powerful enough to describe the arithmetic of the natural numbers (for example Peano arithmetic), there are true propositions about the naturals that cannot be proved from the axioms.

8. No Free Lunch • Thomas Mitchell (The Futility of Bias Free Learning): • There are as many hypotheses consistent with past data that predict one outcome as that predict another off training set (when generalizing). • David Wolpert (The Supervised Learning No Free Lunch Theorem): • When all hypotheses are equally likely, there is no “best” supervised learning algorithm off training set, for the 0-1 loss function. • James Carroll (No-Free-Lunch and Bayesian Optimality): • When all hypotheses are equally likely, then the “best” learning algorithm will report that all outputs are equally likely off training set.

9. Empirical Falsifiable • You can’t prove which theory is true, so stop trying. • Theories should be used that are falsifiable, but which have not yet been falsified after concerted effort. • Such theories will be useful. • Karl Popper (1902-1994)

10. The Marketplace of Ideas “Ideas that can get themselves accepted in a competitive market of ideas will tend to be of better quality than other ideas. The marketplace of ideas therefore improves the quality of our ideas and our thinking. If different • ideas are presented in an “ideas-market”, and if that market is populated by a maximum number of free agents expressing themselves freely, then those competing ideas will be exposed to a maximum number of supporting and dissenting arguments, and the balance of arguments in favor of or against an idea will be compared to the same balance for counter-ideas. The idea with the best balance will “survive”, because alternative ideas will be seen as comparatively defective, given the fact that the arguments in favor of them are weaker or the arguments against them are stronger.”

11. The Marketplace of Ideas • This doesn’t just lead to ideas that are true, but to hypotheses that are useful, in the sense of making the most general predictions about the most things, with the most accuracy. • These survive in the marketplace.

12. Another Rout to the Marketplace: • Rout 2… Induction…. Occam’s razor solves the problem of inference. We compute p(h|data)…. • But we didn’t personally collect all the data, ourselves • Nor can we all do the complex inference for every problem • There are more problems you could be an expert on than there are seconds in your lifetime. • SO… we have to specialize and trade. • But some people lie, so… • We have to determine who to trust, and what data to trust, and ultimately what to believe. • Result: The marketplace of ideas.

13. I, Pencil, My Family Tree as told to Leonard E. Read

14. Collective Intelligence

15. Inefficiencies in the Marketplace of Ideas: • the limited access to information, • the un-manageable volume of information, • the reliability of information, • the need to determine who to consider an expert, • the need to determine what the experts believe, • the difficulty in changing popular notions that are resistant to change, • the incivility, repetition, and lack of focus in discourse, and • the difficulty in overcoming confirmation bias.

16. Current Cultural and Internet Mechanisms to Improve Marketplace Functionality: • Search Engines • Peer Review • Wikis • Forums, Bulletin Boards, and Email Lists • Survey Systems

17. The Canonizer Solution • A wiki to leverage the wisdom of the crowd for knowledge collection and summary • Wiki pages are organized into camps with differing opinions, instead of single pages for each topic • Camps are organized into a hierarchical structure to encourage agreement • A mechanism for dynamically reorganizing the structure of the camp hierarchy while mediating user disagreements to encourage agreement and highlight areas of continuing disagreement • A survey system based upon camp support for determining consensus, and • A mechanism for customizing how votes are weighted based on credentials or expert assignment to allow users to explore different demographic effects, or to allow users to give extra weight to experts in different fields if they so choose. • A forum which also functions as an email list for discussion

20. Weighting Votes (hierarchy of theories of consciousness):

21. Weighting Votes (1 person 1 vote)

22. Weighting Votes (PhDs)

23. Weighting Votes (“expert” philosophers of mind)

24. Weighting Votes (“expert” computer scientists)

25. Deliverables/Results • A history of the debate and conversation • A hierarchy of opinions, driven towards consensus and points of mutual agreement • A single page for each “camp” summarizing the arguments for and against each position. • A survey of participants, showing which view has the most support • A mechanism for varying how the votes are weighted to determine who believes what

26. References • Hume, D., 1747, “An Enquiry Concerning Human Understanding,” p. 108 • Thomas Mitchell, 1980, “The Need for Biases in Learning Generalizations”, Rutgers Computer Science. • David Wolpert, 2001, “The Supervised Learning No Free Lunch Theorem” NASA Ames Research Center. • James Carroll, 2007, “No-Free-Lunch and Bayesian Optimality” in Meta-Learning IJCNN Workshop 2007.