The Scientific Community Game as A Crowdsourcing Platform to Distinguish Good from Bad

1. The Scientific Community Game asA Crowdsourcing Platform to Distinguish Good from Bad Presentation to Clients by Software Development Organization

2. Crowdsourcing Platform • Crowdsourcing • is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call. • enlists a crowd of humans to help solve a problem defined by the system owners. • A crowdsourcing platform is a generic tool that makes it easy to develop a crowdsourcing system.

3. Crowdsourcing Platform • The job, target problem is • to solve instances of a problem and make claims about the solution process. • to build knowledge base of claims and techniques to defend the claims

4. Requirements for Crowdsourcing Platform • Find a good way to combine user contributions to solve the target problem. • Find a good way to evaluate users and their contributions. • Find a good way to recruit and retain users.

5. Combine user contributions • We found an excellent way. • Users build on each others work: strengthening. • Users check each others claims for correct judgment. • Claims are defended and refuted. • Users trade reputation and information.

6. Voting with Justification • I vote • for this claim (agree) because I can defend it and refute its negation. • against this claim because I can oppose it (refute or strengthen).

7. Evaluate users and their contributions • Calculate reputation • confidence by the proposer that a claim is good (gc) • confidence by the opposer (refute or strengthen) that the claim is bad (bc) • The scholars are encouraged to set their confidences truthfully. Otherwise they don't gain enough reputation or they lose too much reputation.

8. Reputation Update up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose up: reputation goes up, but has to provide knowledge that might reveal secret technique. down: reputation goes down, but might gain knowledge that reveals secret technique.

9. Reputation Update up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose • confidence: • proposer: claim is good: gc • opposer: claim is bad: bc • r = result of reputation protocol. • Reputation update: r*gc*bc (various refinements are possible)

10. Perfect • Being perfect means to make perfect decisions. • up: if you are perfect, you will not lose. • down: if the other is perfect, you will not win. up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose

11. Result: We have successfully developed the SCG Crowdsourcing platform: SCG Court • Requirements Document • User’s Guide • Continuous Testing Approach • Well documented software with Design Document for Maintainers • Acceptance Test • CSP tournament • HSR tournament • NetworkFlow tournament

12. Software Development Process • Customized Spiral Process using continuous testing • http://groups.csail.mit.edu/pag/continuoustesting/ • Software Management Approach • Motivating software developers with interesting project that provides numerous learning opportunities. • Overcoming initial resistance by persistence and significant involvement of teaching staff. Adding Ahmed.

13. Software Management Approach • Hire good software developers with significant initiative and involvement in the project. • Flexible project organization: very open to numerous requirements changes. • Trusting software developers with significant responsibilities. • Greek Gods: found subproject that was optimal for their time table. • Dan Marcucci: Let’s use DemeterF the right way.

14. Software Management Approach • I amplified the good ideas that I spotted and did not force them on you! • I continued to refine the game with Ahmed and Yue and we carefully justified all requirements changes. • Reto: continuous testing approach. Setting up svn on assembla.

15. Domain of Requests and Responses • Three kinds of requests • Instance Requests • Response: Instance • Boolean valid(Instance) • Solution Requests • Response: Solution • Boolean valid(Instance,Solution) • Claim Analysis Requests • Response: good/bad decision • determine which claims are • good: agree • bad: refute or strengthen • Make prediction about how protocol will run • Claims are about the relationship between instances and their solutions

16. What is good/bad? • A claim is • good • if both scholars agree • if it is predominantly defended • bad • if scholars don’t agree • if it is predominantly refuted • Refutation is the complement of defense and is based on the requests and responses exchanged.

17. Good/ Bad Meaningfor Mathematical Claims • Good • true claim: if there is no strengthening relation between claims (strengthen(c1,c2) is false if c1!=c2.) • optimal claim: if there is a strengthening relation between claims. • Bad • false claim • non-optimal claim