1 / 42

Organizing Open Online Computational Problem Solving Competitions

Organizing Open Online Computational Problem Solving Competitions. By: Ahmed Abdelmeged. In 2011, researchers from the Harvard Catalyst Project were investigating the potential of crowdsourcing genome-sequencing algorithms.

delta
Download Presentation

Organizing Open Online Computational Problem Solving Competitions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Organizing Open Online Computational Problem Solving Competitions • By: Ahmed Abdelmeged

  2. In 2011, researchers from the Harvard Catalyst Project were investigating the potential of crowdsourcing genome-sequencing algorithms.

  3. So, they collected few million sequencing problems and developed an electronic judge that evaluates sequencing algorithms by how well they solve these problems.

  4. And, they set up a two-week open online competition on TopCoder with a total prize pocket of $6000.

  5. The results were astounding!

  6. -- Nature Biotechnology, 31(2):pp. 108–111, 2013. • “... A two-week online contest ... produced over 600 submissions ... . Thirty submissions exceeded the benchmark performance of the US National Institutes of Health’s MegaBLAST. The best achieved both greater accuracy and speed (1,000 times greater).”

  7. We want to lower the barrier to entry for establishing such competitions by having “meaningful” competitions where participants assist the admin in evaluating their peers.

  8. Thesis Statement • “Semantic games of interpreted logic sentences provide a useful foundation to organize computational problem solving communities.”

  9. Semantic Games • A Semantic Game (SG) is a constructive debate of the correctness of an interpreted logic sentence (a.k.a claim) between two distinguished parties: the verifier which asserts that the claim holds, and the falsifier which asserts that the claim does not hold.

  10. A Two-Party, SG-Based MAX-SAT Competition (I) • Participants develop functions to: • Provide side preference. • Provide values for quantified variables based on values of variables in scope. ∀φ ∈ CNFs ∃v ∈ assignments(φ)∀f ∈ assignments(φ). fsat(f,φ)≤fsat(v,φ)

  11. A Two-Party, SG-Based MAX-SAT Competition (II) • Admin chooses sides for players based on their side preference. • Let Pv be the verifier and Pf be the falsifier. ∀φ ∈ CNFs ∃v ∈ assignments(φ)∀f ∈ assignments(φ). fsat(f,φ)≤fsat(v,φ)

  12. A Two-Party, SG-Based MAX-SAT Competition (III) • Admin gets value provided by Pf for φ. • Admin checks φ ∈ CNFs. If false, Pf loses. • Admin gets value provided by Pv for v. • Admin checks v ∈ assignments(φ). If false, Pv loses. ∀φ ∈ CNFs ∃v ∈ assignments(φ)∀f ∈ assignments(φ). fsat(f,φ)≤fsat(v,φ)

  13. A Two-Party, SG-Based MAX-SAT Competition (IV) • Admin gets value provided by Pf for f. • Admin checks f ∈ assignments(φ). If false, Pf loses. • Admin evaluates fsat(f,φ)≤fsat(v,φ). If true Pv wins, otherwise Pf wins. ∀φ ∈ CNFs ∃v ∈ assignments(φ)∀f ∈ assignments(φ). fsat(f,φ)≤fsat(v,φ)

  14. Rationale (I) • Controllable admin overhead. ∀φ ∈ CNFs ∃v ∈ assignments(φ)∀f ∈ assignments(φ). fsat(f,φ)≤fsat(v,φ) ∀φ ∈ CNFs ∃v ∈ assignments(φ). satisfies-max(v,φ)

  15. Rationale (II) • Correct: there is a winning strategy for verifiers of true claims and falsifiers of false claims. Regardless of the opponent’s actions.

  16. Rationale (III) • Objective. • Systematic. • Learning chances.

  17. Challenges (I) • Participants must take opposing sides! • Neutrality is lost with forcing.

  18. Challenges (II) • Scaling to N-Party Competition using a tournament, yet: • Avoid Collusion Potential especially in the context of open online competitions where Sybil identities are common and games are too fast to spectate! • Ensure that participants get the same chance.

  19. Semantic Game Tournaments

  20. Tournament Design • Scheduler: • Neutral. • Ranking function: • Correct and anonymous. • Can mask scheduler deficiencies.

  21. Ranking Functions • Input : beating function representing output of several games. • Output: a total preorder of participants.

  22. Beating Functions (of SG Tournaments) • bP(pw, pl, swc, slc, sw) : sum of all gains of pw against pl while pw choosing side swc , pl choosing side slc and pw taking side sw. • More complex.

  23. Ranking Functions (Correctness) • Non-Negative Regard for Wins. • Non-Positive Regard for Losses.

  24. Non-Negative Regard For Wins (NNRW) Px Additional wins cannot worsen Px’s rank w.r.t. other participants. Wins Faults

  25. Non-Positive Regard For Losses (NPRL) Implies: Px Additional faults cannot improve Px’s rank w.r.t. other participants. Wins Faults

  26. Ranking Functions (Anonymity) • Output ranking is independent of participant identities. • Ranking function ignores participants’ identities. • Participants also ignore their opponents’ identities.

  27. Limited Collusion Effect • Slightly weaker notion than anonymity. • What you want in practice. • A participant Py can choose to lose on purpose against another participant Px, but that won’t make Px get ahead of any other participant Pz.

  28. Limited Collusion Effect (LCE) Px Games outside Px’s control cannot worsen Px’s rank w.r.t. other participants. Wins Faults

  29. Discovery • A useful design principle for ranking functions. • Under NNRW, NPRL : LCE = LFB • LFB is quite unusual to have. • LFB lends itself to implementation.

  30. Locally Fault Based (LFB) Relative rank of Px and Py depends only on faults made by either Px or Py. Px Py Wins Faults Faults Wins

  31. Fault Counting Ranking Function • Players are ranked according to the number of faults they make. The less the number of faults the higher the rank. • Satisfies the NNRW, NPRL, LFB and LCE properties.

  32. Semantic Game Tournament Design • For every pair of players: • If choosing different sides, play a single SG. • If choosing same sides, play two SGs where they switch sides.

  33. Neutrality • Each player plays nv+ nf - 1 SGs in their chosen side, those are the only games it may make faults.

  34. Related Work • Rating and Ranking Functions • Tournament Scheduling • Match-Level Neutrality

  35. Rating and Ranking Functions (I) • Dominated by heuristic approaches • Elo ratings. • Who’s #1? • There are axiomatization of rating functions in the field of Paired Comparison Analysis. • LCE not on radar. • Independence of Irrelevant Matches (IIM) is frowned upon.

  36. Rating and Ranking Functions (II) • Rubinstein[1980]: • points system (winner gets a point) characterized as: • Anonymity : ranks are independent of the names of participants. • Positive responsiveness to the winning relation which means that changing the results of a participant p from a loss to a win, guarantees that p’s rank would improve. • IIM: relative ranking of two participants is independent of matches in which neither is involved. • “beating functions” are restricted to complete, asymmetric relations.

  37. Tournament Scheduling • Neutrality is off radar. • Maximizing winning chances for certain players. • Delayed confrontation.

  38. Match-Level Neutrality • Dominated by heuristic approaches • Compensation points. • Pie rule.

  39. Conclusion • “Semantic games of interpreted logic sentences provide a useful foundation to organize computational problem solving communities.”

  40. Future Work • Problem decomposition labs. • Social Computing. • Evaluating Thoroughness.

  41. Questions?

  42. Thank You!

More Related