Game Theoretical Modeling and Studies of Peer-Reviewing Methods

Game Theoretical Modeling and Studies of Peer-Reviewing Methods Marius C. Silaghi Florida Tech CS Seminar, Fall 2012

Peer-Reviewing Games • Game Theory Concepts • Peer-Reviewing Background • Model Peer-Reviewing as Game • Theoretical analysis of PR Games • Experimental analysis of PR Games

Game Theory • Is not as much about computer games.

Game Theory • It is about understanding motivations (utilities $$$).

Fundamentals of Game Theory • Understanding what will happen in a given situation • Typical example: “Prisoner’s Dilemma” • Golden Balls Trust • Golden Balls strategies Each can rat out the other or remain silent, resulting in 4 possible outcomes

Fundamentals of Game Theory • Understanding what will happen in a given situation • Typical example: “Prisoner’s Dilemma” • Perfectly rational players • only care for their own felicity (utility) • Payoff matrix utility of x, utility of y

Fundamentals of Game Theory • Understanding what will happen in a given situation • Typical example: “Prisoner’s Dilemma” • Perfectly rational players • only care for their own felicity (utility) • Payoff matrix utility of x, utility of y Mechanism design is about selecting right payoffs to encourage a “social choice” function: W. Vickrey got Nobel Prize in 1996 for 2nd Price Auctions for “truthful bidding”.

Iterated Prisoner’s Dilemma • The game repeats every 20 years for 1000 years  • Strategies: • Tit for Tat • Forgiving Tit for Tat • Optimistic Tit for Tat

Iterated Prisoner’s Dilemma • The game repeats every 20 years for 1000 years  • Strategies: • Tit for Tat • Forgiving Tit for Tat • Optimistic Tit for Tat Strategy Equilibrium studies try to predict behavior in existing games by theoretically or experimentally analyzing their impact on a player’s utilities. A player utilities define its type Player will select best strategy given strategies currently used by other participants.

Rational Players(… = predictable) • Rational players are ones that predictably try to maximize their utility. • Utility can be expressible in $$$. • Can most people can be assumed rational? • (if given enough time and help to think) • One has to take into account the utilities as defined by the beliefs (type) of the given player

Rational Players(… = predictable) • Are avaricious/epicurean/workaholic people rational? • One has to take into account the utilities as defined by the beliefs (type) of the given player. • - obviously, avaricious people believe in the value of the dollar. • If one is alive, one likely has (subjective) beliefs. Can they be manipulated? Yes, since they are predictable: promise’em what they want.

Rational Players(… = predictable) • Are seekers of fame rational? • One has to take into account the utilities as defined by the beliefs (type) of the given player. • - a player that believes in immortality through fame will value fame (quantifiable in money). Tiberius Temple of Artemis in Ephesus burned on July, 356 BC by Herostratus who wished that: ”His name be spread to the whole Earth.” Can they be manipulated? To dissuade copycats, Ephesians ruled that his name should never be pronounced.

Rational Players(… = predictable) • Is a “fanatical altruist” rational? • One has to take into account the utilities as defined by the beliefs (type) of the given player. • - an altruist will be happier if he believes that “others” (family, country, humanity, animals) are happier. Can he be manipulated? Coax him by claiming that “others” love to be bombed!

Rational Players(… = predictable) • Is a religious person rational (maximizing a utility)? • He (concurring with cynics) likes to claim he is not looking for $ / (i.e. rational). • One has to take into account the utilities as defined by the beliefs (type) of the given player. • - for a player that has a degree of belief in afterlife, • his utilities are a function of that religion (believed mechanisms to reach afterlife). Can he be manipulated? Devil lies in details: claim that the correct interpretation of ‘A’ is ‘D’.

Game Theory • It is about understanding motivations (utilities). • Used in real war • Used in designing how an enterprise/country works • Used commonly in macro-economy • Can be used for peer-reviewing • For computer science: • Used in Multi-agent Systems • It is a computational problem (require simulations, models, etc.) Motivation Machine

Game TheoryUsed in real war • Why do soldiers obey their commanders in the army of the enemy? • Could one offer them something such that they defect? • Could one make them think that they (or their country) is better capitulating? • (whether that is true or false) • Why does the president of the enemy country fight for that system? • Could one blackmail/bribe/menace/convince/confuse him into quitting? Most research and funding for game theory seems to be here.

Game TheoryUsed in designing a country • Nicolo Machiavelli, 1469-1527 • Italian ex-politician of the Republic of Florence • Discourses on Livy • El Principe

Game TheoryUsed in designing a country • People obey laws (for fear of police). • besides brainwashing in schools/media • Police obeys (for fear of army). • Army obeys (for fear of secret services). • Secret services obey (for high pay, or fear of another secret service). • checks and balances  • good fences make good neighbors • Poorly designed countries (with the incentives missing/disappearing for some ring in the chain), have been seen collapsing in spectacular manners: • Roman Empire (motivation of soldiers?) • USSR (motivation of KGB?) • Yugoslavia (brainwashing failure, motivation of states in confederation?) 9 countries (11 states) 15

Game TheoryUsed commonly in macro-economy • What combination of fees/taxes/subventions would lead to a strong economy? • (where resources end-up in the hand of those who know/can make most out of them, etc.) • E.g.: subventions only to people owning over 50ha, • concentrates farming land in hands of those who have money for machines and technology.

Game Theory • Can be used for peer-reviewing: • Peer-reviewing is the foundation of modern scientific research and • controls the speed of the development and • significant decisions on allocation of funding.

Index • Game Theory Concepts • Peer-Reviewing Background • Model of Peer-Reviewing as Game • Theoretical analysis of PR Games • Experimental analysis of PG Games • Conclusions

Peer-Review • Features of PR mechanisms keep getting richer to improve and encourage research quality. • Blind reviewing, • Author's reply to comments by reviewers, • Reviewers bid for papers, • Authors rate reviewers, • Authors blacklist reviewers Most scientists regarded the new streamlined peer-review process as ‘quite an improvement.’ scienceforseo.com

Common Blind Peer-Reviewingfor Conferences • Chair assigns each paper to a Senior PC • SPC distributes a paper to 3 PC members (bidding) • PC gives a paper to a reviewer student • Each student reviews and assigns score. • Author sees reviews and answers • Student/PC may change review • PC forwards review to SCP • SCP gathers 3 reviews: rejects if any reject • Chair makes last changes: • Applies threshold (1/10 papers) • Answers complains Chair Accept 7 Senior Program Committee Reject 3 Program Committee Accept 5

Open Peer-Review (e.g. Material Thinking Design Workshop, 2007)Mainly with journals (biology) Proceedings • Reviewers bid on papers. Papers distributed to 3 reviewers. • Each reviewer writes a short article with the review of the paper. • Authors see reviews and: • can withdraw paper, or • may write for each review a short article with an answer. • Reviewers see answers and can withdraw their reviews. • Papers are published together with reviews and answers. • Paper with negative reviews (no withdrawn) are • published as technical reports together with the reviews. main article review articles review answer articles

Open Peer-Review Papers without reviews. What to do? Proceedings • Understanding facts/possible motivations/conclusions • nobody accepted to review • likely not relevant to community, or • reluctance to write negative reviews • but may also be a boycott • fair: tech rep for an archiving fee • all reviews are withdrawn / not submitted in time: • (paper’s fault) could be irrelevant or poor quality • (reviewer’s fault) overcommitted, malicious strategy • assigned reviewer names should be published • accepted or rejected? A 3rd category! • The number of non-reviewed reports is a measure of the quality of the symposium/community. • usable for deciding whether to submit similar articles in the future. main article ? review articles review answer articles

Index • Game Theory Concepts • Peer-Reviewing Background • Model of Peer-Reviewing as Game • Theoretical analysis of PR Games • Experimental analysis of PG Games • Conclusions

Players • Game players: • The researchers - Authors and reviewers, • a repeated/iterated game at each conference • Funding Agency - rewards researchers based on their publications. • mechanism designer, or • player in a game with the researchers next slides with help of R. Vishen

Concepts • Model paper quality by a paper's worth- utility to the society. • Worth is evaluated by expert reviewers. • Assumption: All reviewers in the symposium are equally expert. Reviewers have the same type (association paper  worth)

Assumption of equally expert reviewers • Note: we fail to model people emotionally attached to antagonistic scientific beliefs in a community: • scientists believing in climate warming vs. unbelievers • scientists believing (or not) in the relevance of a given metric: • is/isn’t privacy more important than verifiability (in voting) • is network logic runtime more relevant than real runtime? • scientists with a given emotional belief should probably create their own communities/conferences.

Model • Authors and Reviewers expect rewards from a funding agency. • Assumption: funding agency intends to maximize social value • Social value is defined as “sum of quality of the endorsed papers.” • An article is endorsed if it is published with favorable reviews by experts. • Given a set of texts appearing in a community of type t, the social choice function is : • Maximizing the total utility: trouble

Publication venues and social value • Conferences have multiple venues: • orally presented papers • posters • technical reports

Publication venues and social value • Venues gives a way to automate the accounting of the paper worth, via its impact on the visibility (number of citations): • Technical reports are less endorsed than posters or orally presented papers. • The social value given a set of publication venues ψ (posters, regular papers, etc.) consists of the weighted sum of the worth of the published papers (assume measurable via citation influence).

Utilities (Motivation)Let us convert it all to • Funding agency settings for distributing funds (rewards): • Citation Influence, • Publications count. • Assumption: the funding agency cannot access the worth of a paper directly.

Citation Influence • The citations influence (CI) of an author at a given moment is a metric of the influence of his publications, and it estimates the weighted sum of the worth of his publications on each of the three venues Ψ, Ψ = {regular, poster, technical report}

Utilities (Motivation)Let us convert it all to: • A researcher gets reputation (positive utility) when papers are cited. • often one cannot automatically distinguish good vs bad citations. • A researcher can get bad reputation (negative utility): • for publishing erroneous articles, (as pointed by citations/reviews) or • if his/her review is proven to be incorrect. (only with Open PR) E=mc2

Utilities (Motivation)Let us convert it all to: What about reviewing (what is the motivation?) Being asked to review is like a citation (proof of reputation). To be asked again one has to promptly review when requested. But why writing good reviews rather than random ones? - Fear of “Authors scoring of reviewers” (not typically valuable: tit for tat) - SPC cannot generally notice poor reviews. - (when noticed, no mechanism to disseminate it)

Utilities (Motivation)Let us convert it all to: Deviation from truthful reviewing may pay. • Reviewing takes time (writing random reviews earns you time) • Conferences with thresholds are zero-sum games. • A paper “p” is superseded by a newer paper “n” when the new paper “n” steals the show for “p” (reduces future citations of p).

Citation Influence with superseding • The citations influence (CI) of an author at a given moment is a metric of the influence of his publications, and is given by the weighted sum of the worth of his un-superseded publications of each of the three venues Ψ, Ψ = {regular, poster, technical report}

Index • Game Theory Concepts • Peer-Reviewing Background • Model of Peer-Reviewing as Game • Theoretical analysis of PR Games (#-based) • Experimental analysis of PG Games (CI-based) • Conclusions

Funding Based on Counting articles(Trusted Peer-Reviewing) • Rewards author i based on the number and venue of publications. • Paper superseding not relevant. • Conferences: • with threshold on paper acceptance rate (1/10, 1/20) • without thresholds on paper acceptance

Funding Based on Counting articles(Trusted Peer-Reviewing)No thresholds on paper acceptance rates • Two-players conference, with one submission each, • with {accept, reject} decisions, the payoff matrix is: One-shot game: Best strategy is “make random decision”. Iterated game: Effective strategy is “(forgiving) Tit-for-Tat”

Funding Based on Counting articles(Trusted Peer-Reviewing)Paper Acceptance Thresholds • Conference in order to remain relevant to the funding agency puts a threshold on the ratio of accepted papers. (CBR  CBRz) • In this new version the actions available to players are not {accept, reject}, but the scores {low, high}. • In case of tie, a paper is randomly selected. • 2-players case: Zero-sum game.

(Trusted Peer-Reviewing)Multiple playersPaper Acceptance Thresholds • n-players case (accepting n/k articles, equally worthy submissions): • Zero-sum game. • not pair-wise Zero-sum game • Utility of rejecting a paper (one less competitor): For small CBR communities, a dominant strategy is “always low”. For huge CBR communities, a dominant strategy tends to “random review”. Always “low” is a Nash equilibrium (observed in some small communities)! With Tit-for-Tat opponents, Nash equilibrium is “always low” if the opponent is not met again for (n-1) rounds. With Hits-for-Tat opponents, Nash equilibrium is “always high” Hits-for-Tat: an opponent can strike back many (m>>1) times for one Tat expected penalty:

Truthful reviewing for TPRConclusion • With counting of articles: Truthful reviewing is never in equilibrium. • (under the working assumptions: mainly that nobody notices how you review).

Index • Game Theory Concepts • Peer-Reviewing Background • Model of Peer-Reviewing as Game • Theoretical analysis of PR Games (#-based) • Experimental analysis of PG Games (CI-based) • Conclusions

Experimental studiesSimulations • Assumption: Funding based on CI • Compared mechanisms: • Open Peer-Review (SelectivitY) • Common Blind Review (CBR) • Common Blind Review with paper acceptance threshold (CBRz)

Evaluation – Simulation Experiment • Generate 100 random research communities (i.e. simulations) • 20 researchers • 20 conferences (i.e. 20 years) • All participants are considered equally expert and inventive. • Researchers get ideas for articles with a Uniform distribution at an average of (only) 2 articles per year, and a worth that is uniformly random in [-10, 10]. • Each paper is superseded each year with a probability of 1/5. • Experiments for the weights wo = 0.5, wp = 0.3, and wt = 0.1.

Evaluation - Reviewer Types • Compared Review Strategies i. Truthful reviewing. ii. Truthful reviewing except for papers superseding one’s work, which are rejected. iii. Random reviewing except for papers superseding one’s work, which are rejected. iv. Giving the opposite possible score to all papers (reject good papers and accept poor papers). v. Giving the lowest possible score to all papers. (Tit-for-Tat was not explored here: assumed to have limited relevance)

Evaluation – ExperimentAssumptions about worth of reviews • A misclassifying comment has a negative worth. • if published, it will be accounted only as bad reputation for the reviewer. • Misclassifying comment worth - the difference between the corresponding value and the real worth of the paper (always negative). • The worth of an author’s answer to a negatively misclassifying comment is the same as the “absolute value of the worth of the misclassifying comment”. Otherwise, the answer has worth zero.

Game Theoretical Modeling and Studies of Peer-Reviewing Methods