BAMS 517 Decision Analysis: A Dynamic Programming Perspective

BAMS 517Decision Analysis: A Dynamic Programming Perspective Martin L. Puterman UBC Sauder School of Business Winter Term 2011

Introduction to Decision Analysis -outline • Course info • Dynamic decision problem introduction • Decision problems and decision trees • Single decision problems • Multiple decision problems • Probability • Expected value decision making • Value of Perfect Information • Value of Imperfect Information • Utility and Prospect Theory • Finite Horizon Dynamic Programming

Assigning customers to tables in a restaurant Deciding when to release an auction on eBay Choosing the quantity to produce (inventory models) Deciding when to start a medical treatment or accept an organ transplant Playing Tetris Deciding when to add capacity to a system Advanced patient scheduling Managing a bank of elevators Deciding when to replace a car Managing a portfolio Deciding when to stop a clinical trial Guiding a robot to a target Playing golf Some dynamic decision problems In each case there is a trade-off between immediate reward and uncertain long term gain

Common ingredients of these dynamic decision problems • Problem persists over time. • Problem structure remains the same every period. • Current decisions impact future system behavior probabilistically. • Current decision may result in immediate costs or rewards. • These problems are all examples of Markov decision problems or MDPs or stochastic dynamic programs • They were first formulated in the 1940s for problems in reservoir management (Masse) and sequential statistical estimation problems (Wald) • They were formalized in the 1950s by Bellman and Howard. • Theory was developed between 1960-1990. • Rediscovered in the 1990s by computer scientists • Reinforcement learning • Approximate dynamic programming

Basic Decision Analysis

Decision Analysis • Goal: to understand how to properly structure, and then solve, decision problems of nearly any type • Structuring the decision problem and obtaining the inputs is usually the hard part • Once the right structure has been found, solving for the best course of action is usually straightforward • We will be guided by mathematical and scientific principles • These principles will ensure that: • Our decision-making is rational and logically coherent • We choose the best course of action based on our preferences for certain outcomes and knowledge available at the time of the decision • We might not always be satisfied with the outcome but we will be confident with that the process we used was the best available.

Decision Analysis • Our analysis will tell us what decision ought to be taken, as a rational person, and not what decision people actually tend to make in the same situation • Normative (or prescriptive) analysis, rather than a descriptive analysis • Many studies have shown that people do not always act rationally • The methods we introduce provides a framework that translates your preferences for outcomes and your assessments of the likelihood of each consequence into a recipe for action • Places minimal requirements on your preferences and assessments • Does not impose someone else’s values in place of your own • We begin by exploring how to assess the likelihood of outcomes. We will discuss how to determine your preference for outcomes in a few classes.

Simple decision problems • The basic problem is to select an action from a finite set without knowing which outcome will occur • In order to decide on the proper action, we need to: • Quantify the uncertainty of future events • Assign probabilities to the events • Evaluate and compare the “goodness” of the possible outcomes • Assign utilities to the outcomes • Once these are in place, we have fully specified the decision problem

Assessing Probabilities Through Decision Trees

The election stock market problem • Suppose we are faced with the following opportunity on September 8, 2008. • You can pay $.56 and if Obama wins the election you receive $1 and if he loses you receive $0. • http://iemweb.biz.uiowa.edu/graphs/graph_Pres08_WTA.cfm • Decision: Invest $.56 or do not • Uncertain Event: Obama wins. • Suppose this has probability q • The election stock market problem is perhaps the simplest decision problem we will study. It contains, however, all the basic elements of many more complex problems.

The election problem on September 8

A decision tree for the election problem $0.44 Obama wins q Buy 1 share 1-q Obama loses -$0.56 Do not invest $0

Valuing gambles • Under certain conditions (to be discussed in class 3) it is advantageous to evaluate gambles by their mathematical expectation. • For the previous problem the expected value of the gamble would be .44 q - .56 (1-q) = q - .56

Solving the election problem - a reduced problem We replace the gamble by its expectation – latter we use expected utility of the gamble $( q-.56) Buy 1 share Do not invest $0

The election problem solution • Assume you will choose the decision which maximizes your expected payoff. • If you invest, your expected payoff is q-.56; if you do not your expected payoff is 0. Thus if you thought (on September 8) that q, the probability Obama wins exceeds .56, you will invest, if you don’t, you will not invest. • You will be indifferent, when q =.56. • On September 8, the consensus (among investors in the Iowa Electronic Stock Market) probability of Obama winning was 0.56 • Why? • Thus an Electronic Stock Market provides an alternative to polls when predicting outcomes of random events and a method for assessing probabilities. • Current markets • Wikipedia Article (Gives comparison of accuracy compared to polls)

Odds - definitions • If p is the probability an event occurs; o = p/ (1-p) is called the odds of the event occurring • Often we consider l = ln(o) = ln (p/1-p) which is called the log-odds or logit of p. • Aside: this is a key ingredient in a logistic regression model ln(p/1-p) = β0 + β1x • Thus the odds (on September 8) of Obama winning is o = .56/.44 = 1.27 (to one)

Odds - Examples • On December 30, 2008 The Globe and Mail gave the following odds for various teams winning the Super Bowl : • NY Giants 2 to 1 • Tennessee Titans 4 to 1 • Arizona Cardinals 40 to 1 • They have the following meaning in this context; • If you bet $1 on the Cardinals (on Dec 30) and they win the Super Bowl, you get back $41 dollars for a net gain of $40 dollars • The relation of these odds to probabilities can be determined using decision analysis • What is the implied odds makers probability q, that Arizona wins the Super Bowl?

A decision tree for the Super Bowl problem $40 Arizona wins q Bet on Arizona 1-q Arizona loses -$1 Do not bet $0

Solving the Super Bowl problem You would be indifferent between the two decisions if q = 1/41 or 1-q =40/41 $ 41q -1 Bet on Arizona Do not bet $0

Odds and Bookmaker’s Odds • Based on the decision tree and expectations the probability of winning is 1/41 • So using the above definition of odds you would find the odds of winning is oW = 1/41/ (1 - 1/41) = 1/40 (to 1) • The odds of losing would be oL = 40/41/ (1-40/41) = 40 (to 1) • Thus quoted odds for sports events are the odds of losing. • Hence the odds (on December 30) the Giants don’t win the Super Bowl are 2 to 1 and the odds they win the Super Bowl are 1 to 2. • The implied probability the Giants win the Super Bowl is 1/3. • Another interpretation (courtesy Wikipedia) • “Generally, 'odds' are not quoted to the general public in the format (p/1-p) because of the natural confusion with the chance of an event occurring being expressed fractionally as a probability. ” • Example – Suppose that you are told to pick a digit from 0 to 9. Then the odds are 9 to 1 against you choosing a 7. One way to think about this interpretation is that there are 10 outcomes in 1 you succeed in picking a 7 and in 9 you don’t succeed. • This interpretation doesn’t work for one time events like the Super Bowl. • I’ll refer to these as ‘bookmaker’s odds”.

Games of Chance and Odds The payout on a successful bet on a single number is 35 to 1 plus the amount bet. The true bookmaker’s odds are 37 to 1 on an American roulette wheel (with 0 and 00). (assuming a fair wheel)

A decision tree for a single number bet in roulette $35 Ball stops on 7 1/38 Bet $1 on 7 37/38 Lose -$1 Do not bet $0

Solving the roulette problem -$ 0.0526 Bet $1 on 7 Do not $0

Roulette • Based on the previous table; every $1 bet in roulette has an expected value of negative $0.526. • Thus roulette is an unfavorable game. • Note there is research on how to play unfavorable games optimally based on dynamic programming. • But; if you play many times, and the wheel is fair, you will lose money. • Why do people play?

Money Lines or “odds sets” • Another way of expressing odds. • Used frequently for hockey and baseball betting. • The Globe and Mail (December 31,2008) • In an NHL game the favorite Calgary has a line of -175 and Edmonton the underdog has a line of +155. • This means that if you want to bet on Calgary, you must bet $175 to win $100 and if you want to bet on Edmonton you must bet $100 to $155. • This implies • P(Calgary) = 7/11 = .636 • P(Edmonton) = 20/51 = .392 • What’s happening? What about ties? • Suppose the Calgary probability is correct, (and ties are not possible), then the probability of Edmonton winning should be 1-.636 = .354 and the money line on Edmonton should be 636/.354 = 180! • So “the House” is taking $25 off the payout on a winning Edmonton bet. • The same argument for Calgary implies ? • So again like in roulette “the House” is taking a premium on every bet by reducing the payoff below the expected value of the gamble.

Assigning probabilities to events • The uncertainty of an event will be measured according to its probability of occurrence • For events that have been repeated several times and regularly observed, it’s easy to assign a probability: • Outcomes of gambling games: • Tossing coins, rolling dice, spinning roulette wheels, etc. • Actuarial and statistical events: • A 30-year-old female driver having an accident in the next year • The chance of rain tomorrow, given today’s weather conditions • The number of cars driving over the Lion’s Gate bridge tomorrow between 8 and 9 AM • The number of admits to the emergency room at VGH on January 7, 2009

Assigning probabilities to events • However, not all events occur with statistical regularity: • General Motors will be bankrupt by July 1, 2010 • A democrat will win the 2012 US presidential election • The uncertainty of an event often derives from a lack of precise knowledge • How many jellybeans are in a jar? • Was W.L. MacKenzie King Prime Minister of Canada in 1936? • Or there is not much data available • Will a new medical treatment be effective in a specific patient? • Since these events cannot be repeated in any meaningful way, how can we assign a probability to their occurrence? • We can rely on election stock markets or odds if they are available. • What if they’re not?

Assigning probabilities to events • It is important to recognize that two different people in the same situation might assign two different probabilities to the same event • A probability assignment reflects your personal assessment of the likelihood of an event – the uncertainty being measured is your uncertainty • Different people may have different knowledge about the event in question • Even people with the same knowledge could still differ in their opinion of the likelihood of an event • Someone could coherently assign a probability of ¼ to a coin coming up heads, if he/she had reason to believe the coin is not fair • They are often called subjective probabilities. • The assessment of subjective probabilities is a key topic in research on decision analysis (and forecasting)

Assigning probabilities to events • Example; Suppose we wished to assign a probability to the event “A thumb tack lands with its point up” • How we could we find this probability? • We could guess. • We can gauge our belief of the likelihood of an event by comparing it to a set of “standard” statistical probabilities through a reference lottery. • We can compare the following two gambles to assess this probability: Choice A: Toss the thumbtack. If it lands point up, you win $1; otherwise you receive $0 Choice B: Spin the spinner, If it ends on blue, you win $1; otherwise you receive $0 • We can adjust the portion of the spinner that is blue until we are indifferent between the two choices. • This Probability spinner provides a way of varying the blue portion systematically.

The implied decision tree +1 Thumbtack land up Choice A Thumbtack lands on “tip down” 0 +1 Spinner blue Choice B Spinner red 0

Implications of using reference lottery • If the spinner is set so that the probability of blue is .5 and you prefer A to B, then you believe the probability “thumbtack up” is greater than .5 • If the spinner is set so that the probability of blue is .9 and you prefer B to A, then you believe the probability “thumbtack up” is less than .9 • Repeating this can give a plausible range for the probability of “thumbtack up” . • This is hard to do! • There is a big literature on biases of such assignments. • Alternatively we could construct a distribution of plausible values for this probability and the likelihood of each of these values instead of assigning one number. • Or we could input our assessment into the decision problem and do sensitivity analysis.

Another option – acquire information • Suppose you were faced with Choice A only? What would this gamble be worth? • One approach; provide a prior distribution on the probability of the event p. • Example: Uniformly distributed on [0,1]. • Base the decision on the mean, median or mode of this distribution. • Toss the thumbtack once, and use Bayes’ theorem to update this probability.

Assigning probabilities to events • Let E be an event, and let H represent the knowledge and background information used to make a probability judgment. We denote the assigned probability as P(E|H) • “The probability of event E given information H” • We do not consider probabilities as separate from the information used to assess them • This reflects the fact that we consider all probabilities to be based on the judgment of an individual and the individual’s knowledge at the time of the assessment. • Even though we consider probabilities to based on an individual’s judgment, they cannot be arbitrarily assigned • Certain rules must be obeyed for the assignments to be coherent • Using the method outlined above to assign probabilities avoids incoherent assignments

Axioms of Probability • The probability assignments P(E|H) must obey the following basic axioms: • 0 ≤ P(E|H) ≤ 1 • (Addition law) Suppose that E1 and E2 are two events that could not both occur together (they are mutuallyexclusive). Then P(E1 or E2|H) = P(E1 | H) + P(E2|H) • If E1 and E2 are mutually exclusive and collectively exhaustive, then P(E1 or E2|H) = P(E1 | H) + P(E2|H) = 1 • (Multiplication law) For any two eventsE1andE2, P(E1 and E2|H) = P(E1 | H)P(E2|E1 and H) If E1 and E2 are independent (i.e., P(E2|E1 and H) = P(E2|H)), then P(E1 and E2|H) = P(E1|H)P(E2|H) • These rules can be used to compute probability assignments for complex events based on those for simpler events

The law of total probabilities • This law can be derived from the axioms and the definition of conditional probability. It says that for any two events A and E, • P(A|H)= P(A and E|H) + P(A and Ec|H) = P(A|E and H)•P(E|H) + P(A|Ec and H) • P(Ec|H) • This law is useful because it allows one to divide a complex event into subparts for which it may be easier to assess probabilities. • Also it generalizes to more than just conditioning on E and Ec. • We can replace it by any set (or continuum) of events that partitions the sample space. • It is used widely in probability theory to compute complex probabilities and is fundamental for evaluating Markov chains

Bayes’ rule • This is a very important rule that we will use extensively. • It is a way to systematically include information in assessing probabilities • To simplify notation lets drop the conditioning on H and assume that it is understood that probabilities are conditional on history. • Bayes’ Rule can be written as • It is derived using the definition of conditional probability and the law of total probabilities • It generalizes to any set of events that partitions the sample space.

Updating probability assessments • Suppose that you can’t see inside a jellybean jar containing only red and white beans, but I tell you that either 25% of the beans are red or 25% are black. You think these possibilities are equally likely. • Suppose I pick a bean at random. What is the probability it is red? • Now you draw 5 jellybeans from the jar with replacement, and find that 4 of them are red. How should you revise your belief in the probability that 25% of the beans are red, in light of this information? • Let A be the event “25% of beans in the jar are red” and let E be the event of drawing 5 beans and obtaining 4 red and 1 black • We want to find P(A|E) and we know: • P(A) = P(Ac) = 0.5 • P(E|A) = .75 (.25)4 = 0.00293 • P(E | Ac) = .25 (.75)4= 0.0791

Updating probability assessments • Using Bayes’ rule, we now compute • P(A|E) = .00293(0.5) / [.00293(0.5) + .0791(0.5)] = .0357 • Thus, you should now believe that there is about a 3.5% chance that 25% of the jellybeans are red. • You also think there is about a 96.5% chance that 25% of the beans are black • Obviously, we have received strong evidence regarding the contents of the jar, since our beliefs have gone from complete uncertainty (50%) to high probability (96.5%) • Let’s look at some of the terms involved in Bayes’ rule: • The expression P(A|E) is known as the “posterior” probability of A, i.e., the assessed probability of A after we learn that E has occurred • P(A) is known as the “prior” probability of A, i.e., the assessment of the probability of A before the new information was received • P(E | A) is known is the “likelihood” of E occurring given that A is true

Probabilities for single events • To follow on the previous example, let’s now ask what we think the probabilities are now for the next bean drawn from the jar to be red • We assign a probability of .965 to there being 75% red beans in the jar, and a chance of .035 to there being 25% red beans in the jar • Let A = “75% of the beans in the jar are red” • Let Ac = “25% of the beans in the jar are red” • Let B = “The next bean drawn from the jar is red” • We use the law of total probability to compute P(B): P(B) = P(B | A) P(A) + P(B | Ac) P(Ac) = .75(.965) + .25(.035) = .733 • P(B) is called a marginal probability • What was this probability before sampling?

The “Monty Hall” problem • Monty Hall was the host of the once-popular game show “Let’s Make a Deal” • In the show, contestants were shown three doors, behind each of which was a prize. The contestant chose a door and received the prize behind that door • This setup was behind one of the most notorious problems in probability • Suppose you are the contestant, and Monty tells you that there is a car behind one of the doors, and a goat behind each of the other doors. (Of course, Monty knows where the car is) • Suppose you choose door #1

The “Monty Hall” problem • Before revealing what’s behind door #1, Monty says “Now I’m going to reveal to you one of the other doors you didn’t choose” and opens door #3 to show that there is a goat behind the door. • Monty now says: “Before I open door #1, I’m going to allow you to change your choice. Would you rather that I open door #2 instead, or do you want to stick with your original choice of door #1?” • What do you do?

Summary • Sequential Decision Problems • Decision Trees • Probability Assessment, Odds and Gambling • Probability updating • Monty Hall and Hatton Realty for next time.

BAMS 517 Decision Analysis: A Dynamic Programming Perspective