Distributed Optimization

1. Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010

2. Outline [Yu] • Optimized Sensing: From Water to the Web • Distributed Dynamic Programming • Distributed Solutions to Markov Decision Problems

3. Optimized Sensing • Problem Statement • Greedy Algorithms and Submodularity • Robust Sensing Optimization with Saturate Algorithm • Application in Blogs

4. Problem Statement • How do we detect contamination in drinking water distribution networks? • Which blogs should we read to learn about the biggest, newest stories on the Web? • Fundamental Question: How can we get the most useful information at minimum cost (limited resources)?

5. Solutions to Optimized Sensing • Covers fields of statistics, machine learning, sensor networks, and robotics • With partially observable Marko decision processes, we can get optimal solutions • But it is difficult to scale POMDP to large problems • Introducing a new algorithm based on submodularity

6. Formulation • Sensing quality function F(A) • A: the set of sensor locations Si (i=1~k) • V: the set of all locations • We can also have cost constraints • Total cost of sensor deployment no greater than the budget • Goal: Find A* • This is NP-hard already

7. Greedy Algorithm • Iteratively find Si • This naïve algorithm actually performs pretty well • Why? Submodularity • We get near-optimal solutions • Submodularity: diminishing returns

8. Diminishing Returns

9. Cost-Effective Lazy Forward-Selection (CELP) • Greedy algorithm • Lazy evaluations • Delaying computation until the result is required • A computational technique

10. Robust Sensing Optimization • Idea: Protect system against adversaries that know of our deployment of sensors • Goal: Maximize the worst-case detection performance • Approach • Unfortunately, this naïve extension can fail

11. Failure of Greedy Algorithm on Worst-Case Scenarios • I1, I2: two contamination events • S1, S2, S3: three possible sensor locations • S1: detect I1 immediately, but never I2 • S2: detect I2 immediately, but never I1 • S3: detect both I1 and I2, but only after a long time • We can only place two sensors • Greedy would pick S3 first and then either S1 or S2 • But we know the optimal solution should be S1 and S2 • Solution? Saturate algorithm

12. Saturate Algorithm • Idea: reduce the non-submodular worst-case objective to a submodular optimization problem • Transform non-submodular to submodular • Transformation • Guess optimal solution value C using binary search • Try to find A such that F(A) is no less than C

13. Performance of Saturate

14. From Water to the WebBlog Reading • Problem: Information cascading

15. Improvements • Number-of-posts (NP) model • Reading a big blog can be time-consuming, so they define the cost to be the number of posts • CELP tends to choose blogs with many posts • NP model tends to choose summarizer blogs • But stories appear in summarizer blogs a little late

16. Other Thoughts • What if we are looking for stories to read instead of blogs to read? • We can reverse our information management goal • Find posts instead of blogs • Ref. 10 • End of Paper

17. Distributed Dynamic Programmingfor Path Planning • Asynchronous Dynamic Programming • Learning Real-Time A*

18. Asynchronous Dynamic Programming • Propagate costs from target to start locations

19. Learning Real-Time A* (LRTA*)

20. LRTA*(n) • LRTA with n agents • Faster • Agents break ties differently • They can share the same h-value table

21. LRTA*(2)

22. Distributed Solutions to Markov Decision Problems • As previously mentioned in the Water to Web paper, MDPs can be difficult to scale to big problems • Solution: Exploit independence properties • We address the modularity of actions

23. Action Selection in multiagent MDPs

24. Implementation

25. Subtask Distribution • A global problem is broken down into subtasks • Subtasks are distributed among agents • Each agent has different capabilities Problem

26. Contract Net • Stages • Recognition • Announce • Bidding • Awarding & Expediting • Initial assignment: Not optimal • Anytime property • Improve assignment in negotiation process

27. Assignment problem • Problem definition • A set N of n agents • A set X of n objects • A set M ⊆ N × X of possible assignment pairs, and • A function v : M → R • Find optimal assignment X N M

28. Corresponding Linear Program • Linear program (LP) formulation Profit maximization Resource constraint Optimal solution • Any LP can be solved in polynomial time O(n3)

29. Competitive Equilibrium • Consider a price vector p = (p1, …, pn) • The utility from an assignment j to agent i isu(i, j) = v(I, j) - pj • A feasible assignment S and a price vector p are in competitive equilibrium when for every pairing (i, j) ∈ S it is the case that ∀k, u(i, j) ≥ u(i, k) Every agent will not change its selection S is a optimal solution

30. Naïve Auction Algorithm • Round-robin style • Bid increment is the difference between the utility to i of the best and second-best object The agent will not overbid

31. Problem in Naïve Auction • When more than one object offers maximal utility for an agent • Bid increment is zero

32. Terminating Auction Algorithm • Modify the bid increment ε-competitive equilibrium: u(i, j) + ε ≥ u(i, k)Agents may overbid some objects

33. Scheduling Problem • Problem definition • N is a set of n agents • X is a set of m discrete and consecutive time slots • q = (q1, . . . , qm) is a reserve price vector • v = (v1, . . . , vn), where vi is the valuation function of agent I • Find optimal allocation F

34. Corresponding Integer Program • Integer program (IP) formulation • IPs are not solvable polynomial time

35. Competitive Equilibrium – General Form • Definition • For all i ∈ N it is the case that Fi = argmaxT ⊆ X (vi(T) − ∑j|xj∈T pj) • For all j such that xj ∈ F∅ it is the case that pj = qj • For all j such that xj ∈ F∅ it is the case that pj ≥ qj • May not exist competitive equilibrium Has a competitive equilibrium solution ↕ The LP relaxation of the associated integer program has a integer solution.

36. Ascending Auction Algorithm • Center advertise an ask price • Bid increment is constant

37. Problem in Ascending Auction • If the increment is too large • May not converge to optimal solution

38. Social Laws and Conventions • Social law • A restriction on the given strategies of the agents • Induce a sub-game • Social convention • The sub-game consists of a single strategy for all agent • Other topics • Social goal negotiation • Social norm negotiation • ….