Algorithms for POMDP

1. Algorithms for POMDP Presented by Alp Sardağ

2. Monahan Enumeration Phase • Generate all vectors: Number of gen. Vectors = |A|M|| where M vectors of previous state

3. Monahan Reduction Phase • All vectors can be kept: • Each time maximize over all vectors. • Lot of excess baggage • The number of vectors in next step will be even large. • LP used to trim away useless vectors

4. Monahan Reduction Phase • For a vector to be useful, there must be at least one belief point it gives larger value than others:

5. Monahan Algorithm

6. Monahan’s LP Complication Formulate LP and check for :

7. Eagle’s Variant of Monahan • The optimization occurs in enumaration phase. • If, in the enumaration process, a vector’s components are completely dominated by another vector’s component, discard it. • Generate ji(t) and following condition holds: • Discard ji(t). • Can be applied to check new vector dominates any vector previously enumarated.

8. Sondik’s One-Pass Algorithm • Find theproper set of belief states to plug into the below formula to get all necessary vectors: • The algorithm is guaranteed to visit finite number of regions. • The union of these regions is the entire belief space.

9. Sondik’s One-Pass Algorithm • Simplified version of Sondik’s algorithm:

10. Sondik’s One-Pass Algorithm • How to define a region around this belief state where that vector is guaranteed to be true linear portion of the value function? • Construct a series of constraints when satisfied, region is found. • Then go step (5)

11. Sondik’s One-Pass Algorithm • The condition *(t), generated at , larger for all other a(t), as  varies: • Variations in  can cause changes in a(t). • Need a new constraint to ensure components of a(t) stay the same.

12. Sondik’s One-Pass Algorithm • What affects *(t) and a(t)? • To ensure that every part of the function does not change, these constraint exists for every combination of a and 

13. Sondik’s One-Pass Algorithm • Constraints restrict belief states to lie on the belief state space simplex:

14. Sondik’s One-Pass Algorithm • A constraint consists of a region with all the points on one side of the line:

15. Sondik’s One-Pass Algorithm • The LP constraints at step (4):

16. Sondik’s One-Pass Algorithm • In step (5), find belief states guaranteed not to be in region defined in step (4). • With the new point proceed exactly as step (4). • The algorithm goes until a complete partition of the belief space found.

17. Sondik’s One-Pass Algorithm • To find points in the neighboring regions, points lying on the edge of the region defined by the constraints is used:

18. Sondik’s One-Pass Algorithm • Which constraints are binding: • For each constraint, change its inequality into an equality, • Solve this LP. • If the LP has solution, it is a binding constraint, a non-binding constraint can not pass through the region defined by all other constraints.

19. Cheng’s Relaxed Region • Same as Sondik’s One Pass algorithm except each region specified with fewer constraints. • Defines regions that will typically be larger than the actual vectors’ regions.

20. Cheng’s Relaxed Region • Set of constraints for the relaxed regions of Cheng:

21. Cheng’s Relaxed Region • Corners found with interior algorithm:

22. Cheng’s Linear Support • The algorithm defines an approximate value function over the entire belief space. • Refine this approximation until it reaches the optimal value function.

23. Cheng’s Linear Support • Difference between two algorithms:

24. Cheng’s Linear Support • Initiliaze a search list with extreme points on the belief simplex(e.g. [1,0,0...],[0,1,0,0...]), and an empty set of vectors. • For each of these points the true (t) vector calculated, and added to the set of vectors.

25. Cheng’s Linear Support • Since both the true and the approximation are PWLC, the largest difference must occur at a corner point. • Cheng then finds all the corner points of the regionsinduced by the approximation. • Disregard the corner points seen before and add those not seen before to search list. • Pick a point from the search list, generate the vector. If it is different all the other approximation, add it to the approximation set. • Repeat whole procedure with the new approximation

26. Cheng’s Linear Support

27. Cheng’s Linear Support