1 / 37

CS b553 : A lgorithms for Optimization and Learning

Explore variable elimination techniques for optimization and learning in general networks. Understand coherence, difficulty, intelligence, grade, SAT, letter, job, and happiness variables. Discover the process of eliminating variables efficiently, along with applying different ordering strategies and induced graph interpretation. Learn about factorization, heuristic methods, search algorithms, and key theorems for determining optimal orderings.

jonesmary
Download Presentation

CS b553 : A lgorithms for Optimization and Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS b553: Algorithms for Optimization and Learning Variable Elimination

  2. Last Time • Variable elimination on polytrees • Top down inference • Linear in size of network • Variable elimination in general • No guarantees… • NP hard in worst case… but when?

  3. Variable Elimination in General Networks Coherence Difficulty Intelligence Grade SAT Letter Job Happy

  4. Variable Elimination in General Networks Coherence Difficulty Intelligence Grade SAT Letter Job Happy

  5. Joint distribution • P(X) = P(C)P(D|C)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J) • Apply elimination ordering C,D,I,H,G,S,L

  6. Going through VE • P(X) = P(C)P(D|C)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J) • Apply elimination ordering C,D,I,H,G,S,L • 1(D)=SCP(C)P(D|C)

  7. Going through VE • SCP(X) = 1(D)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J) • Apply elimination ordering C,D,I,H,G,S,L • 1(D)=SCP(C)P(D|C)

  8. Going through VE • SCP(X) = 1(D)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J) • Apply elimination ordering C,D,I,H,G,S,L • 2(G,I)=SD1(D)P(G|I,D)

  9. Going through VE • SC,DP(X) = 2(G,I)P(I)P(S|I)P(L|G) P(J|L,S)P(H|G,J) • Apply elimination ordering C,D,I,H,G,S,L • 2(G,I)=SD1(D)P(G|I,D)

  10. Going through VE • SC,DP(X) = 2(G,I)P(I)P(S|I)P(L|G) P(J|L,S)P(H|G,J) • Apply elimination ordering C,D,I,H,G,S,L • 3(G,S)=SI2(G,I)P(I)P(S|I)

  11. Going through VE • SC,D,IP(X) = 3(G,S)P(L|G)P(J|L,S)P(H|G,J) • Apply elimination ordering C,D,I,H,G,S,L • 3(G,S)=SI2(G,I)P(I)P(S|I)

  12. Going through VE • SC,D,IP(X) = 3(G,S)P(L|G)P(J|L,S)P(H|G,J) • Apply elimination ordering C,D,I,H,G,S,L • 4(G,J)=SHP(H|G,J) What values does this factor store?

  13. Going through VE • SC,D,I,HP(X) = 3(G,S)P(L|G)P(J|L,S)4(G,J) • Apply elimination ordering C,D,I,H,G,S,L • 4(G,J)=SHP(H|G,J)

  14. Going through VE • SC,D,I,HP(X) = 3(G,S)P(L|G)P(J|L,S)4(G,J) • Apply elimination ordering C,D,I,H,G,S,L • 5(S,L,J)=SG3(G,S)P(L|G)4(G,J)

  15. Going through VE • SC,D,I,H,GP(X) = 5(S,L,J)P(J|L,S) • Apply elimination ordering C,D,I,H,G,S,L • 5(S,L,J)=SG3(G,S)P(L|G)4(G,J)

  16. Going through VE • SC,D,I,H,GP(X) = 5(S,L,J)P(J|L,S) • Apply elimination ordering C,D,I,H,G,S,L • 6(L,J)=SS 5(S,L,J)P(J|L,S)

  17. Going through VE • SC,D,I,H,G,SP(X) = 6(L,J) • Apply elimination ordering C,D,I,H,G,S,L • 6(L,J)=SS 5(S,L,J)

  18. Going through VE • SC,D,I,H,G,SP(X) = 6(L,J) • Apply elimination ordering C,D,I,H,G,S,L • 7(J)=SL 6(S,L)

  19. Going through VE • SC,D,I,H,G,S,LP(X) = 7(J) • Apply elimination ordering C,D,I,H,G,S,L • 7(J)=SL 6(L,J)

  20. Comparing Orderings • Consider G,I,S,L,H,C,D

  21. Understanding VE: From BNs to Undirected Graphs • Consider each factor as a variable i • Draw an edge between any variables appearing in the same factor

  22. Building the Undirected Graph P(C) Coherence P(I) P(D|C) Difficulty Intelligence P(S|I) P(G|I,D) Grade SAT P(L|G) Letter P(J|S,L) Job Happy P(H|G,J)

  23. Building the Undirected Graph P(C) Coherence P(I) P(D|C) Difficulty Intelligence P(S|I) P(G|I,D) Grade SAT P(L|G) Letter P(J|S,L) Job Happy P(H|G,J)

  24. Building the Undirected Graph Coherence Difficulty Intelligence Grade SAT Letter Job Happy

  25. Variable Elimination Coherence Difficulty Intelligence Grade SAT Letter Job Happy

  26. Variable Elimination Difficulty Intelligence Grade SAT Letter Job Happy

  27. Variable Elimination Difficulty Intelligence Grade SAT Letter Job Happy

  28. Variable Elimination Intelligence Grade SAT Letter Job Happy

  29. Variable Elimination Intelligence Grade SAT Letter Job Happy

  30. Variable Elimination New fill edge Grade SAT Letter Job Happy

  31. Variable Elimination Grade SAT Letter Job Happy

  32. Variable Elimination Grade SAT Letter Job

  33. Variable Elimination Grade SAT Letter Job

  34. Variable Elimination SAT Letter Job

  35. Induced Graph from a VE ordering Coherence Difficulty Intelligence Grade SAT Letter Job Happy

  36. Induced Graph from a VE ordering Coherence Difficulty Intelligence Grade SAT • Theorem: • The scope of every intermediate factor in VE is a clique in the induced graph • Every maximal clique in the induced graph is the scope of an intermediate factor Letter Job Happy

  37. Determining Optimal orderings • Again, NP hard! • Good heuristics in practice: • Min-neighbors, min-fill, etc • Search among elimination orderings while counting size of introduced factors • Greedy search often works well

More Related