Visual Inspection Planning

1. Visual Inspection Planning By: Alexis Rivera Fai Lung Shih

2. Outline • What is visual inspection (quick review) • Inspection Planning Tool description • Handling visibility regions (AG -> EAG) • Union of viewing regions • How it’s applied to EAG • How it’s applied to SCG

3. yc yp xc zc f xp q p yw zw xw What is visual inspection planning?

4. Visual Inspection Problem (cont.) • Finding optimal camera pose: • For a given set of entities, S • minimize Obj (tx, ty, tz, Φ, θ, Ψ, S) • subject to: • g1j <= 0 (resolution), for j=1 to k • g2a <= 0 (focus) • g2b <= 0 • g3 <= (field of view) • g4i <= 0 (visibility) for i=1 to m

5. Resolution • For each entity j, there is a constraint g1j() • Projects a line of l millimeters to a line of w millimeters

6. Focus • Two constraints, g2a(), g2b() • Require closest and furthest entity vertices from the camera position to be within the far and near limits of the depth of field camera rf rc Far limit Near limit

7. Field Of View • One constraint: g3() • Bounding cone must be contained within the viewing cone Bounding cone Viewing cone

8. e1 e2 e4 e3 x y Visibility • Many equations: g4i() for i=1 to m • Plane equations that bound the visibility of the desired entities Example: To see entities e1, e2, e3, e4, the camera must satisfy equation y < 0

9. Outline • What is visual inspection (quick review) • Inspection Planning Tool description • Handling visibility regions (AG -> EAG) • Union of viewing regions • How it’s applied to EAG • How it’s applied to SCG

10. (Goal) Automated Plan Generation

11. Create AG Create EAG Select EOI Inspection Planning Tool

12. Objective Function Inspection Plan Strategy Create SCG Subnode Strategy Entity Tolerance Constraints Camera Parameter Generate Inspection Plan Inspection Planning Tool Cont.

13. Inspection Flowchart

14. Outline • What is visual inspection (quick review) • Inspection Planning Tool description • Handling visibility regions (AG -> EAG) • Union of viewing regions • How it’s applied to EAG • How it’s applied to SCG

15. Aspect Graph (Stewman) • Viewing volumes as the intersection of hyperplanes • Viewing volumes for all the entities in the object Viewing Volume: H1H2H3 Ex: • Drawbacks – may be too much information

16. Aspect Graph (Stewman) Cont. • Aspect graph of ell

17. Entity Aspect Graph (Yang) • Viewing regions for a subset of entities on the object • EAG simplifies the Aspect Graph • In the EAG, it has four elements (E, V, O, A). • E is a set of entity of interest. • V is a set of viewing domains. • O is a set of lists of observable entities. • A is a set of adjacent pairs of entity viewing domain

18. Entity Aspect Graph (Yang) Cont. • Operations on EAG: • Construction • Contraction ** • Expansion

19. Converting AG to EAG (idea) • Assume initial AG is an EAG that contains all entities • Apply contraction algorithm to AG to create desired EAG

20. Converting AG to EAG (Cont.) • Algorithm for contraction of aspect graph AG1(E1, V1, O1, A1) to construct EAG2(E2, V2, O2, A2) with E2 E1 • Input: AG1(E1, V1, O1, A1), E2 • Output: EAG2(E2, V2, O2, A2) • *Initialize EAG2 • Set V1 = V2, O1 = O2 , A1 = A2 • *Determine the visibility of the entities in E2 for each element in V2 • For every element of O2 • update the list of observable entities (Ovi2) to be the intersection of the original list of • observable entities (Ovi1) and E2. (Ovi2 = Ovi1  E2 ) • *Contract the adjacent viewing domains if their observable entities are the same • For every element of A2,(Vi2, Vj2) • If two adjacent viewing domains (Vi2, Vj2) of their observable entities are the same • (Ovi2 = Ovj2). • Then contracts Vi2 andVj2 by replace them with a new viewing domain, Vk2 = Vi2  Vj2. • Set the corresponding list of observable entities (Ovk2) with Ovi2. • Remove Ovi2 and Ovj2 from O2. • Remove the current arc of EAG2, (Vi2, Vj2), from A2. • For every element of A2, substitute Vi2 and Vj2 by Vk2.

21. N0 {e0e1e2e3} v0 N1 {e4e5e6e7} v1 N2 {e0e6e8e9} v2 N6 {e0e1e2e3 e6e8e9} v6 N7 {e0e4e5e6 e7e8e9} v7 Converting AG to EAG Example:  EOI = {e0e1e2e3} 1) Intersect EOIs with entities in AG

22. N0 {e0e1e2e3} v0 N1 {e4e5e6e7} v1 N2 {e0e6e8e9} v2 N6 {e0e1e2e3 e6e8e9} v6 N7 {e0e4e5e6 e7e8e9} v7 Converting AG to EAG Example: 2) merge nodes with same observable entities

23. N0 {e0e1e2e3} v0v6 N1 {e4e5e6e7} v1 N2 {e0e6e8e9} v2 N6 {e0e1e2e3 e6e8e9} v6 N7 {e0e4e5e6 e7e8e9} v7 Converting AG to EAG Example:(step 3)

24. N0 {e0e1e2e3} v0v6 N1 {e4e5e6e7} v1 N2 {e0e6e8e9} v2 v7 N6 {e0e1e2e3 e6e8e9} v6 N7 {e0e4e5e6 e7e8e9} v7 Converting AG to EAG Example:

25. N0 {e0e1e2e3} v0v6 N1 {e4e5e6e7} v1 N2 {e0e6e8e9} v2 v7 N6 {e0e1e2e3 e6e8e9} v6 N7 {e0e4e5e6 e7e8e9} v7 Converting AG to EAG Example:

26. Converting AG to EAG Cont. • Convert AG to EAG Example (Cube) • EOI = {e0,e1,e2,e3,e4,e5,e6,e7,e8,e9,e10,e11}

27. Converting AG to EAG Cont. • EOI = {e0,e1, e2, e3}

28. Merging viewing regions • Issue • How are two nodes merged? • How do you calculate the union of two viewing regions? • Yang didn’t specify a method for this

29. Outline • What is visual inspection (quick review) • Inspection Planning Tool description • Handling visibility regions (AG -> EAG) • Union of viewing regions • How it’s applied to EAG • How it’s applied to SCG

30. Union of viewing regions • Observations: • Viewing regions must form convex volumes in order to be formulated as linear constraints in the NLP formulation • It is possible for the valid viewing regions to form concave volumes

31. P3 P5 Example: concave viewing space How can we identify such cases?

32. Determining convexity of union of viewing regions • From: Bemporad A, Fukuda K, Torrisi F. D, Convexity recognition of the union of polyhedra, Computational Geometry Theory and Applications, 2001

33. Determining convexity of union of polyhedra • Definitions: • Convex H-Polyhedra • Intersection of a finite set of halfspaces of the Euclidean space Rd y y <= x x

34. Determining convexity of union of polyhedra Cont. • Definitions: • Valid inequality • Let P be a convex polyhedron in Rd. An inequality aTx<=b is called valid for P if it is satisfied by all points in P • Conversely, an inequality aTx <= b is called invalid for P if there exist a point on the other side of the inequality that is in P

35. Example valid/invalid inequalities H1(invalid for P) H1(valid for P) Q Q P P

36. Identifying valid/invalid inequalities • Recall, • An inequality aTx <= b is called invalid for P if there exist a point on the other side of the inequality that is in P • Feasibility Problem • Let (A,b) be the inequalities that define P • Let (c,t) be the inequality to be tested • Max F(x) • Subject to: Ax <= b, c > t • If there is a feasible point for this problem, the inequality is invalid

37. Determining convexity of union of polyhedra • Definitions: Envelope of two polyhedra P and Q • Let P and Q be (possibly unbounded) H-Polyhedra • Define where ( ) is the subsystem of ( ) obtained by removing all the inequalities not valid for the other polyhedron Q (P).

38. Example: env(P,Q) original Q P

39. Example: env(P,Q) Q P Invalid inequalities Env(P,Q)

40. Constructing the env(P,Q) Let Hp, Hq be the set of hyperplanes that define P, Q respectively Let C be the set of hyperplanes that define the env(P,Q) Let Ab be the set of hyperplanes of Q that are not valid for P Let Bb be the set of hyperplanes of P that are not valid for Q For each inequality c in Q if is_valid(Hp, c) then C = [C; c] else Ab = [Ab; c] For each inequality c in P if is_valid(Hq, c) then C = [C; c] else Bb = [Bb; c]

41. Determining convexity of union of polyhedra • Theorem: • Union(P,Q) is convex iff Union(P,Q)=env(P,Q) Env(P,Q) Union(P,Q) Q P

42. Example: is union(P,Q) convex? Any point in this region is outside a pair of invalid inequalities but inside the env(P,Q), therefore the union is concave Q P Invalid inequalities Env(P,Q)

43. Determining convexity of union of viewing polyhedra • Construct env(P,Q) Let be the set of removed constraints Let be the resulting envelope 2. For each pair do: E*= max (x) Subject To: , , If feasible return nonconvex Endfor Return env(P,Q) // union(P,Q) is convex

44. Optimizing the union of regions • Instead of using linear program to check the union of regions is convex or concave • By checking the number of hyperplanes removed • In AG, the viewing regions are disjoint and never overlap • Union of regions in order to be convex • regions are adjacent to each other (share a common hyperplane) • number of hyperplanes removed = 1

45. P1 P0 J P3 P6 P2 I P5 P4 Env(I,J) ={P0, P1 ,P2 ,P4 ,P5} hyperplanes removed ={P3 ,P6} Non adjacent number of hyperplanes removed = 2 concave Optimizing the union of regions Cont. I ={P0, P3 ,P4 ,P5 } J ={P1 ,P2 ,P6 }

46. P0 P1 P3 J P5 P2 I P4 Env(I,J) ={P2 ,P4 ,P5} hyperplanes removed ={P3 , P0 ,P1 } adjacent number of hyperplanes removed = 3 concave Optimizing the union of regions Cont. I ={P0, P3 ,P4 ,P5 } J ={P1 ,P2 ,P3 }

47. P0 P1 J P3 P5 P6 P2 I P4 Env(I,J) ={P2 ,P4 ,P5} hyperplanes removed ={P3 , P6 ,P0 ,P1 } Non adjacent number of hyperplanes removed = 4 concave Optimizing the union of regions Cont. I ={P0, P3 ,P4 ,P5 } J ={P1 ,P2 ,P6 }

48. Env(I,J) ={P0, P1 ,P2 ,P4 ,P5} hyperplanes removed ={P3} adjacent number of hyperplanes removed = 1 convex Optimizing the union of regions Cont. P1 P0 I ={P0, P3 ,P4 ,P5 } J ={P1 ,P2 ,P3 } P3 J P2 I P5 P4

49. Optimizing the union of regions Cont. • Comparison between this two approaches

50. Outline • What is visual inspection (quick review) • Inspection Planning Tool description • Handling visibility regions (AG -> EAG) • Union of viewing regions • How it’s applied to EAG • How it’s applied to SCG