210 likes | 294 Views
Bayesian Networks for Sketch Understanding. Christine Alvarado MIT Student Oxygen Workshop 12 September 2003. Sketching in Design. Mechanical Engineering. Software. A Challenge In Sketch Understanding. Noisy Input.
E N D
Bayesian Networks for Sketch Understanding Christine AlvaradoMIT Student Oxygen Workshop12 September 2003
Sketching in Design Mechanical Engineering Software
A Challenge In Sketch Understanding • Noisy Input There is no one threshold for shapes or constraintsInterpretation depends on context
Naïve Approach • Why not just try all possibilities?
Naïve Approach • Why not just try all possibilities? Arrow?
Naïve Approach • Why not just try all possibilities? Arrow?
Naïve Approach • Why not just try all possibilities? Arrow?
Naïve Approach • Why not just try all possibilities? Arrow?
Naïve Approach • Why not just try all possibilities? Arrow?
Naïve Approach • Why not just try all possibilities? interpretations Must consider n = number of strokes/segmentsS = set of shapeski = subcomponents in shape Si
Naïve Approach • Why not just try all possibilities? interpretations Must consider n = number of strokes/segmentsS = set of shapeski = subcomponents in shape Si And this only considers shapes independently
Previous Approaches Use Rigid Segmentation • Single stroke shapes • Palm Pilot Graffiti • Long et. al. [1999] • Explicit Segmentation • Quickset: Cohen et. al. [2001] • Pause between strokes
Recognition Using Partial Interpretations • Recognition: • Build partial interpretations (PIs) as the user draws based on easily recognizable low-level shapes • Prune unlikely PIs and use likely PIs to find misrecognized low-level shapes • Evaluating PIs Graphical Models: • Missing data = unobserved nodes • Interpretation influenced by top-down and bottom-up information
BN fragments [similar to PRMs, Getoor et. al. 1999] (Define Arrow (Components (Line shaft) (Line head1) (Line head2)) (Constraints (connects shaft.p1 head1.p1) (connects shaft.p1 head2.p1) (= head1.length head2.length) (< head1.length shaft.length) (< (angle head1 shaft) 90) (< (angle shaft head2) 90) (> (angle head1 shaft) 0) (> (angle shaft head2) 0))) L1:L2:L3:C1:C2:C3:C4:C5:C6:C7:C8: Arrow … L1 L2 L3 C1 C2 C3 C8 [Hammond and Davis, 2003] Instantiated and linked together as recognition proceeds
Observation node added when primitive linked to stroke P(Obs|Prim) determined through data collection Primitive shapes/Constraints L1 Obs
Example 0.59 Force-pushes-body Touches F B 0.95 Body(B) Force(F) 0.5 Quad 0.95 Arrow Remaining Arrow Constraints 0.97 0.99 0.99 Connects l1 l2 Line Line Line(l1) Line(l2) Line(l3) Sq. error (Stroke a) Sq. error (Stroke b) Observation
Example 0.59 Force-pushes-body Touches F B 0.95 Body(B) Force(F) 0.5 Quad 0.95 Arrow Remaining Arrow Constraints 0.97 0.99 0.99 Connects l1 l2 Line Line Line(l1) Line(l2) Line(l3) Sq. error (Stroke a) Sq. error (Stroke b) Sq. error (Stroke c) Observation
Example 0.61 Force-pushes-body Touches F B 1 Body(B) Force(F) 0.47 Quad 1 Arrow Remaining Arrow Constraints 0.97 1 1 Connects l1 l2 0.95 Line Line Line(l1) Line(l2) Line(l3) Sq. error (Stroke a) Sq. error (Stroke b) Sq. error (Stroke c) Observation
Example 0.95 Force-pushes-body 0.99 0.97 Touches F B 1 Body(B) Force(F) 0.47 Quad Observation 1 Arrow Remaining Arrow Constraints 1 0.99 1 1 Connects l1 l2 Ellipse 1 Line Line Line(l1) Line(l2) Line(l3) Sq. error (Stroke d) Sq. error (Stroke a) Sq. error (Stroke b) Sq. error (Stroke c) Observation
Current/Future Work • Expand domain/include other domains • Gather sketches from users
Conclusion • Graphical models evaluation Partial Interpretations Context-guided search • More drawing freedom + More robust recognition = More natural interfaces (i.e. The goal of OXYGEN)