Learn how to make your drawings come alive…

1.  COURSE: SKETCH RECOGNITION Analysis, implementation, and comparison of sketch recognition algorithms, including feature-based, vision-based, geometry-based, and timing-based recognition algorithms; examination of methods to combine results from various algorithms to improve recognition using AI techniques, such as graphical models. Learn how to make your drawings come alive…

2. Sezgin Shape Fitting:Circles and Ellipses • Least squares fit • Make a bounding box of stroke and form • Oval is formed from that bounding box. • Circle is easy: • Find out how far the circle is from the center = d, return (d-r)^2 • Ellipse more complicated

3. Sezgin Bezier Curve Fitting • Want to replace with a Bezier curve • http://www.math.ubc.ca/people/faculty/cass/gfx/bezier.html • Bezier Demo • 2 endpoints and 2 control points

4. Sezgin Bezier Curve Fitting • Control Point Approximation • c1 = k*t1 + v • c2 = k*t2 + u • Notation: • Endpoints: u = p1, v = p2 • T1= tangent vector – initial direction of stroke at point p1 (u) • T2 = tangent vector of p2 (v) – initial direction of stroke at point p2 • K = stroke length / 3 • 3 is common approximation

5. To test curve error: • Breaks into linear curves. • If error to great, splits curve in half (by stroke point – not distance)

6. Bezier curve equation • http://www.cl.cam.ac.uk/Teaching/2000/AGraphHCI/SMEG/node3.html • http://www.moshplant.com/direct-or/bezier/math.html • P0 (x0,y0) = p1, P1 = c1, P2 = c2, P3 = p2 • x(t) = axt3 + bxt2 + cxt + x0 • y(t) = ayt3 + byt2 + cyt + y0 • cx = 3 (x1 - x0)bx = 3 (x2 - x1) - cxax = x3 - x0 - cx - bx • cy = 3 (y1 - y0)by = 3 (y2 - y1) - cyay = y3 - y0 - cy - by

7. Matlab Curve Fitting • function [estimates, model] = fitcurvedemo(xdata, ydata) • % Call fminsearch with a random starting point. • start_point = rand(1, 2); • model = @expfun; • estimates = fminsearch(model, start_point); • % expfun accepts curve parameters as inputs, and outputs sse, • % the sum of squares error for A * exp(-lambda * xdata) - ydata, • % and the FittedCurve. FMINSEARCH only needs sse, but we want to • % plot the FittedCurve at the end. • function [sse, FittedCurve] = expfun(params) • A = params(1); • lambda = params(2); • FittedCurve = A .* exp(-lambda * xdata); • ErrorVector = FittedCurve - ydata; • sse = sum(ErrorVector .^ 2); • end • end

8. Yu Overview • Recursive corner finding method • Takes stroke and tries to classify it as • Line, Arc, Circle, Ellipse, Spiral • If error is too great, splits stroke at highest point of curvature and tries again.

9. Line Recognizer • Feature Area

10. Arc Recognizer • Feature Area: Note must have best fit arc

11. Circle Recognition: Fit Direction Graph to Line

12. Circle Fit • Use feature area to circle center

13. Ellipses • Handled similarly • Area should by symmetrical about the major and minor axes

14. Overtraced Circle • Break direction graph every 2 pi • If all circles have about same radius and center point – is circle

15. Spiral • If overtraced circles have ascending or descending radii, then is a spiral (they incorrectly call this a helix in the paper)

16. Developmental Framework for Critical Thinking Observation Interpretation Planning Judgment Foundation Knowledge Identify The Problem Explore Interpretations & Connections Prioritize Alternatives Envision Strategic Innovation Steps in Critical Thinking Confused Fact Finder Biased Jumper Perpetual Analyzer Pragmatic Performer Strategic Re-visioner Performance Patterns Distinguish relevant & irrelevant Information Read conflicting opinions Relate assumptions & biases Analyze pros & cons Prioritize issues and information Justify assumptions Articulate vision Reinterpret information Interventions Step 1 Step 2 Step 3 Step 4 Steps for Better Thinking Performance Patterns, http://www.wolcottlynch.com

17. Computer – Sketch Recognition Observation Interpretation Planning Judgment • Foundation Knowledge: Stroke, Rubine Method, Sezgin Method, Yu Method • Which primitive (or group) or gesture is this stroke? • Line, Arc, Ellipse, RGesture1, RGesture2 • Precedence, Error, Tolerance • What about combination of strokes? May change lower level interpretations… geometric context Foundation Knowledge Identify The Problem Explore Interpretations & Connections Prioritize Alternatives Envision Strategic Innovation

18. Recognizing a Line • Foundation Knowledge: Stroke, Sezgin Method, Yu Method, Geometry • How do we find *line* tolerance & error • Options: • Least-squares error from endpoints: • + Uses endpoints • - Endpoint tails not removed • - Error may be larger than true error • Least-square error with best fit line: • + Find best-fitting line • - Doesn’t necessarily use perceptually important start & end points • + Can remove non-perceptually important tails • feature area: • + In theory, can be compared to other shapes • - Confusing • - Value not apparent • ? Smaller range • ratio: euclidean length/stroke length: • + Easy to calculate • + Uses perceptually important start & end point • - Endpoint tails not removed • Doesn’t differentiate between one point being far away and several points being near • - Bigger range so harder to figure out a good threshold • Least-squares error using best fit, but then use endpoints • - Error not same as what is chosen • + Error is more representative of line • + Perceptually important endpoints

19. Recognizing an Arc • Foundation Knowledge: • Method to find sample arc as part of a circle: • Connect endpoints, find perpendicular bisector of that line • Find where that line intersects stroke • Make two lines connecting center stroke point and endpoints • Find perpendicular bisector of each line • Intersecting point is circle center • Find feature area • A curve of order 2 • Options: • Least-squares error from endpoints with a curve of order 2: • + Uses endpoints • + Easy to compute • - Not actually an arc • Feature area • + Uses real arc • + Faster ? • - Need the line of the arc, because takes the feature area • - Difficult – polygons could be above or below • Idea: Add threshold to radius – for comparing against line • Least squared error with arc itself • + Uses real arc • - Harder to compute • + Faster? • Compute distance of each point to the center – subtract from radius

20. Recognizing a Circle • Foundation Knowledge: • Method to find direction graph slope • Direction graph: Find direction of each point • Direction vs. time since start • Depends on time since start • Direction vs. point number • Depends on sampling rate • Direction vs. stroke length • More time computationally • Find slope • Fit a line to direction graph – use same least square method • Splitting: Spilt it when change in direction (every) 2pi • Circle center: • center of bounding box or • average of all points • Circle radius: • Bounding box / 2 • Average distance from center • Options: • Slope of the direction graph == 2pi/n • Doesn’t handle tails • Overtracing difficult because have to split • Direction graph is linear • Circle least squares • Circle feature area

21. Recognizing an Ellipse • Foundation Knowledge: • Method to find direction graph slope • Use endpoints • Find best fit line of direction graph • Major axis and minor axis not equal • To find major axis • Two points w/ greatest distance is the major axis • Perpendicular bisector is minor axis (where it intersects stroke) • Should points also intersect a calculated center point? • Fit a line to the ellipse • To find center point • Center of bounding box • Center of longest line • Center of mass • Area of Ellipse • PI * (length of major axis/2) * (length of minor axis/2) • Definition of Ellipse • Sum of the distance from focus 1 and focus 2 is constant • X^2 / a^2 + y^2 / b^2 = 1 • A = ½ major axis, b = ½ minor axis • Focal point is the point on the major axis that is distance ‘A’ from where minor axis intersects ellipse • Options: • Slope of the direction graph ~2pi/n • Ellipse least squares – need foci • Ellipse feature area • Small triangles to center vs actual ellipse area

22. Recognizing an Ellipse, Part 2 • Length of Major Axis • Fit a line • Longest distance • Length of Minor Axis • Rotate Ellipse to find height of BB • Average distance of stroke points that intersect minor axis from the major axis • Calculate from perimeter formula (p = strokelength = pi * sqrt(2*(s^2 + b^2) – (a-b)^2/2) • Calculate distance from every point to the major axis: • Minor axis = average distance * pi / 2 • Eigenvector method • Closest point to center

23. Helix Recognition • Find major, minor axis (rotated b b) • Find number of rotations (direction graph from circle) • Combine n helix components • Rotate, scale and translate • X = cos(t) + change in x at t • Y = sin(t) + change in y at t

25. Kim and Kim • 8 possible directions in pixel space • Resample Data of distance 5 pixels • Compute best fit lines of 10 pixels long. • Sharp angles smoothed out.

26. Curvature: Local Convextity • When computing curvature – if curvature sign is the same (moving in same curved direction) add that to total curvature. Fig. 5. Algorithm 1: Curvature estimation only with local convexity

27. Area of Examination • Area continues as long as it is convex. Fig. 6. The effect of the adaptively changing support with k=1. The actual support is denoted by the dotted rectangle. Without convexity criterion, however, actual support is set to the solid-line rectangle.

28. Local Monotonicity • Check if the current direction is < min direction – to find when it changes its sharpness Fig. 8. Algorithm 2: estimate curvature   at point pi.

30. Perceptual Recognition • Follows the line – smoothing points in order perceptual importance.

31. Multipass • Passes through the algorithm 5 times • Each time the strictness threshold gets looser. • Strictness = iteration number (1-5)

32. Examine each line segment • Each line segment to see if want to join with line on either side. • Line segments are sorted by length, so largest lines are processed first • Idea – want to join with a small

33. Threshold setting • Messythreshold = .05 * strictness • Used to merge lines together • Ratio = min(length1, length2) of line lengths

35. Research Projects • Sketch-based Interfaces and Modeling: SBIM’08: • Annecy, France. • Deadline in February, 2008 • Free trip for any accepted papers

36. Project Idea: Cusp/Corner Detection in Complex Shapes • Develop and implement a new corner detection algorithm that has improved recognition accuracy for complex fits. It must be fast and have greater accuracy than current methods.

37. Multi-stroke primitive recognition • Extend the low level recognizers to be multi-stroke recognizers. Detail how things change. – How are they combined? How do you know when strokes should be combined?

38. Compare Drawing Surfaces • Perform a user study on several drawing surfaces. See how the primitives are drawn differently on the different surfaces, and recognize them appropriately. Write up the differences and how one could shift

39. Project Suggestions • Build a finite state machine recognizer for the computability class to easily draw and hand in their diagrams. • Build a physics drawing program that attaches to a design simulator (we have interactive physics 2005) • Build a fashion drawing program. You draw clothes on a person, and it puts them one the person.

40. Project Ideas • Build a robot drawing and simulation program. You draw the robot and have a number of gestures to have it do different things • Gesture Tetris

41. Project Suggestions • Use both rubine and geometrical methods in recognition • Develop new ways for editing. • Build new low level recognizers

42. Projects! • 2 types: • Cool application • Sketch front end to your own research system • Fun application to go on smart board/vis contest • Gesture Tetris • TAMU gesture-based map/directory info • Computability/Physics/EE/MechEng simulator • New recognition algorithm • Significant change to old techniques to make a new application

43. Final Project Handin • Implementation… Build it… • In class Demonstration (5-10 minutes) • Previous work • Find at least 3 relevant papers (not read inc class) • Assign one for class to read, you lead short discussion • Test • Run your recognition system on data. • Find out what data you need (e.g., UML class diagrams) • Each student in class will supply others data • + Find 6 more people outside (to give 15 different people) • Paper • Introduction (why important) • Previous Work • Implementation • Results • Conclusion

44. Improvement Ideas • Instead of testing the error, we test the improvement, if it is great enough, we add the vertex