Learn how to make your drawings come alive…

1.  Lecture 3: SKETCH RECOGNITION Analysis, implementation, and comparison of sketch recognition algorithms, including feature-based, vision-based, geometry-based, and timing-based recognition algorithms; examination of methods to combine results from various algorithms to improve recognition using AI techniques, such as graphical models. Learn how to make your drawings come alive…

2. Class Overview • Paper discussion (lead by Joshua Peschel) • Discussion of previous homework and implementation issues • Lecture on classification details of Rubine • Introduction to Long • Homework

3. Read Paper Discussion: Rubine (lead by Joshua Peschel) • Thoughts?

4. Class Overview • Paper discussion (lead by Joshua Peschel) • Discussion of previous homework and implementation issues • Lecture on classification details of Rubine • Introduction to Long • Homework

5. Homework Implementation Discussion • Compare average feature values for each gesture type. • What did people get?

6. Implementation Issues (Repeat Slide from Lecture 2) • Tablets are faster now than they were • Rubine paper was based on slower data (mouse?) • But, for correct feel, pen needs to be fast • Issues you may have: • Duplicate location (2 consecutive points in same place) • Duplicate time (2 consecutive points at the same time) • Divide by zero (because of above problems) • Make sure your to convert to double before dividing in Java • Remove the second point not the first for duplicate points

7. Performing Recognition on these Features • What do you think about the features? • Can you foresee any recognition problems? • Which features will be more helpful?

8. Rubine Classification • Evaluate each gesture 0 <= c <= C. • Vc = value = goodness of fit for that gesture c. • Pick the largest Vc , and return gesture c

9. Class Overview • Paper discussion (lead by Joshua Peschel) • Discussion of previous homework and implementation issues • Lecture on classification details of Rubine • Introduction to Long • Homework

10. Rubine Classification • Wc0 = initial weight of gesture • Wci = weight for the I’th feature • Fi = ith feature value • Sum the features together

11. Compute gesture covariance matrix • How are the features of the shape related to each other? • Look at one example - look at two features – how much does each feature differ from the mean – take the average for all examples – that is one spot in the matrix • http://mathworld.wolfram.com/Covariance.html • Is there a dependency (umbrellas/raining)

12. Normalize • cov(X) or cov(X,Y) normalizes by N-1, if N>1, where N is the number of observations. This makes cov(X) the best unbiased estimate of the covariance matrix if the observations are from a normal distribution.For N=1, cov normalizes by N • They don’t normalize for ease of next step (so just sum, not average)

13. What does it mean to “Normalize” • Taking the average • But… we want to find the true variance. • Note that our sample mean is not exactly the true mean. • By definition, our data is closer to the sample mean than the true mean • Thus the numerator is too small • So we reduce the denominator to compensate

14. Common Covariance Matrix • How are the features related between all the examples? • Top = non normalize total covariance • Bottom = normalization factor = total number of examples – total number of gesture types (classes)

15. Weights • Wcj = weight for the jth feature of the cth shape • Sum for each feature • Common Covariance Matrix inverted* ij • Average feature value for the ith feature for the cth gesture

16. Initial Weight • Initial gesture weight = • Sum for each feature in class: • Feature weight * average feature value

17. Rubine Classification • Evaluate each gesture 0 <= c <= C. • Vc = value = goodness of fit for that gesture c. • Pick the largest Vc , and return gesture c

18. Rubine Classification • Wc0 = initial weight of gesture • Wci = weight for the I’th feature • Fi = ith feature value • Sum the features together

19. Rubine Adjustments: Eliminate Jiggle • Any input point within 3 pixels of the previous point is discarded

20. Rubine Adjustments:Rejection Technique 1 • If the top two gestures are near to each other, reject. • Vi > Vj for all j != i • Reject if less than .95

21. Rubine Adjustments:Rejection Technique 2 • Mahalanobis distance • Number of standard deviations g is from the mean of its chosen class i.

22. Implementation Issues • Java matrix class • JAMA for Java: http://math.nist.gov/javanumerics/jama/ • or unused for C++: http://www.techsoftpl.com/matrix/download.htm or unused for C++: http://www.programmersheaven.com/download/30784/download.aspx

23. Class Overview • Paper discussion (lead by Joshua Peschel) • Discussion of previous homework and implementation issues • Lecture on classification details of Rubine • Introduction to Long • Homework

24. Long features (from Rubine) • 1. Cosine of initial angle • 2. Sine of initial angle • 3. Size of bounding box • 4. Angle of bounding box • 5. Distance between first • and last points • 6. Cosine of angle between • first and last points • 7. Sine of angle between first • and last points • 8. Total length • 9. Total angle • 10. Total absolute angle • 11. Sharpness

25. Long Features: New • 12. Aspect [abs( – #4)] • 13. Curviness • 14. Total angle traversed / total • length • 15. Density metric 1 [#8 / #5] • 16. Density metric 2 [#8 / #3] • 17. Non-subjective • “openness” [#5 / #3] • 18. Area of bounding box • 19. Log(area) • 20. Total angle / total absolute • angle • 21. Log(total length) • 22. Log(aspect)

26. Class Overview • Paper discussion (lead by Joshua Peschel) • Discussion of previous homework and implementation issues • Lecture on classification details of Rubine • Introduction to Long • Homework

27. Homework • Read Long paper – who wants to present? • Implement Classifier • Apply Rubine features to your classifier (w/out adjustments – save for weekend). Computer accuracy on Math data. • Implement Long Features • Apply Long features to your classifier (compute accuracy on Math data) • Bring in Classification results