Fingerspelling Recognition with Support Vector Machines and Hidden Conditional Random Fields

1. Fingerspelling Recognition with Support Vector Machines and Hidden Conditional Random Fields - A comparison with Neural Networks and Hidden Markov Models - César R. de Souza, Ednaldo B. Pizzolato and Mauro dos Santos Anjo Universidade Federal de São Carlos (Federal University of São Carlos) IBERAMIA 2012 Cartagena de Índias, Colombia2012

2. Introduction Context, Motivation, Objectives and the Organization of this Presentation

3. Multidisciplinary • Computing and Linguistics • Ethnologue lists about 130 sign languages existent in the world • (LEWIS, 2009) ContextMotivationObjectivesAgenda

4. Two fronts • Social • Aim to improve quality of life for the deaf and increase the social inclusion • Scientific • Investigation of the distinct interaction methods, computational models and their respective challenges ContextMotivationObjectivesAgenda

5. This paper • Investigate the behavior and applicability of SVMs and HCRFs in the recognition of specific signs from the Brazilian Sign Language • Long term • Walk towards the creation of a full-fledged recognition system for LIBRAS • This work represents a small but important step in achieving this goal ContextMotivationObjectivesAgenda

6. ContextMotivationObjectivesAgenda Introduction - Brazilian Sign Language Libras Literature Review and Tools Methods Support Vector Machines Conditional Random Fields Experiments Results Conclusion

7. The BrazilianSignLanguage (LIBRAS) Structuresandthe manual alphabet

8. Natural language • Not mimics • Not universal • It is not only “a problem of the deaf or a language pathology” • (QUADROS & KARNOPP, 2004) LIBRASDifficultiesGrammar

9. Highly context-sensitive • Same sign may have distinct meanings • Interpretation is hardeven for humans LIBRASDifficultiesGrammar

10. Fingerspelling is onlypart of the Grammar • Needed when explicitly spellingthe name of a person or a location • Subset of the full-language recognition problem LIBRASDifficultiesGrammar

12. Literature • Layerarchitectures are common • Staticgestures x Dynamicgestures • Oneofthebestworkson LIBRAS handlesonlythemovementaspectofthelanguage(Dias et al.)

13. Fewstudies explore SVMs • Butmany use Neural Networks • No studiesonHCRFsand LIBRAS

14. Example • Recognitionof a fingerspelled wordusing a two-layeredarchitecture Pato HMM Sequenceclassifier P P P A A A T T T O O Staticgestureclassifier ANN

15. YANG, SCLAROFF e LEE, 2009 • Multiplelayers, SVMs • Elmezain, 2011 • HCRF, in-airdrawingrecognition

16. Models and Tools Overview of the chosen techniquesand reasons for their choice

17. Static Gesture Recognition Neural Networks and Support Vector Machines for the detection of static signs

18. Find such that… c b a

19. Biologically inspired • McCulloch & Pitts, Rosenblatt, Rumelhart Neural NetworksSupport Vector MachinesMaximum MarginMultiple Classes

20. Perceptron • Hyperplanedecision • Linearlyseparableproblems • Learning is a ill-posedproblem • Multiple local minima, ill-conditioning • Layerarchitecture • Universal approximator

21. Strong theoretical basis • Statistical Learning Theory • Structural Risk Minimization (SRM) Neural NetworksSupport Vector MachinesMaximum MarginMultiple Classes

22. Riskminimizationthroughmarginmaximization • Capacitycontrolthroughmargincontrol • Sparsesolutionsconsideringonly a fewsupportvectors Neural NetworksSupport Vector MachinesMaximum MarginMultiple Classes • Large-marginclassifiers

23. Problem: binary-only classifier • How to generalize to multiple classes? Neural NetworksSupport Vector MachinesMaximum MarginMultiple Classes

24. Problem: binary-only classifier • How to generalize to multiple classes? • Classical approaches • One-against-all • One-against-one • Discriminant functions • Drawbacks • Only works when equiprobable • Evaluation of c(c-1)/2 machines • Non-guaranteed optimum results With 27 static gestures, this would result in 351 SVM evaluations each time a new classification is required!

25. Problem: binary-only classifier • How to generalize to multiple classes? • Directed Acyclic Graphs • Generalization of Decision Trees,allowing for non-directed cycles • Require at maximum c-1 evaluations Neural NetworksSupport Vector MachinesMaximum MarginMultiple Classes So, for 27 static gestures, only 26 SVM evaluations are required. Only 7.4% of the original effort 

26. Elimination proccess • One class eliminated at a time Candidates A x D A B C D A B C D A lost D lost B x D A x C B C D A B C B lost C lost D lost A lost C x D B x C A x B C D B C A B B lost C lost C lost D lost A lost B lost D C B A

27. However, no matter the model • We’ll have (extreme) noise due pose transitions • How can we cope with that?

28. Dynamic Gesture Recognition Hidden Markov Models, Conditional Random Field and Hidden Conditional Random Fields for dynamic gesture recognition.

29. Find such that givenextremelynoisysequencesoflabels, estimatethewordbeingsigned. hello blyrei hi hil bye hmeylrlwo

30. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

31. Hidden Markov Models • Joint probability model of a observation sequence and its relationship with time Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) A B

32. Hidden Markov Models • Marginalizing over y, we achieve the observation sequence likelihood • Which can be used for classificationusing either the ML or MAP criteria Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

33. Word 1 Word 2 ... Word n Onemodel for eachword

34. Hidden Markov Models have found great applications in speech recognition • However, a fundamental paradigmshift recently occurred in this field Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

35. Probability distributions governing speech signals could not be modeled accurately, turning “Bayes decision theory inapplicable under those circumstances” • (Juang & Rabiner, 2005) Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

36. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

37. Conditional Random Fields • Generalization of the Markov models Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) • Discriminative Models • Model without incorporating • Designates a family of MRFs • Each new observation originates a new MRF

38. Generative Sequence Graphs Naïve Bayes HMM Directional models Discriminative Logistic Regression Linear-chain CRF CRF Infographbasedonthe tutorial by Sutton, C., McCallum, A., 2007

39. Conditional Random Fields • Generalization of the Markov models Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) PotentialFunctions Parameter vector whichcanbeoptimizedusinggradientmethods Potential Cliques Characteristicfunction vector Partitionfunction

40. Conditional Random Fields • Generalization of the Markov models Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) • How do we initialize those models? • Reaching HCRFs from a HMM

42. a11 a12 a13 b11 b12 a21 a22 a23 b21 b22 a31 a32 a33 b31 b32

43. i=1j=1 i=1j=2 i=1j=3 i=2j=1 i=2j=2 i=2j=3 i=3j=1 i=3j=2 i=3j=1 i=1o=1 i=1o=2 i=2o=1 i=2o=2 i=3o=1 i=3o=2 a11 a12 a13 a21 a22 a23 a31 a32 a33 b11 b12 b21 b22 b31 b32

46. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) • Drawback • Assumes both and are known

47. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

48. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

49. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) • Hidden Conditional Random Fields • Generalization of the hidden Markov classifiers Parameter space

50. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) • Hidden Conditional Random Fields • Generalization of the hidden Markov classifiers • Sequence classification • Model without explicitly modeling • Do not require to be known • The sequence of states is now hidden