1 / 46

Incorporating Context and User Feedback in Pen-Based Interfaces

lorant
Download Presentation

Incorporating Context and User Feedback in Pen-Based Interfaces

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


    1. Incorporating Context and User Feedback in Pen-Based Interfaces Martin Szummer & Phil Cowans Microsoft Research Cambridge, UK

    2. The Task – Interpreting Ink --- show selection and editing graphically--- show selection and editing graphically

    3. One possible architecture [screenshot of Journal] temporal grouping, spatial grouping WARNING: loses momentum; takes time and details are not the main point critique: sequential processing (temporal followed by spatial) hard decisions how correct errors?

    4. Making the most of imperfect recognizers The recognizer will make errors. To reduce the impact of the recognizer: Make decisions: soft local (but based on global context) late incorporate user feedback A hypothetical system: Microsoft tablet: show the design Soft – keep uncertainty around; less of a decision Local – When only a small part Late – keep uncertainty from previous stages results in fewer decisions Avoid decisions where possible This talk about the technology needed to support the above.A hypothetical system: Microsoft tablet: show the design Soft – keep uncertainty around; less of a decision Local – When only a small part Late – keep uncertainty from previous stages results in fewer decisions Avoid decisions where possible This talk about the technology needed to support the above.

    5. Proposal Probabilistic model for simultaneous use of: spatial and temporal information local and contextual information classification and segmentation user feedback

    6. Overview Task overview Description of the model Labeling only model Labeling and grouping model Training and inference Examples and results Further work

    7. Previous Work Kumar, Hebert [2003] Discriminative random fields. Saund [2003] Perceptual organization. Mankoff [2001] Uncertainty in user interfaces

    8. Task Stages Divide the input data into ink fragments Then… Group the fragments into perceptually salient objects And… Label each object as either a container or connector

    9. Example - Input

    10. Example - Fragmentation

    11. Example – Grouping / Labeling

    12. Fragmentation Input data consists of ink strokes, described by sampled pen locations. Strokes may span multiple objects. Divide strokes into straight (within a tolerance) fragments. Fragments are assumed to lie in a single object.

    13. The Model Use a Conditional Random Field: Undirected graphical model conditioned on observed data. No need to model ink data (which we already know). For now, just consider labeling.

    14. Potentials

    15. Nonlinearities Exponential: Probit: (Gaussian CDF) Epsilon is a ‘noise’ parameter.

    17. Graph Construction Construct a graph with one vertex per ink fragment. Add edges between nearby fragments. Triangulate graph. Calculate interaction potentials for all edges.

    18. Graph Construction II Speed is dependent on the tree-width of the graph. By constraining the tree-width it should be possible to limit the complexity of the algorithm.

    19. Features Spatial features Angles, lengths, distances… Temporal features Stroke ordering, same stroke… Template features T-Junction identification… 61 observation, 37 interaction features in total

    20. Priors Three feature categories: Binary Counting Continuous Continuous features are replaced by histograms Use a correlated Gaussian prior on continuous features

    21. Example Weights

    22. Labeling And Grouping Use a slight modification of the model Now three interaction categories: Same object (implies same label) Different object, same label Different object, different label

    23. Advantages Of This Approach Joint labeling Makes use of contextual information Simultaneous labeling and grouping Processes can reinforce each other Handles groups directly Doesn’t define groups via labeling Results in a ‘cleaner’ model

    24. Training Train by finding MAP weights on example data. Training data set was 40 diagrams, from 17 subjects with a total of 2157 fragments. Maximise using quasi-Newton gradient ascent (BFGS in MATLAB). Evaluate function and gradient using message passing.

    25. Message Passing Just like standard sum-product, but we are summing over groupings and labelings together rather than just labelings. Fortunately efficient computation via message passing is still possible. Messages are now a list of possible groupings and labelings of the separator variables.

    26. Message Passing II Message update rule: Marginals:

    27. Message Passing III

    28. Inference Unseen data is processed by finding the most probable grouping and labeling. Use the max-product variant of the message passing algorithm. Messages are now the ‘best you can do’ given the separator configuration.

    29. Example - Raw Ink

    30. Example – Labeling Only

    31. Example – Labeling And Grouping

    32. Example 2 - Raw Ink

    33. Example 2 – Labeling Only

    34. Example 2 – Labeling And Grouping

    35. MATLAB Demo

    36. Grouping Error Measured using the metric: Not really sure how this relates to ‘perceptual’ error. There’s potentially a better way of doing this via mutual information.

    37. Results: Accuracy ---- Grouping error should have a more intuitive measure---- Grouping error should have a more intuitive measure

    38. Results: Speed Median evaluation time for labeling only: 0.35 seconds. Median evaluation time for labeling and grouping: 13 seconds. (And that’s not very optimised, using MATLAB!)

    39. Conclusions Exact inference is possible in reasonable time. This approach is capable of providing high-quality results. Joint labeling improves performance. Labeling and grouping simultaneously helps too.

    40. Further Work Multiclass model: Dashed lines, text, arrows, … Relationships: Which containers are connected together. More ‘global’ features: Closed paths, convex containers, … Possibly via re-ordering of N-best lists. More data, better features, …

    41. Observation Weights

    42. Interaction Weights

    43. File: 33.xmlFile: 33.xml

    45. 30.Xml Bad: 51 is mislabelled and misgrouped too, and not corrected on next slide 30.Xml Bad: 51 is mislabelled and misgrouped too, and not corrected on next slide

    47. Publication example 6.Xml From 6_uncorrected.fig (left), and 6_corrected2 (right) ink_demo('X:\Data\OrgChartRecognition\renamed-frag\6.xml', {}, {[41, 43], [50 55]})6.Xml From 6_uncorrected.fig (left), and 6_corrected2 (right) ink_demo('X:\Data\OrgChartRecognition\renamed-frag\6.xml', {}, {[41, 43], [50 55]})

More Related