E N D
1. Incorporating Context and User Feedback in Pen-Based Interfaces Martin Szummer & Phil Cowans
Microsoft ResearchCambridge, UK
2. The Task – Interpreting Ink ---
show selection and editing graphically---
show selection and editing graphically
3. One possible architecture [screenshot of Journal]
temporal grouping, spatial grouping
WARNING: loses momentum; takes time and details are not the main point
critique:
sequential processing (temporal followed by spatial)
hard decisions
how correct errors?
4. Making the most of imperfect recognizers The recognizer will make errors. To reduce the impact of the recognizer:
Make decisions:
soft
local (but based on global context)
late
incorporate user feedback A hypothetical system:
Microsoft tablet: show the design
Soft – keep uncertainty around; less of a decision
Local – When only a small part
Late – keep uncertainty from previous stages
results in fewer decisions
Avoid decisions where possible
This talk about the technology needed to support the above.A hypothetical system:
Microsoft tablet: show the design
Soft – keep uncertainty around; less of a decision
Local – When only a small part
Late – keep uncertainty from previous stages
results in fewer decisions
Avoid decisions where possible
This talk about the technology needed to support the above.
5. Proposal Probabilistic model for simultaneous use of:
spatial and temporal information
local and contextual information
classification and segmentation
user feedback
6. Overview Task overview
Description of the model
Labeling only model
Labeling and grouping model
Training and inference
Examples and results
Further work
7. Previous Work
Kumar, Hebert [2003]
Discriminative random fields.
Saund [2003]
Perceptual organization.
Mankoff [2001]
Uncertainty in user interfaces
8. Task Stages Divide the input data into ink fragments
Then…
Group the fragments into perceptually salient objects
And…
Label each object as either a container or connector
9. Example - Input
10. Example - Fragmentation
11. Example – Grouping / Labeling
12. Fragmentation Input data consists of ink strokes, described by sampled pen locations.
Strokes may span multiple objects.
Divide strokes into straight (within a tolerance) fragments.
Fragments are assumed to lie in a single object.
13. The Model Use a Conditional Random Field:
Undirected graphical model conditioned on observed data.
No need to model ink data (which we already know).
For now, just consider labeling.
14. Potentials
15. Nonlinearities Exponential:
Probit: (Gaussian CDF)
Epsilon is a ‘noise’ parameter.
17. Graph Construction Construct a graph with one vertex per ink fragment.
Add edges between nearby fragments.
Triangulate graph.
Calculate interaction potentials for all edges.
18. Graph Construction II Speed is dependent on the tree-width of the graph.
By constraining the tree-width it should be possible to limit the complexity of the algorithm.
19. Features Spatial features
Angles, lengths, distances…
Temporal features
Stroke ordering, same stroke…
Template features
T-Junction identification…
61 observation, 37 interaction features in total
20. Priors Three feature categories:
Binary
Counting
Continuous
Continuous features are replaced by histograms
Use a correlated Gaussian prior on continuous features
21. Example Weights
22. Labeling And Grouping Use a slight modification of the model
Now three interaction categories:
Same object (implies same label)
Different object, same label
Different object, different label
23. Advantages Of This Approach Joint labeling
Makes use of contextual information
Simultaneous labeling and grouping
Processes can reinforce each other
Handles groups directly
Doesn’t define groups via labeling
Results in a ‘cleaner’ model
24. Training Train by finding MAP weights on example data.
Training data set was 40 diagrams, from 17 subjects with a total of 2157 fragments.
Maximise using quasi-Newton gradient ascent (BFGS in MATLAB).
Evaluate function and gradient using message passing.
25. Message Passing Just like standard sum-product, but we are summing over groupings and labelings together rather than just labelings.
Fortunately efficient computation via message passing is still possible.
Messages are now a list of possible groupings and labelings of the separator variables.
26. Message Passing II Message update rule:
Marginals:
27. Message Passing III
28. Inference Unseen data is processed by finding the most probable grouping and labeling.
Use the max-product variant of the message passing algorithm.
Messages are now the ‘best you can do’ given the separator configuration.
29. Example - Raw Ink
30. Example – Labeling Only
31. Example – Labeling And Grouping
32. Example 2 - Raw Ink
33. Example 2 – Labeling Only
34. Example 2 – Labeling And Grouping
35. MATLAB Demo
36. Grouping Error Measured using the metric:
Not really sure how this relates to ‘perceptual’ error.
There’s potentially a better way of doing this via mutual information.
37. Results: Accuracy ----
Grouping error should have a more intuitive measure----
Grouping error should have a more intuitive measure
38. Results: Speed Median evaluation time for labeling only:
0.35 seconds.
Median evaluation time for labeling and grouping:
13 seconds.
(And that’s not very optimised, using MATLAB!)
39. Conclusions Exact inference is possible in reasonable time.
This approach is capable of providing high-quality results.
Joint labeling improves performance.
Labeling and grouping simultaneously helps too.
40. Further Work Multiclass model:
Dashed lines, text, arrows, …
Relationships:
Which containers are connected together.
More ‘global’ features:
Closed paths, convex containers, …
Possibly via re-ordering of N-best lists.
More data, better features, …
41. Observation Weights
42. Interaction Weights
43. File: 33.xmlFile: 33.xml
45. 30.Xml
Bad: 51 is mislabelled and misgrouped too, and not corrected on next slide
30.Xml
Bad: 51 is mislabelled and misgrouped too, and not corrected on next slide
47. Publication example 6.Xml
From 6_uncorrected.fig (left), and 6_corrected2 (right)
ink_demo('X:\Data\OrgChartRecognition\renamed-frag\6.xml', {}, {[41, 43], [50 55]})6.Xml
From 6_uncorrected.fig (left), and 6_corrected2 (right)
ink_demo('X:\Data\OrgChartRecognition\renamed-frag\6.xml', {}, {[41, 43], [50 55]})