A New Parsing Language for GUI and Visually Structured Documents

A New Parsing Language for GUI and Visually Structured Documents David Lehavi HP Labs Israel

Why bother ? universal interface for all graphical applications Standard Approach: use the DOM And if there is no dom, or a hybrid environment ? New GUI for legacy apps (additional functionality, hiding sensitive data). Software testing (record and replay). Accessibility (speech activated apps). Mobile Devices Web 2.0 (Flash, fragmented toolkit environment) Hybrid environments • DOM inspector • Object type • Set/Get properties

Visual Languages What images do we need to understand ? A two dimensional pixel word: bit map A two dimensional picture word (constructed from graphical tokens) Formal presentation: A•→(B•↓C) We only parse objects which are “cut by lines”. Less restrictive than it seems at first: we may generalize and parse objects which are “cut by curves” (overcome the X) B A C

Intermezzo: using language constructs Following Ken Thompsons work on regular expressions Visual lexer tokens characters requests Universal machine Compiler Language definition bytecode

Challenges in GUI parsing Finding useful language constructs Expressability • Regular languages are too weak to describe recursive structures. Decidability & performance • Context free languages are too strong – they are undecidable. Ease of maintenance: Many GUI’s, and constantly changing. Robust to “lexing noise”: Input may originate from screenshot analysis.

Radio-button-set example A Naïve representation: (Radio•→Text)*↓ Problems: alignment, distances. RTitled_E<Object X> = [ X C 0..50 Text ] RTitled_M<Object X> = [ X C 0..50 Text L L 0..50 X C 0..50 Text] RBS := V{RTitled_M<Radio>*RTitled_E<Radio>}

Using a visibly pushdown meta language EBNF for regular expression VPL = regex + function calls and definitions Adding distances and alignments <RE>=<union>|<simple> <union>=<RE>"|"<simple> <simple>=<concat>|<basic> <concat>=<simple><basic> <basic>=<star>|<elementary> <star>=<elementary>"*" <elementary>= <group>|<token> <group>="("<RE> ")“ <group>=[V>]"("<RE> ")" <name>= standard <call>=<name>"<"<values>">" <values> = comma separated <value> <value>= <call>|<token>|<name> <rule>= <name>"<"<params>">=" (<group>|<col>|<call>) <params>= comma separated <param> <param>="Object" <name> <elementary>=<group>|<call>|<col>|<token> <range>= <int>".."<int> <west>= [TBC]<range> (<call>|<token>|<name>) <row>= (<call>|<token>|<name>) <west>? <south>= [RLC][RLC]?<range><row> <col>= "["<row><south>?"]"

Language & compilation - example RTitled_M<Radio > = [ Radio C 0..50 Text L L 0..50 Radio C 0..50 Text] RTitled_E<Radio > = [ Radio C 0..50 Text ] RBS := V{RTitled_M<Radio>*RTitled_E<Radio>} Each node is a function Concat Kleene-* Col Row Text Radio

Running the VPL code - example Concat Kleene-* Col Universal VPL machine RBS Line Text Radio

Global Robustness to local ambiguities • Visual lexer returns atoms. • Lexer assigns likelihood to any pair (atom, bounding box). • We use conditional likelihood to avoid consistent errors. • A “compound object” has heuristic “likelihood” • VPL graph vertices are no longer functions, but co-routines (user space threads). • sit on (heuristic based) priority queue, and paused when their priority is low. • can be forked when they get multiple return values. 50 % LO, 50 % scroller

Questions ?

A New Parsing Language for GUI and Visually Structured Documents