Syntax Discussion

1. Syntax Discussion

2. The “Stack” • Components • original ANL components: PM, NSM, SD, EM • SSSDK,SSSLIB-using versions of Sched, QM, others • Custom utilities • Throwaway scripts SSSRMAP comps • SSS Software Development Kit (SSSDK) • Component building library for Python • Built-in and custom objects • Examples: Server, … • Choice of XML syntax style • Restriction Syntax • SSSRMAP Syntax • Something better Craig • XML Parsers and Validators • Expat, Xerces, Elementtree, etc. • Communication Library (SSSLib) • Interaction with Service Directory • Communication • Wire Protocols • Basic, Challenge, SSSRMAP, others

3. XML Syntax Style Discussion Outline • The current situation • Desirable features of XML syntax style • Presentation of Restriction syntax • BNF • Examples • Presentation of SSSRMAP syntax • Comparison by feature • More examples • Capturing the best features of both

4. The Current Situation • Initially, we agreed on XML (because of multiple 3rd-party parsers and the usefulness of schemas for specification and for validation) and sockets (because of universality) as the basis of a common communication mechanism. • Did not standardize on wire protocol; instead allowed multiple ones, plus a service directory to keep track of which component used which. We provided the communication library to hide the existence of multiple protocols from component code.

5. The Current Situation (cont.) • Did not standardize on “XML style”, so multiple components were developed, with major investment, using quite different styles • Not a killer. Queue manager uses RS for process manager and SSSRMAP for scheduler, SSSlib for registration • But recently, a desire has been expressed for a uniform style across components. • This could mean major retrofit for some components, since XML style can be entwined in component logic. • Hence this discussion of RS and SSSRMAP, and maybe something better. • The goal here is NOT to suggest that any one of these be mandated for all components, just to understand them.

6. Areas of Agreement • Common goal: command language in XML • Object-orientation a plus • At abstract level, needs to apply a function to a collection of objects • Individual components will deal with different objects and different functions • Steps: • Identification of sets of objects • Or construction • Specification of function (with arguments) • Construction of response message • There are different ways of doing this • RS, SSSRMAP, others…

7. Desirable Features of an XML Style • Completeness • If you can think it, you can write it • High value of validation • Function signature type checking • Protection of components from poorly formed messages • Simplified component code • Extent to which documentation is expressed in schema • Readability • Can a human understand the XML text? • Conciseness • Desirable, but might conflict with readability • Scalability • “Atomicity” • Can race conditions be avoided?

8. Things to Look for in any syntax • A command message does three things • Matching a set of objects in components data store • Either constructs them or identifies them • Can use similar syntax • Applies a function with arguments to that set of objects • Constructs return message • May be complex, containing partial information about objects identified

9. Restriction Syntax BNF [InMSG] ::= <[Command]> [Predicate] </[Command]> [Command] ::= [CommandName] | [CommandName] [Args] [CommandName] ::= [identifier] [Args] ::= [Arg]|[Arg] [Args] [Arg] ::= [identifier]=[literal] ]

10. RS BNF for Predicates [Predicate] ::= <[Object-Description]> | <[Object-Description]> [Predicate] (must match one or more) [Object-Description] ::= [ObjectType] | [ObjectType] [Matchers] | [ObjectType]> <[SubObjectDescription] | [ObjectType] [Matchers]> <[SubObjectDescription]>

11. RS BNF (boring part) • [Matchers] ::= "list of key val pairs" (must match all) • [SubObjectDescription] ::= [ObjectDescription] • [identifier] ::= [letter]|[letter][identifier] • [literal] ::= [identifier] • [ObjectType] ::= [identifier]

12. Restriction Syntax Examples • Examples are explained in terms of • Object matching • Function execution • Data return

13. Simple Data Type, Null function, Simple Matching <get-node-state> <node-state node='ccn22’ adminstate='*‘ state='*’ /> </get-node-state> 1. All node state objects are matched where node = 'ccn22'; adminstate and state can have any value. ('*" is a wildcard) 2. Get functions are actually null operations; that is, they don't actually modify matched objects or perform any object operations. 3. All fields included in the query are included in the response, so any matched objects are returned including fields node, adminstate, and state. Since there will only be one node-state element per node, and nodes are uniquely named, this query will yield either 0 or 1 response elements. The response will look like: <node-states> <node-state node='ccn22' adminstate='offline' state='up'/> </node-states>

14. Simple Data Type, Regular Function, Extended Matching <diagnose type='online' parallel='true'>   <node-state match='extended' node='re:^ccn\d+$' adminstate='=:*'  state='!d=:up'/> </diagnose> 1. All node state objects are matched where node matches the regular expression "^ccn\d+$". The state attribute must be "up". Adminstate may have any value. Extended matching also provides numeric comparisons, range membership and negation. The operator prefix "!d" is ignored when matching. It is only used for data return. 2. The diagnose function is executed on all matching node-state objects, with function arguments type (=online) and parallel (=true). 3. All attributes are returned, as with the simple matching syntax, unless the operator prefix !d is specified. In this case, the node and adminstate fields are returned. The state field is not retured because its operator was prefixed with "!d“ This return would look like: <node-states>   <node-state node='ccn1' adminstate='online'/>   <node-state node='ccn2' adminstate='online'/>   <node-state node='ccn3' adminstate='online'/>   <node-state node='ccn4' adminstate='online'/>   <node-state node='ccn5' adminstate='offline'/>   <node-state node='ccn6' adminstate='online'/>   <node-state node='ccn7' adminstate='online'/>   <node-state node='ccn8' adminstate='offline'/> </node-states>

15. Complex Data Type, Regular Function, Simple Matching <signal-process-group signal='SIGINT' scope='single'>   <process-group pgid='*' submitter='*'>      <process host='ccn221' pid='*'/>      <process host='ccn222' pid='*'/>   </process-group> </signal-process-group> 1.Since almost all fields are '*', and simple matching is employed, there is only one predicate on this example. We are looking for process groups where there exists at least one process on ccn221. 2. the function signal-process-group, with the arguments signal=SIGINT  and scope='single', is applied to each matched process-group  object.  3. The pgid and submitter fields for each process group are returned,  along with hostname and pids for each of the processes on ccn221 or ccn222, not those of the entire process group. <process-groups>   <process-group pgid='2232' submitter='user1'>       <process host='ccn221' pid='2232'/>       <process host='ccn222' pid='2542'/>    </process-group>    <process-group pgid='2240' submitter='user2'>       <process host='ccn222' pid='2531'/>       <process host='ccn221' pid='1432'/>    </process-group> </process-groups>

16. Grouping <signal-process-group signal='SIGINT' scope='global'>         <process-group submitter='desai' pgid='*' totalprocs='32'>             <process host='ccn27'/>          </process-group>          <process-group submitter='lusk' pgid='*' totalprocs='32'/>          <process-group submitter='*' pgid='22'/> </signal-process-group>

17. The “Set” element <set-node-state state='up' adminstate='online'>   <node-state node='ccn83'/>   <node-state node='ccn231'/> </set-node-state>

18. Same Examples in SSSRMAP

19. Simple Data Type, Null function, Simple Matching <Request action='Query' id='1'>        <Object>node-state</Object>        <Get name='node'/>        <Get name='adminstate'/>        <Get name='state'/>        <Where name='node'>ccn22</Where> </Request>

20. Simple Data Type, Regular Function, Extended Matching <Request action='Diagnose‘ id='1'>        <Option name='type' value='online'/>        <Option name='parallel' value='true'/>        <Object>node-state</Object>        <Get name='node'/>        <Get name='adminstate'/>        <Where name='node' op='like'>^ccn\d+$</Where>        <Where name='state' conj='and'>up</Where> </Request>

21. Complex Data Type, Regular Function, Simple Matching <Request action='SignalProcessGroup'>         <Option name='signal' value='SIGINT'/>         <Option name='scope' value='single'/>         <Object>ProcessGroup</Object>         <Get name='pgid'/>         <Get name='submitter'/>         <Get name='ProcessGroup//Process//Host"/>         <Where name='ProcessGroup//Process//Host">ccn221</Where>         <Where name='ProcessGroup//Process//Host“ conj='or'>ccn222 </Where> </Request> This example isn't exactly the same as in RS; it is not possible to only return subobjects that match search criterion (and elide ones that do not) • Validation problems: • Can’t have validation restrict the values of Option name to ‘signal’ and ‘scope’ when Request action is 'SignalProcessGroup‘ • Can’t limit the value of the Object text node to ‘process group’ when the Request action is 'SignalProcessGroup‘ • Can’t validate that 'ProcessGroup//Process//Host“ is a valid hierarchy • Can’t validate that the value of ‘name’ for Get/where elements is appropriate for Object

22. Grouping <Request action='SignalProcessGroup'>   <Option name='signal' value='SIGINT'/>   <Option name='scope' value='global'/>   <Object>ProcessGroup</Object>   <Where name='submitter' group='4'>desai</Where>   <Where name='processgroup//process//host' conj='and‘ group='- 1'>ccn2</Where>   <Where name='submitter' group='-1' conj='or'>lusk</Where>   <Where name='totalprocs' group='-1' conj='and'>32</Where>   <Where name='pgid' conj='or' group='-1'>22</Where>   <Get name='submitter'/>   <Get name='totalprocs'/>   <Get name='pgid'/> </Request>

23. The “Set” Element <Request action='SetNodeState'>   <Object>NodeState</Object>   <Where name='node' group='1">ccn83</Where>   <Where name='node' group='-1'>ccn231</Where>   <Set name='state'>up</Set>   <Set name='adminstate">online</Set> <Request>

24. Similarities and Differences Both do all three things: • Matching • RS in attributes of object list in Predicate • SSSRMAP in “where” clause • Apply function • Named as top-level XML node in RS • Value of action attribute in Request node in SSSRMAP • Function arguments are attributes at top level in RS • Function arguments are option subelements in SSSRMAP (limits value of validation)

25. Similarities and Differences (cont.) • Contents of return message: • Overloading of attributes in predicate for matching in RS • “Get” subobject in SSSRMAP • Complex matching • Both have regular expressions for values – ugly but necessary • SSSRMAP has complex where’s with explicit conjunction/disjunction/parenthesis-counters, semi-infix operators • RS has implicit and’s/or’s, relies on conjunctive normal form to eliminate necessity for parentheses.

26. Comparison by Desirable Features • Completeness • Probably equal; both lack generalized negation • Validation Value • Nod to RS; use of specific commands at higher level enables stricter input editing at schema level • Readability • Equal with slight nod to SSSRMAP if “where” clause is not too complicated • Conciseness • Nod to RS • Atomicity • Probably equal, given recent additions to SSSRMAP • Scalability – nod to RS (example)

27. Critique of Both XML Styles • Restriction syntax • Puts too much in attributes for conciseness • Overloads use for matching and returning • Extended matching is terrible • SSSRMAP • Is “wordy” • Lowers value of validation • Grouping attribute to represent multiple parentheses is hard to read • Partial infix notation is unusual • Both • General grunginess

28. Less Restrictive Syntax(Eliminating the worst of both worlds) • Keep high-level specification of commands as in RS, to raise value of validation • Move attributes in RS to subobjects as in SSSRMAP • Provides more “room” (in attribute space) for specifying matching details • Explicitly specify fields to return as in SSSRMAP, but using attribute space for conciseness

29. Examples in New Syntax Style <GetNodeState>   <NodeState>     <Node>ccn22</Node>     <AdminState match='false'/>     <State match='false'/>   </NodeState> </GetNodeState> <Diagnose mode='online‘ type='parallel'>   <NodeState>     <Node op='re'>ccn\d+</Node>      <AdminState match='false'/>     <State display='false'>up</State>   </NodeState> </Diagnose>

30. Examples in New Syntax Style <SignalProcessGroup signal='SIGINT' scope='single'>   <ProcessGroup>     <PGID match='false'/>      <Submitter match='false'/>      <Processes>        <Process>            <Host>ccn221</Host>           <Pid match='false'/>       </Process>       <Process>           <Host match='false'/>           <Pid match='false'/>       </Process>    </Processes>   </ProcessGroup> </SignalProcessGroup>

31. Advantages and Disadvantages of New Syntax • High value to validation • Gives up some conciseness to clean up overloading of attributes • Eliminates complex where’s

32. Summary(my opinion) • Substantial investment has been made in existing XML interfaces for existing components • The syntaxes in use have their advantages and disadvantages. • There is no need to have a single syntax style. • New components from new sources (e.g. vendors) should be allowed to specify their own interfaces. • They are likely to pick a style in use. • Forcing anyone who has contributed components to re-engineer their interfaces, especially to a style they don’t prefer, will cause more harm to the project than good. • Let 1,000 (or at least 2) flowers bloom!