1 / 33

Dept. of Computer Science, NC State University cbl.ncsu/OpenProjects/OmniFlow/

A Universal Client for Distributed Networked Design and Computing. Franc Brglez Hemang Lavana**. Dept. of Computer Science, NC State University http://www.cbl.ncsu.edu/OpenProjects/OmniFlow/ **H. Lavana is now with Cisco Systems, Inc. Outline.

posy
Download Presentation

Dept. of Computer Science, NC State University cbl.ncsu/OpenProjects/OmniFlow/

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Universal Client for Distributed Networked Design and Computing Franc Brglez Hemang Lavana** Dept. of Computer Science, NC State University http://www.cbl.ncsu.edu/OpenProjects/OmniFlow/ **H. Lavana is now with Cisco Systems, Inc.

  2. Outline • Generic Overview of the Universal Client • DA Background and Motivation • Taskflow Architecture • Taskflow Programming and GUI Highlights • Summary and Conclusions

  3. What Does OmniFlow Client Do ... (1) reads user-programmed interconnections of distributed component programs as task instances (TCP protocols are transparent); (2) dynamically renders a highly interactive GUI of all interconnected components as a hierarchical taskflow; (3) dynamically creates a schedule to execute component programs concurrently, serially, or not at all, depending on the user-defined runtime configuration of the taskflow topology.

  4. Concepts Behind the OmniFlow Client (1) XML defines a cdtML schema of distributed, encapsulated blackbox (whitebox) components (stand-alone programs, point-tools), primitive glue components xJoin, xFork, FSMD, and TaskGraph and DataGraph structures; (2) User-entered cdtML description defines a (hierarchical) taskflow of blackbox (whitebox) instances in a TaskGraph and a DataGraph; (3) OmniFlow parses cdtML description and renders five views of a highly interactive GUI. OmniFlow not only simulates the behavior of distributed and connected programs, it is a program-of-programs with its own highly interactive GUI environment ….

  5. Taskflow GUI and Scheduling Engine cdtML Schema cdtML Taskflows Begin Evaluate ControlJoin False Tcl-xml Parser NoOp Enable Task Repeat until Validate Taskflow Specs Execute BBC or expand WBC Display Taskflow Tree, Graph User False Evaluate ControlFork NoOp No more tasks Foreach child task Taskflow Scheduler End

  6. Taskflow User-Entry (via XML Editor) cdtML Taskflows editor display cdtML Schema Validating XML editor

  7. A cdtML Entry By a `Real Programmer’ (real programmers use plain text editors, e.g. ‘vi’) <Cdtml> <MainTask> <TaskList> <Begin/> <Task instance="(parabola)" taskRef=" \ parabola_defn.cdt#parabola"/> <End/> </TaskList> <TaskGraph> (BEGIN) => (parabola) => (END) </TaskGraph> ………... </TaskInstance> </MainTask> </Cdtml>

  8. Taskflow Instance (cdtXML Start-Up) Start-up Tree View Start-up Graph View Selector View Status View INFO View user-probing of DataGraph

  9. Some OmniFlow Point & Click Features • open, close, ascent, and descent of the tree and the graph hierarchy. • reconfiguration of an invocation, repeatInvocation, and abortInvocationedge into a `closed' or `open' state. • invocation of the taskflow and its schedule by clicking on any of its task nodes. • abort of the taskflow by clicking on one or more of the executing task nodes. • reset of the taskflow state, propagated in a descending order of taskflow hierarchy. • access to view/edit data associated with each task, i.e. input (output) data nodes of each task.

  10. Taskflow Inst. (re-configured in GUI) “green” state shows task completion disabled invocationEdge disabled repeatInvEdge Enabled concurrent execution of tasks {(InitB), (InitC)}, then {(B), (C)}

  11. Related Work • University-based research, e.g. Reuben (DAC’97), Weld (DAC’98), JavaCADD (TR’98), CollabTop (DAC’99), OpenDesign (ICCD’00) • Structured Workflow Modeling and Middleware, e.g. http://www.icis.qut.edu.au/~arthur http://www-4.ibm.com/software/ts/mqseries/ • Collaboration technologies, e.g. http://www.cpsc.ucalgary.ca/grouplab/groupkit/ • Technologies such as XML, OMG, JavaSOFT, COM ….

  12. DA Background and Motivation …. • Informal problem description • Matching S/W components to task instances • Informal taskflow construction for -- concurrent and distributed task execution -- serial and distributed task execution

  13. Informal Problem Description Task A1: std. cell placement of netlist N(a) (@host a1) Task A2: std. cell placement of netlist N(a) (@host a2) Task B1: block placement of netlist N(b) (@host b1) Task C1: routing of placed N(a) & N(b) (@host c1) Task D1: N(a) & N(b) post-layout simulation (@host d1) Notes: • tasks may be hierarchical, owned by distributed team • hosts may be under different file systems !!

  14. Matching S/W Components to Task Ins. Repeat Task A1 (@a1) Repeat Task A2 (@a2) Repeat Task B1 (@b1) Sel 2-of-3 Task C1 (@c1) Do Task D1 (@d1) What are the key challenges in this example? (1) invoke and execute {A1, A2, B1} concurrently (2) when first 2-of-3 from {A1, A2, B1} complete -- abort uncompleted task from {A1, A2, B1} -- invoke and execute C1 (3) when C1 completes, -- invoke and execute D1

  15. Concurrent Taskflow Composition Repeat Task A1 (@a1) Repeat Task A2 (@a2) Beg (0) invoke taskflow Repeat Task B1 (@b1) (1) invoke and execute {A1, A2, B1} concurrently ….

  16. Taskflow Aborts and Synchronization Repeat Task A1 (@a1) (2) when first 2-of-3 from {A1, A2, B1} complete …. Sel 2-of-3 Task C1 (@c1) Repeat Task A2 (@a2) Beg Repeat Task B1 (@b1) -- abort uncompleted task A1 -- invoke and execute C1

  17. Serial Taskflow Composition Repeat Task A1 (@a1) -- invoke and execute D1 Sel 2-of-3 Task C1 (@c1) Do Task D1 (@d1) Beg Repeat Task A2 (@a2) End End Repeat Task B1 (@b1) (4) terminate taskflow (3) when C1 completes …. Note: with Beg/End tasks, a taskflow is a polar graph ...

  18. Taskflow Architecture & Programing ... • Problem Abstraction • Blackbox (Whitebox) as Component Instances • Taskflow as Intersection of TaskGraph, DataGraph • Task Encapsulation Architecture • cdtML Taskflow Schema in XML

  19. Problem Abstraction Given: L blackbox components (executable programs) and data; M whitebox components (compositions of black/whitebox components) and data; K hosts of L+M black/whiteboxes; N owners of L+M black/whiteboxes. Three problems: (1) compose a new whitebox component; (2) transfer data within the new whitebox component; (3) schedule concurrent/serial execution of all component instances contained in the new whitebox component.

  20. WhiteBox as a TaskFlow of Components Programmable: Interactive, user-configurable during runtime: Collaborative: asynchronous/synchronous mode Platform-independent: accessible from a web browser - tasks can be sequenced serially or concurrently - task may be repeated a number of times - the choice of sequence can be decided by data - only one entry and one exit point - enabling/disabling of control edges to control the scope of execution - enabling/disabling taskflow for execution with local or global flow data

  21. Black/Whitebox Ports and Data Types Task A1 (@a1) Task A1 (@a1) Task B1 (@b1) Control ports: Data types: Data ports: task invocation input persistent task abort inout temporary task status output outin

  22. BlackBox/WhiteBox Connectors (Edges) singleInvocation control edge (task-task, opened/closed in GUI) repeatInvocation control edge (task-task, opened/closed in GUI) abortInvocation control edge (task-task, opened/closed in GUI) transfer data edge (data-task, task-data or data-data)

  23. Taskflow: Intersection of Two Graphs (1) ControlGraph AND (2) DataGraph Task Inst.1 End Join Beg Fork Task Inst.3 Task Inst.2 • User can re-configure (on/off) all control edges • Each task instance encapsulates FIVE primitives

  24. Task Instance Architecture … from other tasks … to other tasks Task Instance Layer Encapsulation Layer Control Join FSMD Control Fork Data Mux BBC/ WBC local/flow data

  25. Task Encapsulation Layer (Outline) Finite-State-Machine as per FSM transition table: •••• •••• Datapath (as per datapath table) invoc/abort signal completion status BBC/ WBC Din Dout { Waiting, Enabled, Xecuting, Completed Done, Aborted, TimedOut, Skipped }

  26. cdtML Taskflow Schema (using XML) Encapsulated SingleTask Encapsulated MultiTask TaskInstance ControlJoin SingleTaskDefn MultiTaskDefn DataMux Encaps.Task I/O PortList …... RepeatCond. TaskInstList ControlFork TaskGraph MainTask SingleTaskBody MultiTaskBody TaskInstList BeginFork BeginFork TaskGraph TaskInst1 BeginFork BlackBox Cp. …... EndJoin TaskInst1 EndJoin …... DataGraph EndJoin I/O PortList …...

  27. A Summary of Simplified Definitions A taskflowschedule is a dynamically generated sequence of concurrent and serial task invocations/aborts A task or a taskflow is an instance of an encapsulated blackbox component or (any legacy program accesible via TCP protocol using telnet-, ssh-, http-, socket-based clients) an encapsulated whitebox component (a directed polar graph of blackbox and whitebox components)

  28. Taskflow Programming and GUI ... • early DA projects reported in ICCD'00, demos at DAC’99, DAC'00 • a number of projects reported in PhD Thesis'00 (largest taskflow has 9150 task instances, longest path of 1600 task instances) • demos at DAC'01 Major GUI Views and evaluations (this talk): • selector, tree, graph, status, info • graph view for interactive programming • scalability of taskflow scheduler

  29. Taskflow Tree View (Dyn. Generated) - (taskFlowInstance1 Traditional display of hierarchy …. (follows cdtML schema) - InputList • mIn1 = “fileIn1.t” • mIn2 = “fileIn2.t” - OutputList - TaskList open graphView • (Begin) |x| invoke task } - (TaskA) |o|x| A graph view, shown next, depicts additional user-configurable dependencies …. + InputList multi task + OutputList + TaskList - (TaskB) |x| } + InputList single task + OutputList + (TaskC) |o|x| • (End) • mOut1 = “fileOut1.r”

  30. Taskflow Graph View (Dyn. Generated) sim. (noData) exec. (locData) exec. (flowData) view edit INVOKE: DATA: 8-state color code (TaskA) whitebox Click to view/edit data (Begin) (End) Selector View (TaskC) user- controlled edges Click on any task: (TaskB) - to invoke blackbox - to abort

  31. Interactive Taskflow Programming (TaskA) (TaskB) (TaskA) (TaskA) Invoke2 (Begin) (Begin) (Begin) (End) (End) (TaskC) (TaskC) (TaskC) (TaskB) user disconnects control edges Invoke1 Runtime switch from to serial execution concurrent Can do similarly for abort and repeatInvocation edges

  32. Scalability of the OmniFlow Scheduler high concurrency taskflow 2400 100 0.408 low concurrency taskflow 2400 1600 0.384 Task Longest Overhead Instances Path [seconds/task] -------- ------ ------------ 300 100 0.396 300 200 0.412 9150 1600 0.381 … for a SUN workstation under unix/solaris. Comparable performance has been observed on workstations under linux/macOS/windowsNT

  33. Summary and Conclusions • Using XML: user encapsulates networked component programs, as task instances and programs their interconnections. • Omniflow client reads XML description -- and dynamically renders a highly interactive GUI. • The user-configurable environment may be preferable to hard-wired flow environments of today. For more info about OmniFlow client, thesis, and paper reprints/preprints, see http://www.cbl.ncsu.edu/OpenProjects/OmniFlow/

More Related