“Bookshelf.exe”: Executable Extensions of the Bookshelf

1. “Bookshelf.exe”: Executable Extensions of the Bookshelf Marius Eriksen and Igor Markov University of Michigan, EECS

2. Outline • Making the bookshelf more dynamic • Implementations + data = algorithm evaluations • Potentially more than that… • Related efforts • SatEx, PUNCH, NEOS, OmniFlow • Proposed solution: ``Bookshelf.exe’’ • Services offered: • online execution, ASP, sharing of results, flow scripting,… • Interfaces • Support for optimization-specific concepts • Power versus ease-of-use and modularity • Architecture and implementation details

3. Makingthe Bookshelf More Dynamic • Existing bookshelf: • A static collection of data and implementations • Manual addition of material • No automatic evaluations, reporting of results • Those limitations are not fundamental • Artifacts of the current implementation • Proposal for improvement: “Bookshelf.exe”

4. Related Efforts • SatEx http://www.lri.fr/~simon/satex/satex.php3 • Specialized in satisfiability problems • PUNCH http://punch.ece.purdue.edu • Very broad selection of software • From StarOffice and gnulot to Capo and Feng Shui • Local to Purdue Univ. • NEOS http://www-neos.mcs.anl.gov/neos • Open-source, distributed architecture • Used primarily for linear and non-linear optimization • OmniFlow: DAC 2001, Brglez and Lavana • http://www.cbl.ncsu.edu/OpenProjects/OmniFlow • Distributed Collaborative Design Framework for VLSI • GUI-based flow control, chaining of design tools

5. SatEx • Continual evaluation and ranking of submitted implementations • Results produced and posted on the Web automatically • Intuitive interface • Popular • 93,707 hits since March, 2000 • 23 SAT provers, 32,610 executions • Limited scalability • Runs on oneworkstation • 529 days, 17 hours, 53 minutes and 20 seconds of CPU time • Specialized to one application

6. PUNCH • Very general execution framework • Everything from domain-specific algorithm implementations to GUI-based office applications • Custom-designed filesystem (restricts to Purdue hosts) • 241,458 executions in 5 years • 8,152 on VLSI CAD related applications • 20+ publications • Only maintainer can add executables • No support for evaluation and chaining/flows • No stats for results of runs (cf. “top 20” on SatEx ) • No MIME-like data types (even IE/netscape need them!) • Difficult to use when multiple tools are involved

7. NEOS • Open-source, distributed framework • Wide use, a solid library of implementations • Adding new implementations requires maintainer intervention (but less than on PUNCH) • Each new implementation must come with a host • Distributed maintenance • Loose data typing • No type system for data and implementations • Compare to MIME

8. NEOS (what can be improved) • Independent eval. and verification of results • E.g., PUNCH offers a WL-eval. From the bookshelf • Real-time on-line reporting of results + stats • Provisions for chaining multiple implementations • Scripts to control the execution and evaluation flows • Provisions for pairing exec’s with benchmarks • Goal: SatEx-like evaluation,but for multiple data types

9. OmniFlow • Context: collaborative VLSI Design • collaboration in terms of computational resources • no provisions for sharing results on the Web • Distributed over multiple hosts • Provides GUI-based flow control • supports chaining of design tools • several hard-coded conditions for flow control • no support for execution conditional on results • no scripting language; limited by GUI • Cannot dynamically add hosts

10. Bookshelf.exe • Best of SatEx, PUNCH, NEOS and OmniFlow • Reporting style similar to SatEx (+ alternatives) • The versatility of PUNCH • Scalability and distributed nature of NEOS (or better) • Flow control as in OmniFlow or better • New features • MIME-like data types and optimization-specific concepts • Automatic submission of binaries and source code • Chaining of implementations; scripting for flow control • Support for use models with proprietary data or code

11. Bookshelf.exe (2) • Scalability • Computation is distributed (unlike in SatEx) • Maintenance is automated (unlike in NEOS) • Support for multiple use models • “adapts to users” • Multiple levels of expertise • Multiple levels of commitment • Sharing of public data • Hiding/protection of proprietary data • “Screen-saver” mode, similar to SETI@Home, Entropia, etc

12. Bookshelf.exe (3) • Provides basic algorithm evaluation infrastructure to research groups • Avoid large maintenance costs • Just as usable for larger groups

13. End-user Interface • Web site • Using CGI/PHP for interactive content and database interaction • Submission through the web site (email is ok) • Dynamic addition of computation hosts • Stores data locally or uploads on demand • Automatic Web-based reports and statistics

14. End-user Interface (2) • User accounts (including anonymous) • Every user has private resource-limited “workspace” • Job submissions must be from accounts • Results of jobs stored on accounts • Anonymous jobs possible, but can use fewer resources • users are encouraged to register • Many policy questions (no roadblocks seen) • e.g., can different owners chain their jobs? • Transparent error diagnostics • Greatly improves learning curve

15. End-user Interface (3) • Result notification (similar to NEOS) • E-Mail • Web reports • Script composer for making run flows without needing to know a scripting language • less flexibility, but better learning curve • GUI implemented as form that POSTs scripts

16. Script Composer • HTML Form interface – basic flow with fundamental predicates and functions • All API functions • E.g. “run algorithm ‘optimizer 2’ and store the results” • Several common Perl functions • Conditional statements • E.g. “verify if quality is more than 60” • Iteration support • E.g. “iterate until quality is 100 or the number of iterations is 100”, “run randomized solver 3 timers, report the best result” • Start with one ‘step’ – add an arbitrary number of steps

17. Optimization-specific Concepts in Bookhelf.exe • Type system (similar to MIME) for all types of submissions • Rules for matching submissions of different types • E.g., match a placer with a LEF/DEF benchmark • Types include • Algorithm implementations • Deterministic and randomized optimizers, etc. • Input data, results • Common benchmarks

18. Architecture Overview

19. Component Types • Where relevant, components also have types • Data • Data format • E.g. Cadence LEF/DEF, DIMACS CNF • Implementations • Architecture (solaris, linux, ‘source code’, etc.) • Class (optimizer, randomized solver, etc.)

20. Component Descriptors • Unique id, name and description for each component • Owner field • Component history • E.g. usage of a certain implementation, piece of data or benchmark • Component specific fields • Associated expert for implementation, incremental diff for data • Component version • Component changelog

21. Data Controller • Backing store • Permanent storage of all involved components • Backed by database and file system • Data, results suited for database • Implementations suited for file system storage • Components may be deferred to local storage • Proprietary, non-disclosed components on a host owned by the proprietor of the component • All data stored with complimentary descriptor

22. Job Controller • Responsible for communicating with and controlling all agents (and thus jobs and implementations) • Initiates jobs • Delivers user instructions • Handles data retrieval and storage • Interfaces with data controller • Keeps track of node load, etc. for proper distribution

23. Host Job Controller • Resides on each node in the distributed network • Basic token of presence of DS-OSS • Reports statistics to job controller • Load, memory usage • Signals when ready to take jobs • Launches agents as necessary • Performs self upgrades • New software revisions, bugfixes

24. The Agent • Responsible for running job on node • Provides environment for script to run in • API • Script sandbox/jail • For security reasons • Provides common callbacks if not provided by scripts • Data storage/retrieval

25. The Agent (2) • Input/Output of data from script • Job notification and control • Start, stop, termination • Maintains local demands • Resource usage • Stop on local demand

26. Scripts • In Perl • Interfaces directly with an algorithm, or expert handling it • API support • Full Perl library • Job control functionality • Data storage and retrieval

27. Scripts (2) • Checks for conflicting input data types and algorithm input type • Generates appropriate, understandable errors • A number of automatic tasks • Pass thru data storage (e.g. results of a single run) • Can provide all steps automatically, leaving the user to do exactly what he/she wants

28. The Expert • Wrappers for proprietary implementation interfaces • Knows how to deal with a particular interface • E.g. command line options, configuration files • Translates from well known parameters (specified from the API) to proprietary interface

29. In Conclusion • DSS-OS provides features that are missing in current systems • Existing systems are limited by their architecture and thus cannot provide these needed features. • Provides an infrastructure that is scalable • Computationally scalable • Scalable to proprietary vendors

30. In Conclusion (2) • Scripting support allows for flexible execution flows • Conditional flows, iterative flows • Provides an established infrastructure for smaller research groups • Enjoying advantages that only large proprietary, high-maintenance systems previously could provide.