1 / 14

Systems Group Dept. Computer Science ETH Zurich - Switzerland

XQBench An XQuery Benchmarking Service. Systems Group Dept. Computer Science ETH Zurich - Switzerland. Peter M. Fischer. Motivation and Goal. Benchmarks helped RDBMS/SQL to establish themselves: Highlight technical challenges Provide marketing arguments

rahim-davidson
Download Presentation

Systems Group Dept. Computer Science ETH Zurich - Switzerland

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. XQBench An XQuery Benchmarking Service Systems Group Dept. Computer Science ETH Zurich - Switzerland Peter M. Fischer

  2. Motivation and Goal • Benchmarks helped RDBMS/SQL to establish themselves: • Highlight technical challenges • Provide marketing arguments • No well-defined and well-accepted benchmarks for XQuery • Not another benchmark proposal! (yet?) • Provide a service to perform XQuery benchmarks • Simplify measurements • Compare performance and results • Collect interesting use cases Systems Group. D-INFK. ETH Zurich

  3. Benchmarking Model • Model various use cases of XQuery • Compare apples to apples • Processors (cold start) • Databases (ad-hoc loading, preloaded) • ...? • Currently: evaluate individual read queries on static datasets • Measure overall time always, details where possible • Observe resource consumption and results • Allow users to run • Existing Benchmarks • Contribute Queries • Contribute Documents & Schemas Systems Group. D-INFK. ETH Zurich

  4. Usage (Short Demo) • Experiment Creation and Submission • Choose Engines, Documents, Queries • Notification on submission, execution start, exeuction completion • Pending Experiments shown to registered users • Result Search • Direct Link at Notification • Complete List • Directed GUI search for documents, queries, engines • General XQuery for complex searches/correlations • Result PresentationFor every outcome, multiple views: • Experiment description • HTML (textual) for human consumption • XML for further querying • Plots (PNG, PS) in various dimensions, also plot scripts and data • Execution Log, Engine Output Systems Group. D-INFK. ETH Zurich

  5. Usage (Short Demo) • Public and Private Data/Queries/Experiments • Everybody can run and search standard benchmarks • Registered users can add own data and queries • Private data and outcomes only accessible to owner • Private data and outcomes can be made public • Managing Documents and Queries • Adding new Datasets (including schema) • Adding new documents: Files+Metadata • Adding new queries: Text+Metadata • Access to metadata • Engines: Installation Log, «Adapter», Wiki Page Systems Group. D-INFK. ETH Zurich

  6. Data and Metadata Model Systems Group. D-INFK. ETH Zurich

  7. XQuery Engine Abstraction • No common API, no common set of operations among engines • Unify at least relevant parts • Interaction Model: • Execute (including pre+post operations) • Engine Lifecycle: Start/Stop • Data Lifecycle: Load/Delete • Timing: • Measure from outside • Engine reporting: query execution, parsing, serializing, ... • Tuning currently not supported: we have no good abstractions! (yet?)Running engines (mostly) with out-of-the-box settings Systems Group. D-INFK. ETH Zurich

  8. Abstraction: Adapter Example <adapter> <engine id="BaseX-6.0"> <command> <startup><![CDATA[(java -Xmx6000m -cp #path/BaseX.jar org.basex.BaseXServer -p 1984 > /dev/null 2>&1&); bin/wait_service_up_port.sh 127.0.0.1 1984]]></startup> <executable><![CDATA[java -cp #path/BaseX.jar org.basex.BaseXClient -p 1984 -Uadmin -Padmin -o #result -v #query > #times 2>&1 ]]></executable> ... </command> <times> <factor_time>0.001</factor_time> <time id="t1"> <line>.*Parsing: (\d+\.\d+) ms </line> </time> <doc_processing_time>#t1</doc_processing_time> ... <total_time>#t5</total_time> </times> ... Systems Group. D-INFK. ETH Zurich

  9. Benchmark Service Architecture Systems Group. D-INFK. ETH Zurich

  10. Supported Engines • Policy: cover all stable versions of open-source platforms • Development versions change too often • Will update to «intermediate» if this version fixes a significant problem (e.g. BaseX attribute number limitation) • Commercial: Cost money to run, restrictions on publishing results • Sponsorship, Permission to publish welcome! • Currently: • DB: BaseX, BerkeleyDB, eXist, MonetDB, Sedna • Processor: QizX Free, Saxon B/HE, XQilla, Zorba Systems Group. D-INFK. ETH Zurich

  11. Pre-Installed Benchmarks • Policy: provide as many existing workloads as possible • The usual suspects • XMark + XPathMark • TPoX • Currently collecting more: • The old guard: XMach-1, X007, XBench, Michigan • Recent Proposals: EXRT, XWeb, MeMBer, BooX, ... • Interesting «normal» workloads: FDA, Geodata, ... • Existing Benchmarks secondary goal • You should use/contribute your own workloads • Contributed data, queries and results stay private until you decide to share them! Systems Group. D-INFK. ETH Zurich

  12. Status • ETH provides frontend and several backend nodes (with support from th FLWOR Foundation) • Available at „beta“ stage • Helper infrastructure • Bugtracker • Wiki • Mailing List • Components are/will be open source Systems Group. D-INFK. ETH Zurich

  13. Future Directions • Other languages • XSLT • SPARQL • ... • Improve collaboration features • Extend interaction model • Schema • Indexes • Extended Workloads • Update/Query mixes • User-defined execution flows • ... Systems Group. D-INFK. ETH Zurich

  14. xqbench.org Systems Group. D-INFK. ETH Zurich

More Related