1 / 14

Archiver Hypertable Engined (HyperArchiver)

Explore improvements for the EPICS Archiver embedded DB to enhance reliability in large-scale applications. Consider restructuring and alternative DB options for optimal performance.

kristyp
Download Presentation

Archiver Hypertable Engined (HyperArchiver)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Archiver Hypertable Engined (HyperArchiver) Loris Giovannini, Mauro Giacchini Epics Collaboration Meeting 03/06, 2010 L.Giovannini -- M.Giacchini

  2. Target Make the EPICS Archiver embedded DB more reliable and safe especially in large application with a large number of PVs to store First Approach: re-write the Rtree and double-linked-list embedded DB structure Second approach: look for a complete replacement of the embedded DB with new one L.Giovannini -- M.Giacchini

  3. Why not Beauty ? (Best Ever Archiver Utility) Beauty = Classical Archiver over Oracle DB Pros: SQL statements accept Professional Solution Cons: Not so fast Really expensive (licenses) L.Giovannini -- M.Giacchini

  4. Why not MySQL ? Potential scalability concerns Designed for a single machine (not distributed) What it takes to make it scale Major engineering effort Solutions are usually ad hoc Solutions usually involve horizontal partitioning + replication Solutions involve expensive hardware L.Giovannini -- M.Giacchini

  5. BigTable ? Proprietary DB used by Google over their proprietary Google File System High Performance Large amount of data Stable Reliable Distributed L.Giovannini -- M.Giacchini

  6. HYPERTABLE…what is it ? Hypertable has been developed as an in-house software at Zvents Inc. In January 2009, Baidu, the leading Chinese language search engine, became a project sponsor. Rediff.com is one of the premier worldwide online providers of news, information, communication, entertainment and shopping service. Hypertable is licensed under the GNU General Public License Version 2. “Our goal is nothing less than that Hypertable become one of the world’s most massively parallel high performance database platforms.” Not RDB L.Giovannini -- M.Giacchini

  7. Hadoop/Hypertable Solution Objective • Handles applications with large datasets • Fast fault detection and fail-over • High throughput processing • Centralized scheduling of tasks and execution of batch processes • Can be deployed on low cost, commodity hardware • Eliminates or reduces the need for expensive table joins L.Giovannini -- M.Giacchini

  8. Machines used to the tests L.Giovannini -- M.Giacchini

  9. Oracle Vs Hypertable Oracle insertion time : peak in write test (8Ksamples/sec) Hypertable insertion time: 1.777.600 per 1min 13 sec (24Ksamples/sec) Oracle retrieve time: 36.657 samples in 79.294 sec (~462 vals/sec) Hypertable retrieve time: 57624 samples in 0.36 sec ( 160K vals/sec) in half hour L.Giovannini -- M.Giacchini

  10. Hypertable into CSS THE OUR GOAL • storing: 10000 channels per second • retrieve: 1000 samples for up to 4 signal in less than 1 second L.Giovannini -- M.Giacchini

  11. Retrieval outside CSS First Step: Develop a Java Program for only test the performances for retrieval without any elaboration of data to valuate the possibility to respect our goals using Thrift Java client API (Mandatory to interface with Hypertable) Results 1000 sample for 4 Pv: 1000 sample:22ms 1000 sample:15ms 1000 sample:15ms 56Ksample/sec 1000 sample:19ms L.Giovannini -- M.Giacchini

  12. Retrieval L.Giovannini -- M.Giacchini L.Giovannini -- M.Giacchini

  13. TIME Retrieval into CSS • Second Step: • Develope an Eclipse plug-in org.csstudio.archivereader.htrdb for the support of • Hypertable like as org.csstudio.archivereader.rdb • Working in Progress • The extraction time into CSS is about 10 times worse: • The reasons are under study, there are two probable causes: • initial query latency of Thrift API when connect to Hypertable • wrong classes design into CSS architecture L.Giovannini -- M.Giacchini

  14. Acknowledgments • Kay Kasemir (ORNL) • Doug Judd (Hypertable Core Team) • Bob Dalesio, Ralph Lange, Robert Petkus (BNL) More information www.lnl.infn.it/~epics/ L.Giovannini -- M.Giacchini

More Related