1 / 20

An Intro. to Database Benchmarks

An Intro. to Database Benchmarks. June 2001 Prof. Sang Ho Lee Soongsil University shlee@computing.soongsil.ac.kr. Benchmarks: What and Why. To measure the performance of a hardware and software system in a controlled environment using a standard methodology

december
Download Presentation

An Intro. to Database Benchmarks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Intro. to Database Benchmarks June 2001 Prof. Sang Ho Lee Soongsil University shlee@computing.soongsil.ac.kr

  2. Benchmarks: What and Why • To measure the performance of a hardware and software system in a controlled environment using a standard methodology • Performance tests by which different database hardware/software platforms are compared in terms of price and performance • Motivated and driven by database customer demand (usually, to make acquisition decisions easy)

  3. Performance Assurance • Once you build the system you must do • Quality assurance: Functionality validation • Performance assurance: benchmark • find the performance bugs • looking for bottlenecks (queues) • When performance assurance is complete, model can be used to predict performance on higher loads, new hardware. This is called “capacity planning”

  4. Custom Benchmark vs. Generic Benchmark • Custom benchmark • Created by a particular customer based on a specific application • Can measure precisely capabilities that the customer wants • Very cost and hard to maintain • Generic benchmark • Created to be generally useful for a large group of potential customers with relatively standard applications across an application domain • Also known as domain-specific benchmark • Hybrid benchmark • generic benchmark + custom benchmark

  5. Popular Benchmarks • Wisconsin benchmark (1983) • TP1/DebitCredit benchmark (1985) • Transaction Processing Council formed (Aug. 1988) • TPC-A (Nov. 1989) • TPC-B (Aug. 1990) • AS3AP (1991) • SUN benchmark (001, object operations version benchmark) (1992) • TPC-C (Jul. 1992) • 007 Benchmark (1993) • Sequoia 2000 Storage Benchmark (1993) • TPC-D (Apr. 1995) • BUCKY (1997) • TPC-W (1998) • BORD (2000)

  6. How to Normalize Results • Difficult to compare the performance of the wide variety of database software/hardware platforms (from IBM mainframe to PC) • Current practice • Requires domain-specific qualifications that must be met • Example: ACID properties of transaction for OLTP benchmarks • Reports two specific measures • a measure of peak performance, say transactions per second (tps) • a measure of price/performance, say dollars per tps

  7. Scalar measures vs. Vector measures • Vector measures allow users to distinguish different performance effects in terms of functional origins, namely separability • Think about pros and cons !

  8. Key criteria for domain-specific benchmarks • Relevance: performs typical operations within the problem domain • Portability: easy to implement on many different systems and architectures • Scalability: applicable small and large computer systems • Simplicity: easily understandable otherwise it would lack credibility • Vendor neutrality: must not be biased in favor of a particular implementation

  9. Brief history on OLTP benchmarks • DebitCredit benchmark (1985) • Anon. et al., “A Measure of Transaction Processing Power”, Datamation • TP1 benchmark • an implementation by vendors that typically departs from the DebitCredit specifications one way or another • Aug. 1988: Transaction Processing Council formed • Originally eight vendors, currently 35 vendors joined • Nov. 1989: TPC-A, OLTP with LAN or WAN • Aug. 1990: TPC-B, OLTP with no network • Jul. 1992: TPC-C, On-Line Business Transaction Processing • Jun. 1995: TPC-A/B are frozen • Feb. 2001: TPC-C version 5.0 release

  10. Decision support system (DSS) benchmarks • Decision support system • relatively small number of users • mostly read-only queries, seldom update operation • Queries involve a great many data and indexes, thus use significantly more resources than OLTP transactions • data warehouse application • Wisconsin • AS3AP • Set Query • TPC-D benchmark – obsolete as of June. 1999. • Separated into TPC-R benchmark and TPC-H benchmark

  11. Engineering database benchmarks • Sun benchmark (001 benchmark) • W. Rubenstein, M. Kubicar and R. Cattle (ACM SIGMOD 1987) • R. Cattle, J. Skeen (ACM TODS 1992) • HyperModel Benchmark • T. Anderson, et al. (EDBT 1990) • 007 Benchmark • M. Carley, D. DeWitt, J. Naughton (ACM SIGMOD 1993) • A Complex Object Benchmark (ACOB) • D. DeWitt, P. Futtersack, D. Maier and F. Velez • VLDB conf. 1990

  12. How benchmarks are used: Good stories • Benchmark defines the fields • Accelerates progress • Designer can compare approaches • Users can compare approaches • Give performance agenda • Measures release-to-release progress • Set goals • Something managers can understand • Benchmark definition forms a community/vocabulary • Cross fertilization of people and ideas • Sharing of knowledge

  13. Research issues • What are good criteria for domain-specific benchmarks • Currently, relevance, portability, scalability, simplicity • What about scalar versus vector measures • How to incorporate actual costs of system ownership • Multi-user benchmark • How to achieve saparability of performance effects • How to develop an analytic framework • New applications or models • BLOBs, GIS, IR, Object-relational data model, etc.

  14. Comments on benchmarks (1) • Benchmarking is hard • Tools are a big help, but each experiment takes a day • Always do an atomic model first, check reality against the model • Keep careful notes • Change only one thing at a time (controlled experiments) • Use all the tools you can have: Good luck!

  15. Comments on benchmarks (2) • Watch out (unauthorized) vendor’s claims • Watch out benchmarketing • Benchmark result is certainly one thing, but not everything • Understanding what results say EXACTLY is very important • Must be aware of underlying platforms, networks and hardware, even though we are talking about DBMS (or OLTP) performance • Remember, vendor’s technologies and benchmark specifications continue to evolve over time

  16. References (1) • T. L. Anderson, A. J. Berre, M. Mallision, H. Poster, and B. Schneider, The HyperModel Benchmark, Proc. of the Int. Conference on Extending Database Technology, 1990. • Anon, et al., A Measure of Transaction Processing Power, Datamation, 1985. • D. Bitton, D. J. DeWitt, and C. Turbyfill, Benchmarking Database Systems: A Systematic Approach, Proc. of the Ninth Int. Conference on Very Large Data Bases, 1983. • D. Bitton and C. Turbyfill, A Retrospective on The Wisconsin Benchmark, In: Readings in Database Systems, M. Stonebraker ed., Morgan Kaufman, 1988. • M. J. Carey, D. J. DeWitt, and J. F. Naughton, The OO7 Benchmark, Proc. of the ACM SIGMOD Conference, 1993.

  17. References (2) • M. J. Carey, D. J. DeWitt, J. F. Naughton, M. Asgarian, P. Brown, J. E. gehrke, and D. N. Shah, The BUCKY Object-Relational Benchmark, Proc. of the ACM SIGMOD Conference, 1997. • M. J. Carey, D. J. DeWitt, C. Kant, and J. F. Naughton, A Status Report on the OO7 OODBMS Benchmarking Effort, Proc. of the ACM OOPSLA Conference, 1994. • R. G. G. Cattell, An Engineering Database Benchmark, In: The Benchmark Handbook for Database and Transaction Processing Systems 2nd ed., J. Gray ed., Morgan Kaufmann, 1993. • R. G. G. Cattell and J. Skeen, Object Operations benchmark, ACM Transactions on Database Systems, 1992. • D. J. DeWitt, The Wisconsin Benchmark: Past, Present, and Future, In: The Benchmark Handbook for Database and Transaction Processing Systems 2nd ed., J. Gray ed., Morgan Kaufmann, 1993.

  18. References (3) • J. M. Hellerstein and M. Stonebraker, Predicate Migration: Optimizing Queries with Expensive Predicates, Proc. of the ACM SIGMOD Conference, 1993. • W. Kim, Completeness Criteria for Object-Relational Database Systems, UniSQL Inc, 1996. • S. H. Lee, S. J. Kim, and W. Kim, The BORD Benchmark for Object-Relational Databases, Springer-Verlag Lecture Notes in Computer Science, 2000. • P. O’Neil, The Set Query Benchmark, In: The Benchmark Handbook for Database Transaction Processing Systems 2nd ed., J. Gray ed., Morgan Kaufmann, 1993. • P. O’Neil, Database Performance Measurement, In: The Computer Science and Engineering Handbook, A Tucker ed., CRC Press, 1997.

  19. References (4) • W. B. Rubenstein, M. S. Kubicar, and R. G. G. Cattell, Benchmarking Simple Database Operations, Proc. of the ACM SIGMOD Conference, 1987. • H.Schreiber, Justitia – A Generic Benchmark for the OODBMS Selection, Proc. of the Fourth International Conference of Data and Knowledge Systems for Manufacturing and Engineering, 1994. • K. Shanley, History and Overview of the TPC, Tranasaction Processing Performance Council, www.tpc.org, 1998. • M. Stonebraker, J. Frew, K. Gardels, and J. Meredith, The SEQUOIA 2000 Storage Benchmark, Proc. of the ACM SIGMOD Conference, 1993. • M. Stonebraker, Object-Relational DBMSs, Morgan Kaufmann, 1996.

  20. References (5) • Transaction Processing Performance Council, TPC Benchmark C standard specification version 5.0, www.tpc.org, 2001. • Transaction Processing Performance Council, TPC Benchmark H standard specification version 1.3.0, www.tpc.org, 1999. • Transaction Processing Performance Council, TPC Benchmark R standard specification version 1.2.0, www.tpc.org, 1999. • Transaction Processing Performance Council, TPC Benchmark W version 1.4, www.tpc.org, 2001. • C. Turbyfill, C. Orji, and D. Bitton, AS3AP: An ANSI SQL Standard Scaleable and Portable Benchmark for Relational Database Systems, In: The Benchmark Handbook second edition, J. Gray ed., Morgan Kaufmann, 1993.

More Related