1 / 47

MY DATABASE SYSTEM IS THE ONLY THING I CAN

MY DATABASE SYSTEM IS THE ONLY THING I CAN. TRUST. @ ANDY_PAVLO. Thirty Years Ago…. I NTERACTIVE T ransactions. S mall # of CPU C ores. S mall M emory S izes. Voter Setup. TP C-C BENCHMARK. Warehouse Order Processing. Application. Transaction:. NewOrder Transaction.

amadahy-mitchell
Download Presentation

MY DATABASE SYSTEM IS THE ONLY THING I CAN

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MY DATABASE SYSTEM ISTHE ONLY THING I CAN TRUST @ANDY_PAVLO

  2. Thirty Years Ago…

  3. INTERACTIVE Transactions Small # of CPUCores Small MemorySizes

  4. Voter Setup TPC-C BENCHMARK Warehouse Order Processing Application Transaction: NewOrder Transaction Check whether user has already voted. Insert new vote. Update vote count for contestant. Check item stock level. Create new order information. Update item stock levels.

  5. Voter MySQL/Postgres TPC-C BENCHMARK Warehouse Order Processing MySQL Postgres txn/sec CPU CORES

  6. CPU Overhead TRADITIONAL DBMS Measured CPU Cycles 30% 28% 30% 12% OLTP Through the Looking Glass, and What We Found There SIGMOD, pp. 981-992, 2008.

  7. Hardware Upgrade Replication Distributed Cache ShardingMiddleWare NoSQL

  8. HOW TO SCALE UP WITHOUT GIVING UP TRANSACTIONS?

  9. Distributed Main Memory Transaction Processing System H-Store: A High-Performance, DistributedMain Memory Transaction Processing SystemProc. VLDB Endow., vol. 1, iss. 2, pp. 1496-1499, 2008.

  10. x Disk Oriented i Main Memory Storage Concurrent Execution Serial Execution / Heavyweight Recovery CompactLogging

  11. H-Store Architecture Diagram Stored Procedure Partitions Transaction Execution VoteCount: InsertVote: Procedure Name Input Parameters Transaction Result INSERT INTO votes VALUES (?, ?, ?); SELECT COUNT(*) FROM votes WHEREphone_num = ?; Application run(phoneNum, contestantId, currentTime) { result = execute(VoteCount, phoneNum); if (result > MAX_VOTES) { return (ERROR); } execute(InsertVote, phoneNum, contestantId, currentTime); return (SUCCESS); } Single-Threaded Execution Engines CMdLog Snapshots

  12. Voter MySQL/Postgres TPC-C BENCHMARK Warehouse Order Processing MySQL Postgres H-Store 40x txn/sec CPU CORES

  13. Distributed Transactions

  14. Distributed TPC-C TPC-C BENCHMARK 8 Cores per Node 10% Distributed Transactions H-Store txn/sec Nodes

  15. Distributed Txn Example DISTRIBUTED TRANSACTIONS Query Count P1 P2 P3 P4 Application

  16. KNOW WHAT TRANSACTIONSWILL DO BEFORE THEY START

  17. BUT PEOPLE ALWAYS GIVE ME BAD ADVICE

  18. DON’T GET INVOLVED WITH COMPUTERS. YOU’LL NEVER MAKE ANY MONEY.

  19. DON’T GET A PHD. EVERYONE WILL THINK YOU ARE A JERK.

  20. THE DATABASE SYSTEMALWAYS HAS MORE INFORMATION

  21. DO USE MACHINE LEARNINGTO PREDICTTRANSACTION BEHAVIOR. On predictive modeling for optimizing Transaction execution in parallel oltp systems Proc. VLDB Endow., Vol 5, Iss. 2, pp. 85-96, 2011

  22. Houdini Overview PREDICTIVE MODELS

  23. Creating Models Model Generator Feature Clusterer Classifier WORKLOAD SELECT * FROM WAREHOUSEWHERE W_ID = 10; SELECT * FROM DISTRICT WHERE D_W_ID = 10 AND D_ID =9; INSERT INTO ORDERS (O_W_ID, O_D_ID, O_C_ID,…) VALUES(10, 9, 12345,…); ⋮ ______________________________________________________ ______________________________________________________ SELECT * FROM WAREHOUSE WHERE W_ID = 10; SELECT * FROM DISTRICT D_W_ID = 10 AND D_ID =9; INSERT INTO ORDERS (O_W_ID, O_D_ID, O_C_ID) VALUES (10, 9, 12345); ⋮ SELECT * FROM WAREHOUSE WHERE W_ID = 10; INSERT INTO ORDERS (O_W_ID, O_D_ID, O_C_ID) VALUES (10, 9, 12345); ⋮ SELECT * FROM WAREHOUSE WHERE W_ID = 10; INSERT INTO ORDERS (O_W_ID, O_D_ID, O_C_ID) VALUES (10, 9, 12345); ⋮ ______________________________________________________ ______________________________________________________ Decision Tree Feature1 Markov Models Feature2 Feature2

  24. Houdini Example DISTRIBUTED TRANSACTIONS Application

  25. Houdini TPC-C TPC-C BENCHMARK 8 Cores per Node 10% Distributed Transactions Naïve Houdini Optimal 2x txn/sec Nodes

  26. Houdini Example DISTRIBUTED TRANSACTIONS Application Zzzz… Zzzz… Zzzz… Zzzz… SP1- Waiting for Query Result SP2- Waiting for Query Request SP3- Two-Phase Commit

  27. TRANSACTION STALL POINTS Base Partition Remote Partition 18% 5% 73% 45% 37% 22% SP1- Waiting for Query Result SP3- Two-Phase Commit SP2- Waiting for Query Request Real Work

  28. DO SOMETHING USEFULWHEN STALLED

  29. DON’T BE SURPRISED IF YOU & KB DON’T LAST THROUGH GRAD SCHOOL.

  30. DON’T BE STAN’S STUDENT IF YOU GO TOBROWN.

  31. DO USE MACHINE LEARNINGTO SCHEDULESPECULATIVE TASKS. The Art of Speculative Execution In Progress (August 2013)

  32. Serializable Schedule SERIALIZABLE SCHEDULE Distributed Transaction Verify C Zzzz… Single-Partition Transaction Speculative Transaction C C Speculative Transaction Single-Partition Transaction C C

  33. Hermes Overview SPECULATIVE TRANSACTIONS … Transaction Queue Speculation Candidate: Zzzz… WRITE X Distributed Transaction: READ X READ X

  34. Hermes Overview SPECULATIVE QUERIES QueryY: SELECT S_QTY FROM STOCK WHERE S_W_ID = ? AND S_I_ID = ?; Distributed Transaction:

  35. Mapping: Example Transaction Parameters: w_id=0 i_w_ids=[1,0] i_ids=[1001,1002] GetWarehouse: CheckStock: SELECT * FROM WAREHOUSE WHERE W_ID = ? SELECT S_QTY FROM STOCK WHERE S_W_ID = ? AND S_I_ID = ?;

  36. Hermes Overview VERIFICATION Distributed Transaction Speculative Transactions Query1 Query2 Query3 Query1 Query3 Query3 Query3 Query1 Query2 Query3

  37. Hermes TPC-C TPC-C BENCHMARK 8 Cores per Node 10% Distributed Transactions Spec Queries None All Spec Txns txn/sec Nodes

  38. Optimize Single-Partition Execution H-Store: A High-Performance, Distributed Main Memory Transaction Processing System Proc. VLDB Endow., vol. 1, iss. 2, pp. 1496-1499, 2008. Minimize Distributed Transactions Skew-Aware Automatic Database Partitioning in Shared-Nothing, Parallel OLTP Systems Proceedings of SIGMOD, 2012. Identify Distributed Transactions On Predictive Modeling for Optimizing Transaction Execution in Parallel OLTP Systems Proc. VLDB Endow., vol. 5, pp. 85-96, 2011. Utilize Transaction Stalls The Art of Speculative Execution In Progress (August 2013)

  39. Future Work

  40. One Size Almost Fits All

  41. N H-STORE S-STORE N-STORE

  42. One Size Almost Fits All ™

  43. N H-STORE S-STORE N-STORE

  44. Escape From Planet Zdonik (i.e., Andy Needs to Get Tenure)

  45. Beyond the ‘Stores • Non-Partitionable Workloads. • The Poor Man’s Spanner. • Scientific Databases.

  46. DON’T MESS IT UP WITH KB.

  47. StanZdonik “The Thrill”Stonebraker Sam Madden Ugur Cetintemel DavidDeWitt DanAbadi EvanJones Saurya Velagapudi Xin Jia CarloCurino JustinDeBrabant YangZou Visawee Angkana. NingShi John Meehan

More Related