Nectar: Efficient Management of Computation and Data in Data Centers

1. Nectar: Efficient Management of Computation and Data in Data Centers Lenin Ravindranath Pradeep Kumar Gunda, Chandu Thekkath, Yuan Yu, Li Zhuang

2. Motivation Resources are poorly managed in a data center Computation Storage • Redundant computations • Wasting resources Manually managed • Unused files occupying space • Redundant output files

3. Goal Efficiently manage resources in a cluster Nectar Computation Storage

4. Key Insight User Single query interface for computation and data access Query Interface DryadLINQ Data Center Computation Storage

5. Goal Efficiently manage resources in a cluster Nectar Computation Storage

6. Computation PROBLEM: Redundant Computation • Programs share sub queries • Programs share partial data sets SOLUTION: Caching • Cache results of popular sub queries • Automatically rewrite user query to use cache X.Select(…) X.Select(…).Where(…) X.Select(…) (X+X’).Select(…)

7. Does caching help? • Analyzed logs from production clusters • Logs of 3 months (Oct – Dec 2008) • 33 virtual clusters, 36000 jobs • Parsed SCOPE programs, extracted sub queries • Simulated caching

8. Caching helps • About 50% cache hit on 10 clusters • More than 30% cache hit on 20 clusters • 35% on average

9. Goal Efficiently manage resources in a cluster Nectar Computation Storage

10. Storage PROBLEM: Manually managed • Unused files occupying space 50% data was never accessed in the last 275 days

11. Storage SOLUTION: Automatically manage data • Track usage and delete infrequently used files • Store programs which re-computes the data

12. Query Interface User Query Interface DryadLINQ Data Center Computation Storage

13. Goal Efficiently manage resources in a cluster Nectar Computation Storage

14. Nectar User Query Interface Nectar DryadLINQ Data Center Computation Storage

15. Nectar Architecture DryadLINQ program P Query Nectar Client Cache Server Query Rewriter Cache entries Add T to cache Add R to cache P’ DryadLINQ Cluster T Dryad R

16. Nectar Architecture Nectar Client Cache Server Query Rewriter

17. Query Rewriter X X X’ X’ Select Select Select R R Concat Cache (R+R’)

18. Query Rewriter X X X’ X’ Select Select Select Order by Order by Order by R R Merge Sort Cache (R+R’)

19. Query Rewriter • Generates multiple plans • Using multiple cache entries • Selects the best plan • Based on benefit • Execution time • Output Size • Whether pipeline is broken • Operators supported • Select, Where, Order by, Group by, Join X.Select(…) X.Select(…).Where(…)

20. Nectar Architecture Nectar Client Cache Server Query Rewriter

21. Cache Server SQL Server Cache Policy Garbage Collector Cache Server Fingerprints

22. Cache policy • Insertion Policy • Always add program output to cache • Sub query outputs are added to cache • Popularity exceeds a threshold • Savings exceeds a threshold

23. Garbage Collector • Storage pressure • Delete infrequently used files • Deletion policy • Based on savings • Cache type • Mark and sweep algorithm • Delete cache entry • Reachability analysis • Delete files Cache Server Distributed FS

24. What if I try to access a garbage collected file?

25. Nectar Architecture Nectar Client Cache Server Query Rewriter Program store

26. Program Store • Store executed programs in the cluster • Output file is tied to its corresponding program that generates the output • If a file is deleted, the program is executed to regenerate the output

27. Managing Data P ToPartitionedTable (lenin\foo.pt) Nectar Client P’ Program Store Cache Server DryadLINQ Program Dryad Program Distributed FS A31E4.pt foo.pt FP usr Nectar

28. Managing Data P FromPartitionedTable (lenin\foo.pt) Nectar Client Program Store Cache Server DryadLINQ Dryad Distributed FS A31E4.pt foo.pt FP usr Nectar

29. Managing Data P FromPartitionedTable (lenin\foo.pt) Nectar Client Program Store Cache Server DryadLINQ Dryad Program Distributed FS A31E4.pt KJ1LM.pt foo.pt FP usr Nectar

30. Goal Efficiently manage resources in a cluster Nectar Computation Computation Storage Storage Unifiedcomputation and data

31. Distributed cache servers Program store Program store Cache Server Cache Server SQL Server SQL Server Partitioned by query fingerprint Centralized Garbage collector Nectar Client Hash based on query fingerprint

32. Summary • We built Nectar • Automatically manage data • Efficiently manage computation Components • Query Rewriter • Automatically rewrite queries to use cache • Cache server • Popular sub queries are cached • Garbage collected based on usage • Program store • Store programs which regenerates the output

33. Status • Almost done with development • Query Rewriter • Including other operators • Fingerprinter • Program static analysis • Cache Server • Program Store • In the process of deploying

34. Can we do better?

35. Cluster Utilization • Most clusters have more than 40% Idle time • Even the busiest clusters have 10-20% idle time

36. Exploiting idle time • Do speculative caching • Cache popular data before query issued • Run program on new streams when available • No side effects • Executed only when cluster is idle • Low priority jobs • Output garbage collected with high priority • More electric bill? Not Really!

37. Questions

38. Backup

39. Caching Results