Caching: tuning your golden hammer

1. Caching: tuning your golden hammer By Ian Simpson from Mashery: an Intel Company

2. What’s being covered (or not) • Caching within the JVM • Not focusing on external caches like memcached • Not focusing on sharding • Both are important, but we only have an hour

3. Motivation for this talk • Overall, caching is great! • More performance! • Greater scalability! • When do we have too much of a good thing? • How do we know if/when we’re doing it wrong?

4. When is more not better? • Ultimately caching isn’t free (but is usually cheap) • Some caches are not effective (low cache/hit ratio) • Some caches hold on to too much data • Some caches cause resource contention • If used incorrectly, caches can destroy performance

5. Memory: abundant but not free • When we cache, we store data in memory • Generally a replacement for fetching data from (slow) persistent storage • Servers have lots of memory! • The JVM only handles so much memory gracefully* * This varies based on the garbage collection strategy and JVM vendor, which is beyond the scope of this talk

6. JVM heap • Split between young, tenured, and permanent • Objects start in young gen • Most objects in young gen are short lived and GC’d • Objects in young that survive GCs go to tenured • Objects in tenured are long lived (e.g. cached data)

7. Heap o’ trouble • Tenured grows as more objects survive • GCs cause “stop the world” events, or pauses • GCs on tenured get more costly the larger it gets • Long GC pauses can cause performance problems • Full heap can cause thrashing (lots of GCs)

8. Cache structure and data retention • Usually caches are maps of keys to values • Under the covers, often Map<K,V> • Anything you put in a Map is strongly referenced • Any reachable strong reference can’t be GC’d • Caches need to be reachable to be useful • Caches don’t automagically clean up stale data

9. How do we solve heap issues? • Don’t cache more than is necessary • Cache/hit ratio should be high (> 80%) • Understand how and when data is used • Estimate how large your data is or will be • Use background thread to clean stale data • Guava has support for this

10. Can I just not use the heap? • Yes! You can do off heap caching! • No! It’s not particularly straight forward! • On 32-bit JVM, 4GB (or 2) – max heap available • On 64-bit JVM, all available memory – max heap • Space is part of the Java process • Not managed by the garbage collector

11. What’s so difficult about off heap? • You have to manage your own memory • Allocated as a block via java.nio.ByteBuffer • Can also be allocated via sun.misc.Unsafe • You need to know where things are in the array • Serialization into and deserialization from byte[] • Cost to allocate block of memory

12. Blocking vs non-blocking cache

13. Blocking vs non-blocking cache • Cache miss means data has to be loaded • Loading data means blocking threads requesting it • Generally unavoidable unless returning nothing is OK • Cache hit on stale data means reloading • Blocking on reloading data can be optional • Return stale data, reload asynchronously

14. Blocking cache: pros and cons • Pros • Stale data is predictably stale • Less resource intensive than loading asynchronously • Cons • Can have huge latency on popular keys across threads • Read mutexes can result in serial read performance • Better with many keys and low key contention

15. Non-blocking cache: pros and cons • Pros • Can return stale data and load in the background • No read mutex required! • Cons • Requires a thread pool for asynchronous loading • Data is unpredictably stale • Better with fewer keys and high key contention

16. Which is which? What do I use? • EhCache is a blocking cache • Guava cache can be either • Neither is the best choice in all cases • Data characteristics will help you decide • Not sure what to use? Talk to your coworkers!

17. Example cases • Product catalog with millions of items • Blocking • Subset of products on the homepage • Non-blocking • Subset of products on sale for limited time • Questionable – contention and staleness both important • Split data on TTL, static vs dynamic • Refresh price/inventory frequently and asynchronously

18. Local vs distributed caching

19. Local vsdistributed caching • Local means each server has its own cache • Distributed means multiple servers share cache • Local is very simple • Distributed is very complex • Greater detail of distributed is outside scope

20. Local: pros and cons • Pros • Basically just a decorated Map • No serialization overhead • Good ones available for free • Cons • Less efficient: more cache misses and memory usage • In large clusters, cache/hit can suffer greatly • In clusters, consistency becomes a problem

21. Distributed: pros and cons • Pros • Fewer cache misses in clusters; improves with size • Consistency across cluster • Great for splitting up large data sets • Cons • Complex: more data management and network chatter • Serialization overhead for remote retrieval • Enterprise solutions cost $$$, free ones more barebones

22. Which one do I use? • Again, no one perfect solution; depends on data • Local is quick and dirty • Distributed is better for large data sets • Distributed is usually better for clusters • Distributed comes at a cost of complexity and $$$ • Not sure which one? Talk to your coworkers!

23. Example cases • CMS with large view templates • Distributed (size and consistency) • Display data like greetings, canned messages, etc • Local (small, changes infrequently) • Session store for user behavior • Questionable – consistency and resource concerns • Not mission critical data, but good to have • Local with sticky sessions an option

24. Getting closer with your data

25. Questions you should ask • How expensive is it to load? • Size: IO/memory bound; electrons only move so fast • Cost: high DB or CPU load means you must cache • Frequency: death by 1000 cuts • How often will I hit cache? • Example: static content often means high cache hit ratio • Example: highly diverse data means low cache hit ratio

26. Preparation through estimation • Estimate how data will grow • Number of new records per month? • How much is considered active? • Estimate how diverse data will become • Thousands of unique entries? Millions? • Subset of data that is more popular? • Work with product owners on expected use of data

27. And now for a recap!

28. A recap: fundamentals • Caching is great, but it’s not free • Get to know your data • Consider how much memory your app will use • Consider how expensive it is to load data • Take cache/hit ratios seriously • Not everything needs a cache

29. A recap: choices to make • Non-blocking: great for bounded data • Blocking: great for diverse data • Non-blocking: uses more resources, more stale data • Blocking: bad when keys have high contention • Local: simple, cheap, but can impact memory more • Distributed: great for big data sets & large clusters

30. Getting runtime info • Track cache statistics via JMX • Cache hits and misses • Total objects in memory • Tools like Nagios and AppDynamics can track JMX • Guava doesn’t have JMX, but it’s easy to add

31. BUT WHICH ONE DO I USE?! • Ultimately, you’ll have to figure it out • … with the help of your coworkers  • Before you can decide on what to use, you need to understand the characteristics of your data

32. Shameless self promotion • Website: http://www.theotherian.com • Has several cache and memory related posts • GithubGists: http://gist.github.com/theotherian • Lots of code samples (referenced in blog posts) • Interested in these problems? • Mashery is hiring! http://mashery.com/careers • API management for companies big and small • Scalability is core to our business

33. Obligatory “Questions” slide • Well?