Two-Tier Adaptive Heap Management for Oracle WebLogic Server on the Oracle JRockit JVM

2. Two-Tier Adaptive Heap Management for Oracle WebLogic Server on the Oracle JRockit JVM RafiulAhad Adriano SantosVice President CMTS Product Development IT Oracle

3. Program Agenda • Background • Problem Definition • Shortcoming of Existing GC Policies • Overview of 2-Tier Adaptive Heap Management (AHM) • Implementation of 2-Tier AHM GC Policy • Some Results of Testing of 2-Tier AHM • Conclusions

4. Background • PDIT builds, provisions, and maintains the key infrastructure necessary for development, enterprise business, and cloud operations • We are the users of Oracle products and technologies • In this presentation we will discuss • An issue we face with JVM heap sizing for certain applications • Our solution approach for JRockit JVM • Results of testing the solution on a test application • This presentation does not imply that • Oracle recommends this approach for all aplications or • The technology will be implemented in JRockit or any JVM

5. Problem Definition • In an environment consisting of • Multiple JVMs running on a host with large CPU capacity • Each JVMruns memory-intensive applications whoseheap usage varies with some spikes • How do we Minimize • Out-of-Memory Errors (OOME) and • Swapping • While maximizing the number of JVMs or load on the host

6. Existing GC Policies' Shortcoming • Single policy for the whole heap • Minimize pause time at the expense of compaction • Slow heap compaction => higher probability of swapping when heap usage spikes • Or compact heap at the expense of pause time • Rarely used due to longer pause time • Consequences of minimize pause time policy • Too low max heap size => frequent OOME • Too high max heap size => swapping and/or fewer JVMs or load per host

7. Overview of 2-Tier AHM GC Policy • Heap is divided into two tiers • Normal Tier: from 0 to soft limit (SL) • Normal GC (e.g., genpar) used in normal tier • Special (Shared) Tier (Red Zone): from SL to –Xmx • AHM GC policy used in the red zone (after every normal old GC) favors heap compaction and return of freed memory to the host • When and how much memory to free is controlled by parameters • Thus memory can be released to host more quickly so that other JVMs can share this memory • Designed to accommodate heap usage spikes with little or no swapping • -Xmx has the same semantics • Any attempt to use more heap than –Xmx will raise OOME

8. Overview of 2-Tier AHM GC Policy Hard Limit (-Xmx) Special Tier (Red Zone) (AHM GC Policy Used) Soft Limit Committed Heap Size Free Heap Normal Tier (Existing GC Policy Used) Used Heap Size Used Heap Figure 1: Normal State, Existing GC Policy Used

9. Overview of 2-Tier AHM GC Policy Hard Limit (-Xmx) Special Tier (Red Zone) (AHM GC Policy Used) Committed Heap Size Free Heap Soft Limit Used Heap Size Normal Tier (Existing GC Policy Used) Used Heap Figure 2: Red Zone State, AHM GC Policy Used (in conjunction with Normal GC Policy)

10. 2-Tier AHM GC Policy Salient Parameters • MaxHeapMemorySoftLimit (SL) • The soft limit for the max heap memory • MinFreeHeapRatio • Min amount of heap expressed in % of SL that must be free under normal condition • min_free_heap_size = (MinFreeHeapRatio * SL) /100 • RedZoneReleaseThresholdRatio • Min % of red zone size that must be freed up by each heap compaction • min_compact_heap_size = (RedZoneReleaseThresholdRatio * red zone size) / 100 • HostFreeMemoryThreshold • A more aggressive action is taken if free memory at the host is less than this number • MaxAllowedGCsInRedZone • # of OLD GCs allowed in the red zone before JVM is considered for termination

11. 2-Tier AHM GC Policy After every old GC, if JVM’s committed heap size > SL do • compact the heap to max(SL,(used heap size + min_free_heap_size)) size if min_compact_heap_sizebytes can be freed and returned to the host • If free memory (including free-able IO buffers) on host is less than HostFreeMemoryThreshold then • If used memory < SL then • compact the heap to SL if heap has not been already been compacted • Else if number of GCs in red zone > MaxAllowedGCsInRedZone then • create ADR incident if configured • schedule graceful JVM termination

12. Benefits of AHM • Allows us to increase max heap size to avoid OOME when heap usage spikes due to unexpected load spikes or regular but short high loads • Aggressive heap compaction in red zone minimizes swapping and allows other JVMs on the same host to use the freed memory • Reduces effective committed heap size resulting in more JVMs or load

13. Implementation of 2-Tier AHM • As an MXBean deployed as an application • Use any MBean browser to • Define soft limit • Define AHM GC policy by setting appropriate values for MXBean attributes • Monitor Red Zone behavior • Many attribute values can be set via JVM start up parameters

14. AHMMXBean Attributes With Default Values (Stats and Some Info Attributes not Shown)

15. Implementation of 2-Tier AHM for Rockit JVM Observations on JRockit JVM heap usage behavior • When –Xmx is increased, committed heap size also increases for the same application (larger free heap) • JRockit JVM GarbageCollectorMBean provides: • setHeapSizeLock(true) to set the max heap size to committed heap size • setAllocatedHeapSizeTarget(x) to compact committed heap to size (x ± ∂) • Adjustments for JRockit heap allocation policy • Lock heap size at soft limit when committed heap size approaches soft limit to control committed heap growth • Unlock to expand beyond soft limit or to allow committed memory to go below soft limit

16. 2-Tier AHM Algorithm for JRockit after each Old GC do if (isHeapSizeLocked()) { //unlock heap if free heap is too small or too large if ((freeCommittedHeapSize < getMinFreeHeapSize()) || (freeCommittedHeapSize > getMaxFreeHeapSize())) unlockHeapSize(); } else { // heap size is not locked if(committedHeapSize > (softLimit - delta) doRedZoneAction(usedHeapSize, committedHeapSize, softLimit); else { resetRedZoneCounter(); if((usedHeapSize + getMinFreeHeapSize()) < softLimit) { if((committedHeapSize >= (softLimit - delta)) && (committedHeapSize <= (softLimit + delta))) setHeapSizeLock(true); //lock heap at committed size else if((committedHeapSize + getSoftLimitProximityThresholdSize()) > softLimit) setAllocatedHeapSizeTarget (softLimit); // we will lock near this in the next round } } }

17. 2-Tier AHM Algorithm for JRockit void doRedZoneAction (long usedHeapSize, long committedHeapSize, long softLimit) { long targetHeapSize = Math.max((usedHeapSize + getMinFreeHeapSize()), softLimit); booleanheapHasBeenShrunk = false; if((committedHeapSize - targetHeapSize) >= getReleaseHeapThresholdSize()) { setAllocatedHeapSizeTarget(targetHeapSize); heapHasBeenShrunk = true; } if(getHostFreeMemory() > getHostFreeMemoryThreshold()) return; // do nothing if there is enough host memory free if (usedHeapSize >= softLimit) if((numberOfGCsInRedZoneCounter >= getMaxAllowedGCsInRedZone()) && isForcedShutdownEnabled()) // Schedule a shutdown. We will check heap again before actual shutdown scheduleShutdown(); else // compact the heap to soft limit if it has not been shrunk yet if(!heapHasBeenShrunk) setAllocatedHeapSizeTarget (softLimit);

18. 2-Tier AHM Test • Test application runs a number of threads each consuming from 6MB to 10MB heap memory • Test environment is a single host with 8GB RAM running a WLS Admin Server and four Managed Servers • Each Managed Server • is started with –Xmx = 1812MB; 300MB used with no load; SL = 1450MB • runs 145 threads (heap usage varies from 870MB to 1450 MB) • Test is run for 30 minutes 4 times (No AHM, AHM) x (No Spike, Spike) • Observations: • Substantially lower total OC pause time with AHM due to less swapping • When –Xmx was reduced to 1450MB for no AHM, OOME occurred

19. Results of AHM Test (No Spike)

20. No Spike Swap Activity (vmstat) Comparison

21. Results of AHM Test (w/ 300MB Spikes)

22. 300MB Spikes Swap Activity Comparison

23. 2-Tier AHM Usage Scenarios • To minimize occasional OOME • Deploy AHMMXBean in WLS on JRockit JVM • Start WLS with 25% higher value for –Xmx • E.g., -Xmx = 2GB instead 1.6GB • Default attribute values are designed to minimize OOME and swap • To increase load where possible • Start with soft limit set to working set size • Set -Xmx quite high to avoid OOME • Adjust AHMMXBean attribute values accordingly • AHMMXBean attribute values can be set and monitored with Mission Control, EM, or any Mbean browser and used in WLDF Watch rules

24. Conclusions • We don’t need AHM if • H ≥ M * N where H is the host free RAM size, M is the maximum heap size a JVM needs, and N is the number of JVMs you want to run on the host • We could benefit from AHM if • H < M * N but H ≥ W * N where W < M is the working set heap size of JVM • The application may have load spikes raising the heap size to M • We want to avoid OOME and swapping (performance degradation) • Then, we set SL = W, -Xmx = M, and configure some swap space

25. Q&A