Java Flight Recorder Behind the Scenes

2. Java Flight Recorder Behind the Scenes Staffan LarsenJava SE Serviceability Architect staffan.larsen@oracle.com @stalar

3. Java Flight Recorder Quick Glance • Tracer and Profiler • Non-intrusive • Built into the JVM itself • On-demand profiling • After-the-fact capture and analysis • First released in 7u40 Photograph by Jeffrey Milstein http://butdoesitfloat.com/In-the-presence-of-massive-gravitational-fields-this-perfect

4. Tracer and Profiler • Captures both JVM and application data • Garbage Collections • Synchronization • Compiler • CPU Usage • Exceptions • I/O • Sampling-based profiler • Very low overhead • Accurate data

5. Non-intrusive • Typical overhead in benchmarks: 3-5% (!) • Often not noticeable in typical production environments • Turn on and off in runtime • Information already available in the JVM • Zero extra cost Photograph: Andrew Rivett http://www.flickr.com/photos/veggiefrog/3435380297/

6. Built into the JVM itself • Core of JFR is inside the JVM • Can easily interact with other JVM subsystems • Optimized C++ code • Supporting functionality written in Java

7. On-demand profiling • Start from Java Mission Control • Or from the command line • Easily configure the amount of information to capture • For a profile, a higher overhead can be acceptable • When done, no overhead • Analyze the recording with a powerful GUI afterwards

8. After-the fact analysis • In its default mode, very low overhead • Designed to be always-on • Uses circular buffers to store data • In-memory or on-disk • When an SLA breach is detected, dump the current buffers • Dump will have information leading up to the problem

9. Enough Talking • Demo!

10. Agenda • Overview of JFR • Demo! • Configuration topics • Implementation details

11. Configuration • Enable -XX:+UnlockCommercialFeatures -XX:+FlightRecorder • Start -XX:StartFlightRecording=filename=<path>,duration=<time> • Alternate jcmd <pid> JFR.start filename=<path> duration=<time>

12. Advanced Configuration • Per recording • Maxage, maxsize • In-memory, on-disk • Globals • -XX:FlightRecorderOptions • Repository path • Max stacktrace length

13. Troubleshooting • -XX:FlightRecorderOptions=loglevel=[ERROR, WARNING, INFO, DEBUG, TRACE]

14. How is it built? High-level JMX Java Events • Information gathering • Instrumentation calls all over the JVM • Application information via Java API • Collected in Thread Local buffers • Moved to global buffers => to disk • Managed via JMX • Binary, proprietary file format JVM Compiler GC Runtime Threads JFR Global Disk

15. “Everything is an Event” Events • Event record • Event ID • Thread ID • Start Time • End Time • Stacktrace ID • Payload: int, long, double, string, class, thread,… • Event Description • Name, Description text, • For each payload item: name, type, description, content type Event ID Thread ID Start End Stacktrace ID Payload

16. Event types • Duration Events • Timing for something • Ex: GC • Instant Events • Single point in time • Ex: Thread starts • Requestable Events • Happens with a specified frequency • Ex: CPU Usage every second

17. Content Type • Describes the semantics of a value • Bytes • Percentage • Address • … • Used to correctly display the value in the UI

18. Self describing file format • Everything needed to parse an event is included in the file • If the VM adds a new event, there is no need to update the UI parser • Of course, the information will not be part of any of the specialized UIs, but the raw data is still there and can be analyzed and correlated • Single file, no dependencies … Header Events Event definitions

19. Event definition in Hotspot <event id="ThreadSleep" path="java/thread_sleep"label="Java Thread Sleep”...> <value field="time" type="MILLIS" label="Sleep Time"/></event> • XML definitions are processed into C++ classes

20. Event emission in Hotspot JVM_Sleep(int millis) { EventThreadSleep event; ... // actual sleep happens here event.set_time(millis); event.commit(); } • Done! Data is now available in JFR

21. Thread Park <event id="ThreadPark" path="java/thread_park” label="Java Thread Park”has_thread="true" has_stacktrace="true" is_instant="false"> <value type="CLASS" field="klass" label="Class Parked On"/> <value type="MILLIS" field="timeout" label="Park Timeout"/> <value type="ADDRESS" field="address" label="Address of Object Parked”/> </event>

22. UnsafePark(jbooleanisAbsolute, jlong time) EventThreadParkevent; JavaThreadParkedState jtps(thread, time != 0); thread->parker()->park(isAbsolute != 0, time); if (event.should_commit()) { oop obj = thread->current_park_blocker(); event.set_klass(obj ? obj->klass() : NULL); event.set_timeout(time); event.set_address(obj ? (TYPE_ADDRESS)obj : 0); event.commit(); }

23. Screenshot Event Graph

24. Screenshot Event Details

25. Buffers • (pretty picture of threads->global (pool)->disk) • “Circular” • Designed for low contention Thread Local Buffer Global Buffer Disk Chunk Repository Java JVM

26. Recordings • A recording can specify exactly which events to enable • XXX events with Y settings each • But: we ship two preconfigured settings • “default”: provides as much information as possible while keeping overhead to a minimum • “profile”: has more information, but also higher overhead • You can configure your own favorites in Mission Control

27. Filtering data • Enable/disable • Thresholds • Frequency

28. Many simultaneous recording sessions • This works great • Each session can have its own settings • Caveat: If there are multiple sessions all of them get the union of the enabled events • Ex: If event A is enabled in on recording, all recordings will see event A • Ex: If

29. Designing for long-running recordings in a dynamic runtime • Can’t leak memory • Can’t aggregate information eternally • Can’t keep references that prohibits class unloading • Dynamic Runtime • Classes can come and go • Threads can come and go • Solution: checkpoints, constant pools

30. Referencing classes, threads, stacktraces • To keep size of event small, these are referenced by an ID • “Constant pool” • Ex: First time a class is referenced, it is assigned an ID and tagged • When a checkpoint is created, information for all tagged classes is written to the file • Also, when a tagged class is unloaded, its information is written out

31. Problem: Many events reference classes • If every event contained the class name, we would waste lots of space • Solution: Use an ID number to refer to the class • Problem: How to look up the ID? • Solution: Have a lookup table in the recording. Event ID 17 class 4711 table.get(4711) -> “java.lang.Integer” count 2

32. When do we write the lookup table? • Problem: classes can be unloaded at any time • Can’t wait until the recording ends – class may not be around • Solution: write out IDs when classes are unloaded

33. Optimizing the size of the lookup table • Problem: many classes are loaded, not all are referenced in events, we want to save space • Solution: when a class is referenced, it is also “tagged” • Write only tagged classes to the lookup table

34. Leaking data • Problem: over time many classes will be tagged, the size of the lookup table will increase • Solution: reset the tags each time a lookup table is written to disk • A recording file may contain many lookup table, each one is only valid for the data immediately preceding it … data lookup data lookup

35. Checkpoints • At regular intervals, a “checkpoint” is created in the recording • Has all the metadata needed to parse the recording since the last checkpoint • Ex: An event references a class via a class id

36. Optimizations • Fast Timestamping • Stacktraces

37. Benchmarks • ?

38. More information • Whitepaper: • User Guide:

39. Differences vsJRockit • I/O: File paths, socket address • Exceptions • Reverse call trace view in MC • Deeper (configurable) stacktraces • Internal JVM differences: GC • Easier configuration in MC

40. Take-away -XX:+UnlockCommercialFeatures-XX:+FlightRecorder

41. Q&A @stalar staffan.larsen@oracle.com serviceability-dev@openjdk.java.net

42. The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract.It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.