1 / 61

Fly-by-Wire

Fly-by-Wire Instrument Mediated Performance Tuning Kirk Pepperdine kirk@kodewerk.com Fly-by-Wire A fly-by-wire system literally replaces physical control of the aircraft with an electrical interface. Wikipedia Computer mediated controls between the operator and the machine

Audrey
Download Presentation

Fly-by-Wire

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Fly-by-Wire Instrument Mediated Performance Tuning Kirk Pepperdine kirk@kodewerk.com

  2. Fly-by-Wire • A fly-by-wire system literally replaces physical control of the aircraft with an electrical interface. Wikipedia • Computer mediated controls between the operator and the machine • First attempted in test flight of F-8 (from NASA) • Pilot incapable of flying without assistance

  3. Fly-by-Wire F-8 (from NASA galleries)

  4. Reality vs. Senses • We learn to fly using sight • Licensed under Visual Flight Rules (VFR) • What happens when we can’t see? • Human senses are ill equipped for flight • Even when we can see we don’t always see things correctly • In order to see anything our brain fills in details

  5. Reality vs. Senses • Conflict • All of our senses tell us one thing • You believe you are straight and level • Instrumentation tells us something else • Your attitude is sending you toward the ground

  6. Fly-by-Wire My name is Bob and today I’ll be your pilot

  7. IFR Training Hoods

  8. Who I am • Kirk Pepperdine • Engaged around the world to solve Java performance problems • Co-author of http://www.javaperformancetuning.com • Associate Editor http://www.theserverside.com • Member of Sun Java Champions program

  9. Our Typical Story • Customer JoGoSlo Ltd calls us in desperation • Users complain about poor performance • Customers consider abandoning the project • 1 Million € invested • Management applying pressure • Developers • Can’t reproduce the problem in development • Recoding parts of the system hasn’t produced results • Common comment, “maybe we should do this” • Working under stress

  10. Effects of Stress • Learning and problem solving capabilities are impaired • During a stressful event, catecholamines suppress activity in areas at the front of the brain inhibiting the ability to handle complex social or intellectual tasks and behaviors • Why Zebras don’t get Ulcers (Robert M. Sapolsky) • In organizations you see “tribal” behaviours

  11. Effects of Stress • Skills needed are impaired by the situation • Tend to rely on what you already know • Pilots may revert to VFR • Risk of crashing • Developers look at code • Risk missing the real problem

  12. Senses

  13. Reality

  14. Senses public static boolean checkInteger(String testInteger) { try { //fails if not a number Integer theInteger = new Integer(testInteger); return (theInteger.toString() != "") && //not empty (theInteger.intValue() > 10) && ((theInteger.intValue() >= 2) && (theInteger.intValue() <= 100000)) && (theInteger.toString().charAt(0) == '3'); } catch (NumberFormatException err) { return false; } }

  15. Senses public static void testDataset(String dataset) throws IOException { DataInputStream rdr = new DataInputStream(dataset); long starttime = System.currentTimeMillis(); int truecount = 0; String s; try { while ((s = rdr.readUTF()) != null) if (checkInteger(s)) truecount++; } catch (EOFException e) {} rdr.close(); System.out.println(truecount + " (count); time + (System.currentTimeMillis() - starttime)); }

  16. Reality

  17. Reality vs Senses • Why isn’t code enough? • Developers work in a static medium • Performance tuners work in a dynamic medium that includes • Users • Application • Hardware • Need to consider dynamic nature of systems • Users

  18. Static View public static int[][] mul( int[][] a, int[][] b) { int rows = a.length; int columns = (a[ 0]).length; int[][] result = new int[rows][columns]; for ( int i = 0; i < rows; i++) for ( int j = 0; j < columns; j++) result[i][j] = a[i][j] * b[j][i]; return result; }

  19. Dynamic View public static void main(String[] args) { int[][] a = { {1, 1}, {1, 1}, {1, 1}, {1, 1}, {1, 1}, {1, 1}, {1, 1}, {1, 1}}; int[][] b = { {1, 2, 3, 4, 5, 6, 7, 8}, {8, 7, 6, 5, 4, 3, 2, 1}}; long start = System.currentTimeMillis(); for ( int i = 0; i < 50000; i++) mul( b, a); System.out.println(System.currentTimeMillis() - start); start = System.currentTimeMillis(); for ( int i = 0; i < 50000; i++) mul( a, b); System.out.println(System.currentTimeMillis() - start); }

  20. Dynamic Observation

  21. Dynamic Observation • Response time is king • Defines the user experience • 130ms vs 300ms • Same code • Same amount of data • Why the difference? • What are the elements of a system?

  22. Elements of a System

  23. Elements of a System • Each layer contributes resources • Each layer defines whole classes of performance problems • Change a layer, change the problem • This include people

  24. Hardware • Simply may not have enough • Buy more • May not be able to utilize more may • Utilization measured using • Unix : vmstat, iostat, mpstat, netstat, top • Windows : perfmon, taskmanager

  25. Hardware - CPU • CPU consumption reaches 100% for extended periods of time • Real values are often 40-60% • Measure using • vmstat in Unix • Taskmanager in Windows

  26. Hardware - CPU • Two different types of consumption • Application (user) • Operating System (system) • Need to be considered separately • Directs tuning efforts

  27. Hardware – CPU • High system utilization • Tune OS • Application usage of system calls • Locks, frequent I/O, excessive context switching • High user utilization • Focus on the application (or JVM) • Not necessarily bad • Use response times as a guide • Consider peaks in utilization (capacity)

  28. Hardware - Memory • System starved of memory • Will cause high system utilization • May appear to be an disk I/O bottleneck • Measured using • vmstat, swap, and scan rate columns • perfmon to report on counters • Best solution is buy more memory

  29. Hardware – I/O • Symptoms • Low CPU utilization • Poor response times • Cannot boost CPU utilization • Can be measured with • iostat or netstat • Perfmon (network or disk counters) • Reported as absolute numbers • Need to know hardware supported rates

  30. Hardware • If CPU, memory, disk and network I/O are all normal move to the JVM

  31. JVM • Java memory is contained in Java heap • Managed by separate threads • Java process starved for memory will appear to be CPU bound • Large heap can cause lots of OS page swapping • Measure using –verbose:gc flag • Analyize output with GcViewer (Tagtram)

  32. JVM Memory • Garbage collector (GC) manages Java heap memory • Look for GC efficiency • Defined as the portion of time spent managing memory over the run time duration • Values > 10% imply GC bottleneck • Tune generational spaces • Reduce memory utilization

  33. Heaps of Heap 1.330: [GC 1232K->755K(1984K), 0.0016326 secs] 1.377: [GC 1264K->808K(1984K), 0.0017882 secs] 1.448: [GC 1319K->905K(1984K), 0.0020653 secs] 1.595: [GC 1417K->919K(1984K), 0.0019377 secs] 1.682: [GC 1431K->1027K(1984K), 0.0029697 secs] 2.457: [GC 1538K->1104K(1984K), 0.0033163 secs] 2.583: [GC 1616K->1248K(1984K), 0.0034569 secs] 2.745: [GC 1759K->1355K(1984K), 0.0020179 secs] 2.861: [GC 1866K->1462K(1984K), 0.0021721 secs] 2.964: [GC 1963K->1488K(2112K), 0.0019511 secs] 2.967: [Full GC 1488K->1190K(2112K), 0.0472200 secs]

  34. Heaps of Heap 1.330: [GC 1232K->755K(1984K), 0.0016326 secs] 1.377: [GC 1264K->808K(1984K), 0.0017882 secs] 1.448: [GC 1319K->905K(1984K), 0.0020653 secs] 1.595: [Full GC 1488K->1190K(2112K), 0.0472200 secs] • Runtime = 1595 – 1330 = 265 • GC Time = 1.6326 + 1.7882 + 2.0653 + 47.22 = 52.7061 • GC efficiency = 52.7061 / 265 = 19.8%

  35. GcViewer

  36. JVM Threads • Too many threads • High system utilizations • High cpu utilization with slow progression • Threads also stress memory • Too few threads and processes will starve for CPU • Measuring using ^c in the console window • Dumps state of all threads

  37. Webserver Thread Dump Full thread dump Java HotSpot(TM) Server VM (1.4.2_08-b03 mixed mode): "RMI ConnectionExpiration-[192.168.0.15:34113]" daemon prio=1 tid=0x0892f658 nid=0x2d7a waiting on condition [5b86f000..5b86f494] at java.lang.Thread.sleep(Native Method) at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:447) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(902)-192.168.0.15" daemon prio=1 tid=0x41e112b8 nid=0x2d7a runnable [5ccff000..5ccff414] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) at java.io.BufferedInputStream.read(BufferedInputStream.java:201) - locked <0x49978800> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(FilterInputStream.java:66) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport) at java.lang.Thread.run(Thread.java:534)

  38. Webserver Thread Dump "PingThread-8692809" daemon prio=1 tid=0x081a3058 nid=0x2d7a waiting on condition [5bda9000..5bda9294] at java.lang.Thread.sleep(Native Method) at org.exolab.jms.client.rmi.RmiJmsConnectionStub$PingThread.run(RmiJmsConnectionStub.java:249) "EventManagerThread" daemon prio=1 tid=0x083101f8 nid=0x2d7a in Object.wait() [5caa9000..5caa9514] at java.lang.Object.wait(Native Method) - waiting on <0x47aa2800> (a java.lang.Object) at java.lang.Object.wait(Object.java:429) at org.exolab.jms.events.BasicEventManager.run(BasicEventManager.java) - locked <0x47aa2800> (a java.lang.Object) at java.lang.Thread.run(Thread.java:534) "PingThread-18183604" daemon prio=1 tid=0x08136dc8 nid=0x2d7a waiting on condition [5cc7f000..5cc7f494] at java.lang.Thread.sleep(Native Method) at org.exolab.jms.client.rmi.RmiJmsConnectionStub$PingThread.run( RmiJmsConnectionStub.java:249)

  39. Webserver Thread Dump "RMI RenewClean-[192.168.0.15:34113]" daemon prio=1 tid=0x081a2c68 nid=0x2d7a in Object.wait() [5ba10000..5ba10594] at java.lang.Object.wait(Native Method) - waiting on <0x4858a940> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111) - locked <0x4858a940> (a java.lang.ref.ReferenceQueue$Lock) at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run( DGCClient.java:500) at java.lang.Thread.run(Thread.java:534) "BoundedThreadPool0-33" prio=1 tid=0x41ec8710 nid=0x2d7a in Object.wait() [5ca29000..5ca29594] at java.lang.Object.wait(Native Method) - waiting on <0x47a5> (a org.mortbay.thread.BoundedThreadPool$PoolThread) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool) - locked <0x47a520b8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

  40. Webserver Thread Dump "BoundedThreadPool0-32" prio=1 tid=0x5a159ed0 nid=0x2d7a runnable [5c9a9000..5c9a9614] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:99) at org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:257) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:192) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:293) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool) - locked <0x47a521f8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

  41. Webserver Thread Dump "BoundedThreadPool0-31" prio=1 tid=0x5ad60d78 nid=0x2d7a runnable [5c929000..5c929694] at org.mortbay.jetty.HttpGenerator.prepareBuffers(HttpGenerator.java:878) at org.mortbay.jetty.HttpGenerator.flushBuffers(HttpGenerator.java:681) at org.mortbay.jetty.HttpGenerator.complete(HttpGenerator.java:671) at org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java:388) at org.mortbay.jetty.HttpConnection.access$1500(HttpConnection.java:38) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete( HttpConnection.java:598) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:487) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:196) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:293) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool) - locked <0x47a52158> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

  42. Webserver Thread Dump "BoundedThreadPool0-30" prio=1 tid=0x41e2f878 nid=0x2d7a in Object.wait() [5c8a9000..5c8a9714] at java.lang.Object.wait(Native Method) - waiting on <0x47a5> (a org.mortbay.thread.BoundedThreadPool$PoolThread) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool) - locked <0x47a52298> (a org.mortbay.thread.BoundedThreadPool$PoolThread) "BoundedThreadPool0-29" prio=1 tid=0x5a4c5650 nid=0x2d7a in Object.wait() [5c828000..5c828794] at java.lang.Object.wait(Native Method) - waiting on <0x47a5> (a org.mortbay.thread.BoundedThreadPool$PoolThread) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool) - locked <0x47a52108> (a org.mortbay.thread.BoundedThreadPool$PoolThread) "BoundedThreadPool0-28" prio=1 tid=0x5a4c53f8 nid=0x2d7a runnable [5c7a8000..5c7a8814] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:99) at org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:257) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:192) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:293) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector)

  43. Webserver Thread Dump "BoundedThreadPool0-27" prio=1 tid=0x41e6a640 nid=0x2d7a in Object.wait() [5c728000..5c728894] at java.lang.Object.wait(Native Method) - waiting on <0x47a5> (a org.mortbay.thread.BoundedThreadPool$PoolThread) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool) - locked <0x47a52338> (a org.mortbay.thread.BoundedThreadPool$PoolThread) "BoundedThreadPool0-26" prio=1 tid=0x41e693f8 nid=0x2d7a in Object.wait() [5c6a8000..5c6a8914] at java.lang.Object.wait(Native Method) - waiting on <0x47a5> (a org.mortbay.thread.BoundedThreadPool$PoolThread) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool) - locked <0x47a52248> (a org.mortbay.thread.BoundedThreadPool$PoolThread) "BoundedThreadPool0-25" prio=1 tid=0x086b1c50 nid=0x2d7a in Object.wait() [5c628000..5c628994] at java.lang.Object.wait(Native Method) - waiting on <0x47a4> (a org.mortbay.thread.BoundedThreadPool$PoolThread) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool) - locked <0x47a4e180> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

  44. Webserver Thread Dump • According to client, system was idle • Did not accept any more connection requests • Let’s go back a few slides… • Why was HttpGenerator.prepareBuffers() being called?

  45. Webserver Thread Dump "BoundedThreadPool0-31" prio=1 tid=0x5ad60d78 nid=0x2d7a runnable [5c929000..5c929694] at org.mortbay.jetty.HttpGenerator.prepareBuffers(HttpGenerator.java:878) at org.mortbay.jetty.HttpGenerator.flushBuffers(HttpGenerator.java:681) at org.mortbay.jetty.HttpGenerator.complete(HttpGenerator.java:671) at org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java:388) at org.mortbay.jetty.HttpConnection.access$1500(HttpConnection.java:38) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete( HttpConnection.java:598) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:487) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:196) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:293) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool) - locked <0x47a52158> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

  46. JVM Threads • Use thread pooling to control over-utilization • Acts to calm traffic • Smoothen traffic reduces thrashing • Reduced thrashing increases throughput • Should reduce response times

  47. Application • Applications • Interact with external systems • Add locks to protect software resources • The only things to consider are lock contention and interactions with external systems

  48. Application • Measure using ^c in the console window • Lock contention • Threads waiting on monitors • Waits for external systems • Almost all communications with external systems occur via sockets • Threads waiting on socket read/write

  49. Application • Most problematic external system • Relational databases • Monitor with intercepting technology • P6spy acts like a jdbc driver • Logs interactions • Logs can be viewed with Ironeye (source forge)

  50. Ironeye

More Related