Introduction Monitor agent / Monitor viewer

1. Introduction Monitor agent / Monitor viewer By Kåre Jonsson Date: 2007-dec-17

2. Main concepts • One agent program in an environment where something can be measured, probably close. • One central viewer program for reporting. • Agent is extreamley sparse on resources. • Viewer not adding load to production.

3. Architecture and interaction SMTP HTTP url download Tomcat HTTP Monitor agent Monitor viewer File system Database file *.rrd Java Runtime API

4. Cluster Monitor agent HTTP HTTP Tomcat Monitor viewer HTTP HTTP Monitor agent Monitor agent

5. Process view SAF Viewer OS OS OS OS

6. Monitor agent Monitor agent Tomcat Monitor agent Monitor viewer Monitor agent Monitor agent Monitor agent Monitor agent Monitor agent

7. Two separated cluster deployments Monitor agent Tomcat Monitor viewer Monitor agent HTTP HTTP Tomcat Monitor agent Monitor viewer HTTP HTTP Monitor agent Monitor agent Monitor agent Monitor agent Monitor agent

8. Use extra node for viewers Optionally deployed outside the production nodes Monitor agent Tomcat Monitor viewer Monitor agent HTTP HTTP Tomcat Monitor agent Monitor viewer HTTP HTTP Monitor agent Monitor agent Monitor agent Monitor agent Monitor agent

9. Two cluster deployments with agent reuse Monitor agent Tomcat Monitor viewer Monitor agent HTTP HTTP Tomcat Monitor agent Monitor viewer HTTP HTTP Monitor agent Monitor agent Monitor agent Monitor agent Monitor agent

10. Questions Viewer asks Agent. Protocol. • What types/instances? KERNEL, DISK, NETWORK... • What measurements? (kernel: contextswitch, idletime, systemtime ...) (network: tcpError..., ..., ...) • Give me the .rrd file of ... • What is your internal RAM log? (Diagnosis)

11. Process view customized case Viewer SAF OS OS Custom Viewer OS OS Custom Custom

12. Viewer Agent Tomcat servlet Monitor viewer Monitor agent kernel Executables Closed Opened Predefined Cluster view Configuration Scripts for platform Standard Customized Customized scripts Customized Cluster view Optional customization On demand post processors On demand scripts Deployment considerations TCP ports Instances monitoragents.xml

13. Development Make measurement script Add to Monitor agent ring Presentation

14. Write measurement script • Find command or file or download that contains parseable data in which the value appears. • Find a systematic way to extract it. • Decide how often to measure. • Write the Monitor agent script to register the values. Here an example with the context switch (Solaris). D – Duty S – Solution

15. Find command D: Find command or file or download that contains parseable data in which the value appears. S: The command ”vmstat 5 2” $> vmstat 5 2 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s1 s3 s4 in sy cs us sy id 0 0 0 18847200 7733416 8 36 29 0 0 0 11 0 3 0 0 328 316 642 0 1 99 0 0 0 18995744 7672152 0 7 0 0 0 0 0 0 0 0 0 325 220 657 0 0 99 $>

16. Find systmetic way D: Find a systematic way to extract it. S: It is the third lines 18th fragment (some from previous page is copied to here) $> vmstat 5 2 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s1 s3 s4 in sy cs us sy id 0 0 0 18847200 7733416 8 36 29 0 0 0 11 0 3 0 0 328 316 642 0 1 99 0 0 0 18995744 7672152 0 7 0 0 0 0 0 0 0 0 0 325 220 657 0 0 99 $>

17. Decide periodicity D: Decide periodicity. How often to measure. S: An example is once every 2 minutes. That is 120 seconds. In general this is depending on what is measured.

18. Write Monitor agent script D: Write the Monitor agent script to register values. S: The following is the contents of the file contextswitch.mtr <PERMANENT> <SCRIPT> { } </SCRIPT> </PERMANENT> <ROUTINE> <PERIODICITY>120</PERIODICITY> <SCRIPT> { command("vmstat 5 2"); register(“ContextSwitch”, value(fragment(3, 18))); } </SCRIPT> </ ROUTINE >

19. Deploying Execute the script $> monitoragent –i=X contextswitch.mtr Put many scripts on the commandline. The –i=X is omitted in this session. One script can produce any number of measurements. Not just one as in this example.

20. Monitor agent ”Processes” . . . .mtr .mtr .mtr .mtr .mtr .mtr Monitor agent Java Processor/OS

21. Extend Monitor agent ring Add the ”-i=X contextswitch.mtr” on your Monitor agent invokation line. Any number of ”-i=X Y.mtr” is allowed. Have the following cluster questions in mind • Will the Monitor agent process fail over? • Should the routine fail over like the Monitor agent? • Should the routine execute in parallell on the nodes?

22. Customization Monitoring Agent / Viewer To do To read

23. To do • Add measurement routines • Scripts for measurements (.sh, .ksh, .pl, .py) • Scripts for on demand contents (.sh, .ksh, .pl, .py) • Measurement routines (.mtr) • Integrate • Presentation • Map symbols in map.xml • Graph compositions • On demand view • Integrate

24. Absolute winning concepts • Optimal deployment. • Optimal behaviour. • Extremely low memory consumption • Extremely low disk usage • No spontaneous network activity • Almost always ”sleeping” • Configured in scripting language programming. • Mainly builds on command line invocations. • Type references. Monitor viewer asks for instance names. Cluster configuration remains independent of deployment.

25. More winning concepts • Open format. .rrd-files of JRobin at www.jrobin.org. • Viewer extendible with curve compositions. • Agent extendible. Add –i=X X.mtr as suitable. • Command line style of the invocation of the processes, i.e. –p=... –debug=... –confdir=... –datadir=... -ondemand • No spontaneous activity • CGI like functionality

26. All about Monitor agent configuration What one can do in Monitor agent scripting is in monitoragent_devguide.pdf To program in Monitor agent scripting is to configure the same!

27. All about Monitor viewer configuration What one can do in Monitor viewer is in monitorviewer_devguide.pdf

28. Memory allocation

29. Average for some measurements

30. Many measurements from one agent

31. Idle time for all nodes

32. Full screen 1 3 2

33. Upper left frame