1 / 46

(Ab-) using Round Robin Databases with Perl

(Ab-) using Round Robin Databases with Perl. Mike Schilli, Yahoo! 06/18/2008. Overview. What are Round Robin Databases? RRD and Perl – which module to use? Practical Examples. Round Robin Databases. Archives of fixed size for unlimited data Overwrite old spots if full.

Jims
Download Presentation

(Ab-) using Round Robin Databases with Perl

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. (Ab-) using Round Robin Databases with Perl Mike Schilli, Yahoo! 06/18/2008

  2. Overview • What are Round Robin Databases? • RRD and Perl – which module to use? • Practical Examples

  3. Round Robin Databases • Archives of fixed size for unlimited data • Overwrite old spots if full

  4. Round Robin Archives start

  5. Round Robin Archives • Overwrite if full

  6. Round Robin Archives • RRDs with multiple archives with different granularity

  7. Data Flow with rrdtool Data Source Archive

  8. Data Source Conversions

  9. Steps and Heartbeat

  10. Steps and Heartbeat

  11. Steps and Heartbeat

  12. Generous Heartbeat

  13. Generous Heartbeat

  14. Steps and Heartbeat • Long story short: Heartbeat > Step if you’re roughly feeding in “step” intervals. • Example: • Step=300 (5 min) • Heartbeat=600

  15. Feeding PDPs to the Archive

  16. Feeding PDPs to the Archive • Consolidation Functions (MAX, AVERAGE, etc.) • xfiles factor (allowed percentage of missing values)

  17. Simple Example Web Orders: 0:00 100 1:00 110 2:00 120 3:00 130 4:00 140 5:00 150

  18. Simple Example

  19. Simple Example rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' '1211875200:100' rrdtool 'update' 'myrrdfile.rrd' '1211878800:110' rrdtool 'update' 'myrrdfile.rrd' '1211882400:120' rrdtool 'update' 'myrrdfile.rrd' '1211886000:130' rrdtool 'update' 'myrrdfile.rrd' '1211889600:140' rrdtool 'update' 'myrrdfile.rrd' '1211893200:150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

  20. RRDTool • Arcane syntax • Perl API RRDs.pm • Pretty close to RRDTool syntax • Calls rrdtool under the hood

  21. To the Rescue: RRDTool::OO • CPAN Module RRDTool::OO • Takes the drudgery out of the RRD syntax • Does what you want by default • Translates between Perl and RRD (watch it with debug on) • Named parameters instead of fixed order

  22. Simple Example #!/usr/local/bin/perl -w use strict; use RRDTool::OO; use DateTime; my $rrd = RRDTool::OO->new( file => "myrrdfile.rrd" );

  23. Simple Example my $start_dt = DateTime->today( time_zone => "local" ); my $dt = $start_dt->clone();

  24. Simple Example $rrd->create( step => 3600, start => $start_dt->epoch(), data_source => { name => "orders", type => "GAUGE", }, archive => { rows => 100 } );

  25. Simple Example for my $hour (0..5) { $dt->add(hours => 1); $rrd->update( time => $dt->epoch(), value => 100 + $hour*10, ); }

  26. Simple Example $rrd->graph( image => "mygraph.png", vertical_label => 'Orders per Hour', start => $start_dt->epoch(), end => $dt->epoch(), draw => { type => "area", color => '0000FF', legend => "Web Orders", } );

  27. Simple Example

  28. RRDTool Commands rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' '1211875200:100' rrdtool 'update' 'myrrdfile.rrd' '1211878800:110' rrdtool 'update' 'myrrdfile.rrd' '1211882400:120' rrdtool 'update' 'myrrdfile.rrd' '1211886000:130' rrdtool 'update' 'myrrdfile.rrd' '1211889600:140' rrdtool 'update' 'myrrdfile.rrd' '1211893200:150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

  29. Where RRDtool shines • Graph time-based data • Don’t worry about scaling or axis text • Stacked graphs

  30. Where RRDtool sucks • Can’t add data for past events • Can’t add data twice at the same timestamp

  31. Pitfalls • Make sure that the first update happens after RRD “start” time • Updates need to happen at increasing timestamps • On large step sizes (1 year), limit the number of rows to fit within < 2030.

  32. Debugging with Log4perl use Log::Log4perl qw(:easy) Log::Log4perl-> easy_init( $DEBUG );

  33. Log4perl Output rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' '1211875200:100' rrdtool 'update' 'myrrdfile.rrd' '1211878800:110' rrdtool 'update' 'myrrdfile.rrd' '1211882400:120' rrdtool 'update' 'myrrdfile.rrd' '1211886000:130' rrdtool 'update' 'myrrdfile.rrd' '1211889600:140' rrdtool 'update' 'myrrdfile.rrd' '1211893200:150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

  34. Example: Stock Portfolio

  35. Example: Measure Power Usage • Mastech MAS-345 • Serial Interface (RS-232)

  36. Example: Measure Power Usage

  37. Example: Measure Power Usage

  38. Aberrant Behaviour

  39. The End Thanks!

  40. References • RRDTool::OO on CPAN • RRD::Simple on CPAN • Portfolio Rendering article: http://perlmeister.com/talks/lm-200805.pdf • RRDTool article: http://www.linux-magazine.com/issue/44/Perl_RDDtool.pdf • RRDTool article (in German): http://www.linux-magazin.de/Artikel/ausgabe/2004/06/perl/perl.html

  41. Bonus Slides

  42. Aberrant Behaviour • Holt Winter Algorithm • Data Smoothing/Prediction

  43. Aberrant Behaviour • Holt Winter Algorithm • RRAs: • HWPREDICT • SEASONAL • DEVSEASONAL • DEVPREDICT • FAILURES • RRA:HWPREDICT:rows:alpha:beta:seasonal_period

  44. Aberrant Behaviour • alpha (baseline): 0..1 • 1: recent observations carry greater weight • 0: past observations carry greater weight • beta (slope): 0..1 • gamma (seasonal): 0..1 • failures defaults to 7

  45. Aberrant Behaviour • Example: • alpha = 0.50 • beta = 0.50 • gamma = 0.01 • seasonal_period = 2 • threshold = 7 • window_length = 9

More Related