Real World Performance Tuning

1. Real WorldPerformance Tuning Ask Bjørn Hansen OSCON 2001

2. Performance Tuning • Show more pages • Faster • With less resources • Design the architecture right • Optimize the code • (in that order!)

3. Memory usage • N connections = N fat mod_p erl processes • Fat processes doing little other than buffering to slow clients

4. Memory usage 20 connections per second + Each request takes 3 seconds to write to the network = 60 active mod_perl processes + 15 spare processes for peaks = 75 active mod_perl processes * 20MB-12MB shared = 8MB memory = 600MB memory

5. Reverse proxy • Offload the buffering to a reverse proxy • Can • Do caching • Serve static content • Distribute requests to different backend processes • All in a “slim” process

6. Reverse proxies • squid • best caching • not as flexible for a "reverse proxy" as mod_proxy • apache/mod_proxy • simple to configure • known environment • can cache • mod_rewrite

7. apache/mod_proxy • Specify what to pass through RewriteRule ^/(foo/.*) http://localhost:8001/$1 [P] • Specify what NOT to pass through RewriteCond %{REQUEST_URI} !^/images/ RewriteRule /(.*) http://localhost:8002/$1 [P] • Allow fix up of $r->connection->remote_ip LoadModule proxy_add_forward_module modules/mod_proxy_add_forward.so

8. Memory Usage with apache/mod_proxy • mod_proxy 20 connections per second + Each request takes 3 seconds to write to the network = 60 active mod_proxy processes + 15 spare = 75 running mod_proxy processes 75 mod_proxy processes (300KB each) = ~25MB memory

9. Memory Usage with apache/mod_proxy • mod_perl 20 connections per second + Each request takes <.05 seconds to write to the proxy = 1-2 active mod_perl processes + 3 spare = 5 running mod_perl processes 5 mod_perl processes (8MB non-shared each) = 40MB memory

10. Memory Usage with apache/mod_proxy 25+40MB = 65MB total memory usage • >500MB less than the mod_perl alone!

11. Basic httpd.conf tuning • mod_proxy MaxClients 512 StartServers 50 MinSpareServers 20 MaxSpareServers 100 • mod_perl MaxClients 5 StartServers 3 MinSpareServers 1 MaxSpareServers 5 Port 80 Listen 8001

12. mod_status ExtendedStatus on <Location /server-status> SetHandler server-status Order deny,allow Deny from all Allow from 1.2.3.5 </Location>

13. <VirtualHost> configuration NameVirtualHost 1.2.3.4 <VirtualHost 1.2.3.4> ServerName jobs.perl.org ... RewriteCond %{REQUEST_URI} !.*\.(gif|png|jpg)$ RewriteRule /(.*) http://localhost:8010/$1 [P] </VirtualHost> <VirtualHost 1.2.3.4> ServerName onion.perl.org ServerAlias learn.perl.org ... RewriteRule /(.*) http://localhost:8020/$1 [P] </VirtualHost>

14. URI configuration RewriteRule /(bar/.*) http://localhost:8030/$1 [P] RewriteRule /(foo/.*) http://otherhost:8040/$1 [P] • Each backend process can run as a different user • A process per developer • A process per customer • A process per site • Backends can run on different machines

15. Focus on code quality • The mod_perl guide recommends not using the IO:: modules or use vars qw(%foo); • I say use them if you would like to • far fewer mod_perl processes • use Exporter to export your function names as needed • Of course, don't go crazy and use POSIX and CGI.pm everywhere

16. Load balancing • mod_rewrite can do simple load balancing • the mod_perl processes can be behind a load balancer, RewriteRule /(foo/.*) http://modperl.farm.foo.com:8030/$1 [P] • mod_backhand

17. Caching • Browsers/end user proxies can cache from servers • set the right headers, content-length, last-modified • Reverse proxies • you set the rules, if complete documents can be cached • Application can cache from other parts of the system (eg database) • even the database can cache some of what it has done

18. HTTP headers • Expires: • Content-Length: • Cache-Control: • Pragma: (old "Cache-Control") • When the complete documents can be cached. • If-Modified-Since: • 304 response

19. Application caching • Generate static content every N minutes • for small set of files • Save results from the SQL database • in a local BerkeleyDB or similar • Generate a Storable or BerkeleyDB file centrally and rsync it to each mod_perl server • Create summary tables in the database • to lighten the load of heavy queries • Mix and match for fragments

20. Caching summary • Decide for how long data can be considered "fresh enough" • Cache fragments of pages where possible • Make each part of the system cache and aggregate as much as possibly • Make SQL queries as simple and fast as possibly

21. Databases • Databases are hard(er) to scale • Reverse proxy minimizes the need for many concurrent database connections • Apache::DBI minimizes the number of new connections made • Caching minimizes the number of lookups • Summary tables can make the lookups faster

22. Summary • Architecture more important than code • Use many proxies • Allowing fewer heavy backends • Caching is fundamental

23. Resources • The mod_perl guide's performance section • http://perl.apache.org/guide/performance.html • lot's of nitty gritty details • DBI and mod_perl • http://www.saturn5.com/~jwb/dbi-performance.html • Database performance • Tim’s Advanced DBI Tutorial • http://www.cpan.org/authors/id/TIMB/DBI_Talk5_2001.tar.gz • These slides • http://develooper.com/modperl/