400 likes | 587 Views
Building Fast and Secure Web Services with OKWS. Max Krohn - MIT CSAIL krohn@mit.edu. SparkMatch v1 (2000). SparkMatch v1 (2000). ~500,000 signups. WS1. Database. Load Balancer. WS2. WS3. MySQL. Apache ModPerl . SparkMatch Post-Mortem. v1 (2000) DB bottleneck due to table locking.
E N D
Building Fast and Secure Web Services with OKWS Max Krohn - MIT CSAIL krohn@mit.edu
SparkMatch v1 (2000) ~500,000 signups WS1 Database Load Balancer WS2 WS3 MySQL Apache ModPerl
SparkMatch Post-Mortem • v1 (2000) • DB bottleneck due to table locking
SparkMatch v1 (2000) ~500,000 signups WS1 Database Load Balancer WS2 WS3 MySQL Apache ModPerl
SparkMatch v2 (2001) ~1,000,000 signups Database WS1 Load Balancer WS2 WS3 WS4 MySQL Apache ModPerl WS5 WS6
SparkMatch Post-Mortem • v1 (2000) • DB bottleneck due to table locking • v2 (2001) • too many DB connections • Perl was too slow • difficult to implement efficient caching
SparkMatch v3 (2002) DB1 WS1 Load Balancer WS2 DB2 WS3 MySQL Apache C Modules
SparkMatch Post-Mortem • v1 (2000) • DB bottleneck due to table locking • v2 (2001) • too many DB connections • Perl was too slow • difficult to implement efficient caching • v3 (2002) • development cycle was too long • code was unsafe
Some Desired WS Features • Thin, fast server (no cruft) • Smart gzip support • Small # database connections per WS • Easy to build fast, efficient caches • Leaves free memory for caches • Easy and safe to write C/C++ code
Not Necessary • Support for static content • Images • Styles Sheets • Terms & Conditions • Bottleneck is access link Apache suffices!
Static/Dynamic Split WS1 ~$1,000 per Mb/s per Month DB1 Load Balancer WS2 DB2 WS3 New York City - Dynamic ~$100 per Mb/s per Month Texas - Static Apache Apache Apache Apache
Apache/PHP For Dynamic Content? • History of buffer overruns in Apache and PHP (see paper). • Users likely to introduce more when writing C extensions. • Poor isolation strategy.
Apache/PHP Isolation Apache/PHP Search Index and Profiles Process 1 Process 2 Search.php Search.php ShowProfile.php ShowProfile.php ChangePW.php ChangePW.php Passwords logger config logger config
Apache/PHP Isolation Apache/PHP Search Index and Profiles Process 1 Process 2 Search.php Search.php ShowProfile.php ShowProfile.php ChangePW.php ChangePW.php Passwords logger config logger config
What We Built • The OK Web Server (OKWS) • Specialized for dynamic content service. • Meets our earlier-stated requirements.
OKWS Design • A Web site consists of many Web services. • e.g., Search, ShowProfile, ChangePW • A and B are distinct services if they access different pools of data. • One-to-one mapping between Web Services and Unix processes.
OKWS Isolation Strategy • Process pool fixed at startup (~10). • Each obeys least-privilege principle. • Isolates processes: • From SQL database access • From each other • From the OS (filesystem in particular) • From DBs they need not access.
How To Build a Web Service ShowProfile ~2000 LOC RPC ~50 LOC R/O In-Memory Profile Cache Server Profile-DB Translator SQL DB
client OKWS Block Diagram GET /ShowProfile HTTP/1.1 launcher demux HTTP Response Pass FD Search ShowProfile ChangePW RPC RPC RPC Profile-DB Translator PW-DB Translator logger (write to FS) SQL pubd (read from FS) DB2 DB1
OKWS Process Isolation Log Jail Docs Jail Run Jail launcher UID=0 demux UID=okd logger UID=oklog Search UID=51001 ShowProfile UID=51002 ChangePW UID=51003 pubd UID=www Web Server Machine Profile-DB Translator PW-DB Translator DB1 DB2 Database Machines
If Service A is Compromised… • cannot access its own DB outside the RPC interface provided. • cannot access setuid executables. • cannot access logs, config files, source files, privileged ports. • cannot send service B signals • cannot trace service B's system calls • cannot access B's database
OKWS Implementation • Developed C++ with the SFS libraries • Single Threaded / Event Driven • Services developed with C++/SFS. • DB translation libraries: • SFS-style asynchronous RPC • Standard threaded client libraries.
Perl-Like Development Tools • Refcounted garbage collection. • Refcounted safe string classes. • RPC Stubs for network communication. • Perl-compatible regular expressions. • OKWS preprocessor for Perl-style heredocs.
1 Service = 1 Process, 1 Thread • No synchronization required for in-memory caches. • Small, fixed-size process pool more memory available for caching. • Each service needs one connection to database machine. • No pooling required.
OkCupid Deployment • 23 Public Web Services • 12 Database Translators
Site Performance • Pentium Dual 2.4GHz Xeon / FreeBSD • 1M+ users, 100k+ now active • Example throughput observed near 200 req/sec (~20KB each) • one WS handled this load with 7% CPU utilization
OKWS Performance Testing • In paper, compared to: • Apache2/PHP4 • Haboob (JDK1.3) • Flash v1 • Results: OKWS gets better throughput, responsiveness.
Performance Datapoint • C/C++ faster than interpreted code • Methodology: Generate a 10x30 HTML-formatted multiplication table. • (Same LOC for both PHP and OKWS)
OKWS Partial Summary • Thin, fast server (no cruft) • No DB pooling required. • Easy to build fast, efficient caches • Not wasteful of memory. • Easy and safe to write C++ code.
Conclusion • Used to build real Web sites! • Match Service (OkCupid.com) • New Diversion (Movie-Madness.org) • Could Also be Used For • Blogging Sites • Social network sites • HTTP Front end to distributed systems • GPL'ed at: www.okws.org
The End www.okws.org