S3: A Secure Scalability Service for Dynamic Content

S3: A Secure Scalability Servicefor Dynamic Content Bruce Maggs Carnegie Mellon University and Akamai Technologies Joint work with Charlie Garrod and Amit Manjhi and Natassa Ailamaki, Phil Gibbons, Todd Mowry, Chris Olston, and Anthony Tomasic.

CNN.com Page views/day (in millions) Number of requests a website receives is unpredictable CNN, NY Times, ABC News unavailable from 9-10 AM (Eastern Time) Content providers’ dilemma: how many resources to provision? Need on-demand scalability

CNN.com 50k Page views/day (in millions) 1.2k Page was 1.2k instead of 50k on 12 Sep, 01 Used Akamai on Election day 50k Content Delivery Network (CDN) Solution Source: http://www.tcsa.org/lisa2001/cnn.txt http://www.akamai.com/en/html/about/press/press479.html

DB App Server Web Server Typical Web-Site Architecture Request Users Executecode AccessDB Response Home server

CDN Architecture Internet core Users CDN nodes Content providers CDNs excel at delivering static content.

Advantages of CDNs • Large infrastructure handles load spikes • Clients charged on a per-usage basis • no need to guess what resources to provision • Moves data closer to end-users • decreases latency and increases throughput

CDN Application Services CDN’s can also run applications but for data-intensive dynamic applications… Internet DB Users database server becomes the bottleneck!

Methods to scale the database component • In-house database scalability: [DBCache, DBProxy, MTCache, NEC Cache Portal] • Must provision for peak load • Database outsourcing: Database as a service [Hacigumus+ ICDE ’02, SIGMOD ’02] • Have to cede control of data • Database Scalability Service (DBSS): Shared infrastructure that caches applications’ data [INRIA/LIP6, CIDR ’05, SIGMOD ’06, ICDE ’07]

S3 Database Scalability Service • CDN-like proxy nodes cache results of database queries • reduces load on central database servers • All database updates sent to central server • clients don’t cede ownership of their data • Uses publish/subscribe system to maintain data consistency • avoids additional load at the central server • Content provider may encrypt database requests/responses to protect sensitive data

Database Scalability Service users: Content Delivery Network DBSS Internet home server databases:

Database Scalability Service Internet users: Web and application servers DBSS home server databases:

Database Scalability Service client apps: DBSS Internet home server databases:

Outline • Need for on-demand scalability • S3 invalidation mechanism • Security-scalability tradeoff • Reducing latency

Addressing consistency • TTL is wasteful: • Often refresh cached data unnecessarily (workloads dominated by reads) • Must set TTL=0 for strong consistency! • Solution: update or invalidate cached data only when affected by updates • Naïve approach: home organizations notify proxy servers of relevant updates  not scalable Our approach: Fully-distributed, proxy-to-proxy update notification mechanism

update update notification Multicast Environment update notification Distributed Consistency Mechanism users proxy node • Distributed app-level multicast environment, e.g., Scribe • Forward all updates to backend home servers

Configuring Multicast Channels • Key observation: Web applications typically interact with DB via a small, fixed set of query/update templates (usually 10-100) • Example: SELECT qty FROM inv WHERE id = ? UPDATE inv SET qty = ? WHERE id = ? Templates: natural way to configure channels Options: Channel-by-query or Channel-by-update

Channel-by-Query Option • One channel per query template Q: C(Q) • Few subscriptions/cached result • Many invalidation notifications/update Conflicts determined lazily (upon update)

Channel-by-Update Option • One channel per update template U: C(U) • Many subscriptions/cached result • Few invalidation notifications/update Conflicts determined eagerly (when caching Q)

Parameter-Specific Channels • Optimization: consider parameter bindings supplied at runtime … for example: • Q5: SELECT qty FROM inv WHERE id = ? • When issued with id = 29, create extra parameter-specific channel C(5, 29) • Subscribe to both C(5) and C(5, 29) • Upon update: • If update affects a single item with id = X, send notification on channel C(5, X) • Saves work if X  29 • Updates affecting multiple items sent to C(5)

S3 Prototype • Tomcat as proxy web server/servlet container • Proxy database cache written in Java • Queries: access cached data when possible • Cache JDBC query results (i.e., materialized views) • Index results by JDBC query representation • MySQL4 as back-end database • Updates: sent to back-end database • Invalidation notifications delivered via Scribe • Experiments on Emulab (Utah) – Thanks!

Benchmark Applications • Bookstore (TPC-W, from UW-Madison) • Online bookseller, a standard web benchmark • Changed the popularity of books • Auction (RUBiS, from Rice) • Modeled after Ebay • Bulletin board (RUBBoS, from Rice) • Modeled after Slashdot Benchmarks model popular websites

Selective: cache queries only if subscribed to parameter-dependent groups

Impact of Cooperative Caching

Guaranteeing security in a DBSS setting Limit ability to observe an application’s data by: • DBSS administrator • Unauthorized application through the DBSS Security-Scalability tradeoff in the DBSS setting Analyzing the code helps in managing this tradeoff

A simple solution for guaranteeing security • Outsource database scalability • Home server: master copies of all data—handles updates directly • No query execution on the DBSS • DBSS caches query results (read-only)—kept consistent by invalidation • All data passing through the DBSS can be encrypted: • Query, Update, Query results

Result Result A Simple Example toys (toy_id, toy_name) No Invalidations Nothing is encrypted Empty Q1: toy_id=15 Q1 U1 DBSS Home server Database Q1: SELECT toy_id FROM toys WHERE toy_name=“GI Joe” U1: DELETE FROM toys WHERE toy_id=5 Invalidate Results are encrypted Empty Q1: Q1 U1 More encryption leads to more invalidations

Challenge: providing scalability while guaranteeing security When updates occur, DBSS needs to invalidate Application faces a dilemma in what data to encrypt (secure) More encryption Less encryption Conservative Invalidation Precise Invalidation Security Scalability Security-scalability tradeoff

Opportunity for managing the tradeoff Not all data is equally sensitive Data Sensitivity Completely insensitive Extremely sensitive Moderately sensitive Bestsellers list Inventory records, customer records Credit Card Information Care but worried about scalability impact Secure at all costs Don’t care • But for most data, nontrivial to assess: • Data-sensitivity • Scalability impact of securing the data

Given templates: Can statically identify data not needed for precise invalidation Key Insight: arbitrary queries and updates not possible function get_toy_id ($toy_name) { $template:=“SELECT toy_id FROM toys WHERE toy_name=?”; $query:=attach_to_template ($template, $toy_name); execute ($query); … }

Data not useful for invalidation: examples Example 1: Q1: SELECT toy_id FROM toys WHERE toy_name=? Q2: SELECT toy_name FROM toys WHERE toy_id=? No data is needed for precise invalidation Example 2: Q1: SELECT toy_id FROM toys WHERE toy_name=? U1: DELETE FROM toys WHERE toy_id=? Query parameters are not needed for precise invalidation (the query result is needed though)

Security without hurting scalability Data not needed for invalidation Can secure “for free” (without hurting scalability) Security Conscious Scalability Approach [SIGMOD ’06] As a result, Tradeoff has to be only managed over remaining data

5 ms 100 ms Home server CDN and DBSS Users Sample experiment: methodology • Scalability: max # concurrent users with acceptable response times • Security: # templates with encrypted results • California Privacy Law determined sensitive data • Non-transactional invalidation • Start with a cold cache

Benchmark Applications • Bookstore (TPC-W, from UW-Madison) • Online bookseller, a standard web benchmark • Changed the popularity of books • Auction (RUBiS, from Rice) • Modeled after Ebay • Bulletin board (RUBBoS, from Rice) • Modeled after Slashdot Benchmarks model popular websites

x x x x x x Security-Scalability Tradeoff U1: DELETE FROM toys WHERE toy_id=5 Security Scalability X denotes encrypted, visible

Magnitude of Security-Scalability tradeoff Scalability (number of concurrent users supported) 0 0 Benchmark Applications

Security Results Query data that can be encrypted “for free” 7 7 7 4 6 17 and result 14 18 12 Bboard Bookstore Auction

Security Results in Detail • Auction: The historical record of user bids was not exposed • Bboard: The rating users give one another based on the quality of their posting • Bookstore: Book purchase association rules discovered by the vendor – customers who purchase book A also purchase book B

SCSA Scalability Conscious Security Approach (SCSA) to managing the tradeoff 900 Nothing encrypted 600 Scalability (Number of concurrent users supported) Everything 300 encrypted 0 0 5 10 15 20 25 30 Security (Number of query templates with encrypted results) 1. Easy to either get good scalability or good security 2. SCSA presents a shortcut to manage the tradeoff

Contributors to User Latency Request, high latency Database Web server App server Response, high latency Traditional architecture high latency DBSS Database CDN DBSS architecture A single HTTP request  Multiple database requests 42

Sample Web Application Code function find_comments ($user_id) { $template:=“SELECT from_id, body FROM comments WHERE to_id=?” $query:=attach_to_template ($template, $user_id) $result:=execute ($query) foreach ($row in $result) print (get_body ($row), get_name (get_id ($row))) } • (N+1) queries are issued because: • Convenient for programmers to abstract database values • No effect in the traditional setting Found many examples in the benchmark applications 43

Reducing User Latency in a DBSS Setting Transformations to reduce number of round-trips Group execution of queries: MERGING transformation Overlap execution of queries: NONBLOCKING transformation Transformed program and SQL Web Application Code Transformed Code Procedural program with embedded SQL Holistic transformations using src-to-src compilers 44

The MERGING Transformation www.ebay.com John Names of users who have posted comments about John Content Delivery Network 1 Query • Find user_ids who have made comments • For each user_id, find name of the user Database Scalability Service N Queries High latency 45

The MERGING Transformation SELECT from_id, u.name FROM comments, users u WHERE from_id = u.id AND to_id = ? Find names of users who have commented about John Names of users who have posted comments about John  • Find user_ids who have made comments • For each user_id, find name of the user Assuming constant cache hit rate, the #round-trips to the database decreases by a factor of (N+1) 46

The NONBLOCKING Transformation www.amazon.com John Home page Content Delivery Network • Greet user • Get names of related books Database Scalability Service High latency Issue queries concurrently to reduce latency 47

Applicability of the Transformations Either transformation applies to 25% (Auction), 75% (Bboard), and 50% (Bookstore) dynamic runtime interactions 48

BBOARD Application: Impact on Latency Average latency in ms Transformations Overall latency decreases by 38%, the DBSS-DB latency decreases by 65% 49

Impact of Latency on Scalability Improved scalability Scalability Threshold Latency curve Latency Reduced latency curve Simultaneous users supported Reducing latency improves scalability 50

S3: A Secure Scalability Service for Dynamic Content