320 likes | 418 Views
Nondeterministic Queries in a Relational Grid Information Service. Peter A. Dinda Dong Lu Prescience Lab Department of Computer Science Northwestern University http://plab.cs.northwestern.edu. Overview. RGIS: GIS system based on the relational data model using SQL
E N D
Nondeterministic Queries in a Relational Grid Information Service Peter A. Dinda Dong Lu Prescience Lab Department of Computer Science Northwestern University http://plab.cs.northwestern.edu
Overview • RGIS: GIS system based on the relational data model using SQL • Complex compositional queries can be posed • “Find me 16 hosts on the same LAN that together have 32 GB of RAM” • Can be very expensive to answer • Joins: worst case O(n^m) for m tables of size n • Introduce nondeterminism • User gets random sample of result set • Automated query transformation
Outline • Overview • Model • Implementation • Nondeterministic queries • Performance evaluation • Related work • Conclusions D. Lu and P. Dinda, Synthesizing Realistic Computational Grids, SC 2003 D. Lu, J. Skicewicz, and P. Dinda, Scoped and Approximate Queries in a Relational Grid Information Service, Grid 2003
RGIS Model of a Grid • Annotated network topology graph • Annotation examples • Hosts: memory, disk, OS, NICs, etc. • Router/Switch: backplane bandwidth, ports • Link: latency and bandwidth • Highly dynamic data in streams, not DB • Virtualization, Futures, Leases • Virtual machines module Software endpoint router iplink host Network Data link maclink macswitch Physical connectorswitch connectorlink
Outline • Overview • Model • Implementation • Nondeterministic queries • Performance evaluation • Related work • Conclusions D. Lu and P. Dinda, Synthesizing Realistic Computational Grids, SC 2003 D. Lu, J. Skicewicz, and P. Dinda, Scoped and Approximate Queries in a Relational Grid Information Service, Grid 2003
Software Metadata Network Types Data Link Security Physical
RGIS Design (Intersite) B A Update Push To Friend Site RGIS Server RGIS Server Update Push To Friend Site • Site RGIS server pushes local updates to friend sites • Site RGIS server consolidates updates from site and friend sites • Site RGIS server answers all queries originating from its site C RGIS Server
Insert/Update/Delete Dual Xeon 1 GHz, 2 GB, 8x36 GB RAID5, Oracle 9i x x
2,700 lines of • authored SQL • 4,000 lines of • generated PL/SQL • 22,000 lines of • authored Perl • Main dependencies • DBI to Oracle 9i • SOAP::Lite • CGI • Not finished yet!
RGIS Design(Per Site) This talk
Outline • Overview • Model • Implementation • Nondeterministic queries • Performance evaluation • Related work • Conclusions D. Lu and P. Dinda, Synthesizing Realistic Computational Grids, SC 2003 D. Lu, J. Skicewicz, and P. Dinda, Scoped and Approximate Queries in a Relational Grid Information Service, Grid 2003
Motivation • Queries for compositions of resources easily expressed in SQL: • But such queries can be very expensive to execute • However, we typically don’t need the entire result set, just some rows, and not always the same ones • And we need them in a bounded amount of time select h1.insertid, h2.insertid from hosts h1, hosts h2 where h1.os=‘LINUX’ and h2.os=‘LINUX’ and h1.mem_mb+h2.mem_mb>=3072 “Find 2 hosts with Linux that together have 3 GB of RAM”
Why Not Just Limit? • Oracle rownum, MySQL limit clause • “Return first k rows of result set” • Problem: Always get the SAME answer • Problem: May STILL take a long time • Results not discovered until near the end • Problem: Query time related to DATA as well as k
Query Approaches Nondeterministic results (this paper) All results Available in Grid 2003 Paper Approximate results Scoped results Return Random Sample of Result Set
Nondeterministic Version of Query select nondeterministically h1.insertid, h2.insertid from hosts h1, hosts h2 where h1.os=‘LINUX’ and h2.os=‘LINUX’ and h1.mem_mb+h2.mem_mb>=3072 within 2 seconds
Implementing non-deterministic queries select nondeterministically h1.insertid, h2.insertid from hosts h1, hosts h2 where h1.os=‘LINUX’ and h2.os=‘LINUX’ and h1.mem_mb+h2.mem_mb>=3072 within 2 seconds Using Oracle-Specific Extensions SELECT H1.INSERTID, H2.INSERTID FROM HOSTS H1 SAMPLE(P), HOSTS H2 SAMPLE(P) WHERE (H1.OS='LINUX' AND H2.OS='LINUX' AND H1.MEM_MB+H2.MEM_MB>=3072) Query Manager and Rewriter Random sample ofinput tables withSelection Probability Pdetermined by time constraintand server load
Implementing non-deterministic queries select nondeterministically h1.insertid, h2.insertid from hosts h1, hosts h2 where h1.os=‘LINUX’ and h2.os=‘LINUX’ and h1.mem_mb+h2.mem_mb>=3072 within 2 seconds Using Our Schema (Not Oracle-Specific) Rest of Talk SELECT H1.INSERTID, H2.INSERTID FROM HOSTS H1, HOSTS H2 , INSERTIDS TEMP_H1 , INSERTIDS TEMP_H2 WHERE (H1.OS='LINUX' AND H2.OS='LINUX' AND H1.MEM_MB+H2.MEM_MB>=3072) AND(H1.INSERTID=TEMP_H1.INSERTID AND TEMP_H1.rand > 982663452.975047 AND TEMP_H1.rand <= 1025613125.93505) AND (H2.INSERTID=TEMP_H2.INSERTID AND TEMP_H2.rand > 1877769069.94039 AND TEMP_H2.rand <= 1920718742.90039) Query Manager and Rewriter Random sample ofinput tables withSelection Probability Pdetermined by time constraintand server load
Implementing non-deterministic queries Host insertid random_number 0 N x x+y Random Starting Point y=P*N Reshuffling Requirement
Deadlines • Hard-limiting • Time-limited thread or process forked • Climbing • Start with low probability p, issue query, if no results, double probability, try again, keep going until no more time or have results • Estimation • Like climbing, but do polynomial estimation over previous runs to estimate if next run will exceed deadline
Outline • Overview • Model • Implementation • Nondeterministic queries • Performance evaluation • Related work • Conclusions D. Lu and P. Dinda, Synthesizing Realistic Computational Grids, SC 2003 D. Lu, J. Skicewicz, and P. Dinda, Scoped and Approximate Queries in a Relational Grid Information Service, Grid 2003
GridG: Synthesing Realistic Computational Grids • Generates a Grid as an annotated layer 3 topology • Hosts, routers, links • Graph conforms to power laws of Internet topology • Annotations include: • memory, clock speed, cpu type, number of CPUs, operating system type, link bandwidths, router bandwidths, etc. • Memory distribution according to Smith study of MDS contents http://www.cs.northwestern.edu/~urgis/GridG
Nondeterministic query performance Select two hosts that together have >3GB of RAM Meaningful tradeoff between query processing time and result set size is possible
Nondeterministic query performance Select n hosts that together have >3GB of RAM, holding query time constant Can use tradeoff to controlquery time independent of query complexity
Deadlines Max Min Find 2 hosts with collective 600 GB RAM (VERY RARE)in 50K host grid
Extending RGIS to Support Grid Computing On Virtual Machines • Virtuals • Each RGIS object has a unique id • Virtualization table associates unique id of virtual resources with unique ids of their constituent physical resources • Virtual nature of resource is hidden unless query explicitly requests it • Futures • An RGIS object that does not exist yet • Futures table of unique ids • Future nature of resource hidden unless query explicitly requests it
Related Work • SLP, X.500, LDAP • Condor ClassAds • MDS • R-GMA • Redline • Random sampling from databases • Olsen, others
Conclusions • GIS system based on relational data model • Powerful queries, but expensive to execute • Nondeterminism to control query time • Can be implemented without RDMBS support • Automated query translation in RGIS • Several techniques to implement deadlines for queries
People and Acknowledgements • Students • Jason Skicewicz, Andrew Weinrich (Web + Soap), Jack Lange (CDN) • Collaborator • Relational Grid Resources Project at Indiana • Beth Plale • http://www.cs.indiana.edu/~plale/projects/RGR • Funder • NSF
For MoreInformation • URGIS Site • http://www.cs.northwestern.edu/~urgis • Prescience Lab • http://plab.cs.northwestern.edu Join The User Comfort Study! http://comfort.cs.northwestern.edu Special Advertising Section