1 / 17

Data Sets and Data Resources Malcolm Atkinson Director of National e-Science Centre nesc.ac.uk

Data Sets and Data Resources Malcolm Atkinson Director of National e-Science Centre www.nesc.ac.uk 25 th June 2003 GGF8 Seattle DAIS WG Second WG Meeting at GGF8. Generic Idea. Abstract / Physical Value Passed or Stored. Data Set. Source Program. Consumer Program. Data Set. Source

elaine-lowery
Download Presentation

Data Sets and Data Resources Malcolm Atkinson Director of National e-Science Centre nesc.ac.uk

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Sets and Data Resources Malcolm Atkinson Director of National e-Science Centre www.nesc.ac.uk 25th June 2003 GGF8 Seattle DAIS WG Second WG Meeting at GGF8 GGF DAIS WG

  2. Generic Idea • Abstract / Physical Value Passed or Stored Data Set Source Program Consumer Program Data Set Source Program Store GGF DAIS WG

  3. Questions • Do we need the concept? • Once Identified remains Identified • Where ever copied, stored, moved • Value (information) remains constant • Type(s) remain constant • Basis for reference, e.g. parameters pass by reference • If yes, is it the right name? • Data Set • Value Set • Is it a Grid Service? If not, what is it? • What do we talk about? • The Data Set or the End Point • What are its properties? • Kinds of data set • Operations on the whole • Creation & access • Verification of Identity / Constancy • Relationship with Data Resource? • Same, Special case of or Different GGF DAIS WG

  4. DBMS DB ResultSet Conceptual ModelExternal Universe External data resource manager External data resource External data set GGF DAIS WG

  5. DBMS DB ResultSet Conceptual ModelDAI Service Classes Data resource manager Data resource Data activity session Data request Data set GGF DAIS WG

  6. DBMS DB Data request ResultSet Example of Data Set Use Data Resource Client Data activity session GGF DAIS WG

  7. DBMS DB Data request ResultSet Example of Data Set Use Data Resource Client Data activity session Staging Store Consumer Consumer GGF DAIS WG

  8. DBMS DB Data Resource Data request Client Data activity session ResultSet Staging Store Consumer Consumer Rds1 Sds1 Sds1 Rds1 Rds1 Example of Data Set Use (Audit) Log …..…… …….. ……. ……. GGF DAIS WG

  9. DBMS These Data Sets? These End Points? DB Data request ResultSet What do we talk about? OR Data Resource Client Data activity session Staging Store Consumer Consumer GGF DAIS WG

  10. Kinds of Data Setby Content / Format Other standards, I.e. not defined by DAIS GridService DAIS general across eds DataSet WebRowSet XMLSequence BinXDataSet Directory Tree DAIS specific to eds ADODataSet ParameterSet FileSet ? Extension not defined by DAIS GGF DAIS WG

  11. Delivery Push as one Unit Push incrementally Push at most once Push to subscribers Hold until get all Hold until get fragments Hold until any get Materialisation Materialise on Demand Materialise Eagerly Materialise in Parallel Connection Method Publish and subscribe Pass handle via client Declarative, … Longevity Until eol Until eol or destroy Until used n times Security Encryption method Unit of Access (items) Multiples of Row Multiples of Children … Many other properties Cursor available Reset available Compression Kinds of Data Setby Behaviour / Usage GGF DAIS WG

  12. Data Set No update Copy & Move Retains persistent identity (GUID) DS description Retains value Retains type Simple(?) access repertoire Lifetime bound to physical eds Minimal SDEs Data Resource Rich Repertoire of Access & Update Methods Value changes Type (may) Change Has a defined drm Located in edr Lifetime independent of edr Complex SDEs Relationship with Data Resource GGF DAIS WG

  13. Data Movement Currently Delivery Specification Functions on a das GGF DAIS WG

  14. Motivation & QoS Matters • Results of no value • If not delivered to a consumer • Updates • Need supplied values • These may be large! • Supplier and Consumer(s) • Same process or different processes • Same machine or same cluster • Same LAN or Wide Area hop • Same architecture, hosting environment or different • Are messages (notifications) transmitted from triggers always small? Treated differently? • Are they in or out of scope? GGF DAIS WG

  15. Requirements • Information Preservation • A value transmitted arrives as the same value • For all values of all types • Composability • Any Grid (Data) Service with • Any Data Movement Mechanism • Performance & Costs • Choices to be supported? GGF DAIS WG

  16. Requirements 2 • Integration • Available between all services • Available for all Information • Data, Code, Processes, State • DAIS-WG current focus is Information • Higher-level view of data • Control • What is moved (Query result|DB | File I set of (files | results), …) • Where From & To* (Notation in Request document?) • What units of transfer (Logical | Physical) • What synchronisation (and lifetime) • Privacy, e.g. Encryption • Dynamic selectivity & Flow control • What compression / encoding • What transformations en route • What monitoring mechanisms GGF DAIS WG

  17. Fundamental Issues • What portType(s)? • What is (are) their functions? • What semantics of the functions? • Effects on data • Effects on GD* state • Reflection in SDEs • Failure modes GGF DAIS WG

More Related