1 / 54

Pads/ML: A Functional Data Description Language

Pads/ML: A Functional Data Description Language. David Walker Princeton University with: Yitzhak Mandelbaum (Princeton), Kathleen Fisher and Mary Fernandez (AT&T). Data, data everywhere!. Incredible amounts of data stored in well-behaved formats:. Databases:. Tools Schema Browsers

lynnea
Download Presentation

Pads/ML: A Functional Data Description Language

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pads/ML:A Functional Data Description Language David Walker Princeton University with: Yitzhak Mandelbaum (Princeton), Kathleen Fisher and Mary Fernandez (AT&T)

  2. Data, data everywhere! Incredible amounts of data stored in well-behaved formats: Databases: Tools • Schema • Browsers • Query languages • Standards • Libraries • Books, documentation • Conversion tools • Vendor support • Consultants… XML:

  3. Ad hoc Data • Vast amounts of data inad hoc formats. • Ad hoc data is semi-structured: • Not free text. • Not as rigid as data in relational databases. • Examples from many different areas: • Physics • Computer system maintenance and administration • Biology • Finance • Government • Healthcare • More!

  4. Ad Hoc Data in Biology format-version: 1.0 date: 11:11:2005 14:24 auto-generated-by: DAG-Edit 1.419 rev 3 default-namespace: gene_ontology subsetdef: goslim_goa "GOA and proteome slim" [Term] id: GO:0000001 name: mitochondrion inheritance namespace: biological_process def: "The distribution of mitochondria\, including the mitochondrial genome\, into daughter cells after mitosis or meiosis\, mediated by interactions between mitochondria and the cytoskeleton." [PMID:10873824,PMID:11389764, SGD:mcc] is_a: GO:0048308 ! organelle inheritance is_a: GO:0048311 ! mitochondrion distribution www.geneontology.org

  5. Ad Hoc Data in Chemistry O=C([C@@H]2OC(C)=O)[C@@]3(C)[C@]([C@](CO4) (OC(C)=O)[C@H]4C[C@@H]3O)([H])[C@H] (OC(C7=CC=CC=C7)=O)[C@@]1(O)[C@@](C)(C)C2=C(C) [C@@H](OC([C@H](O)[C@@H](NC(C6=CC=CC=C6)=O) C5=CC=CC=C5)=O)C1

  6. Ad Hoc Data in Finance HA00000000START OF TEST CYCLE aA00000001BXYZ U1AB0000040000100B0000004200 HL00000002START OF OPEN INTEREST d 00000003FZYX G1AB0000030000300000 HM00000004END OF OPEN INTEREST HE00000005START OF SUMMARY f 00000006NYZX B1QB00052000120000070000B000050000000520000 00490000005100+00000100B00000005300000052500000535000 HF00000007END OF SUMMARY www.opradata.com

  7. Ad Hoc Data from Web Server Logs (CLF) 207.136.97.49 - - [15/Oct/1997:18:46:51 -0700] "GET /tk/p.txt HTTP/1.0" 200 30 tj62.aol.com - - [16/Oct/1997:14:32:22 -0700] "POST /scpt/dd@grp.org/confirm HTTP/1.0" 200 941 234.200.68.71 - - [15/Oct/1997:18:53:33 -0700] "GET /tr/img/gift.gif HTTP/1.0” 200 409 240.142.174.15 - - [15/Oct/1997:18:39:25 -0700] "GET /tr/img/wool.gif HTTP/1.0" 404 178 188.168.121.58 - - [16/Oct/1997:12:59:35 -0700] "GET / HTTP/1.0" 200 3082 214.201.210.19 ekf - [17/Oct/1997:10:08:23 -0700] "GET /img/new.gif HTTP/1.0" 304 -

  8. Ad Hoc Data: DNS packets 00000000: 9192 d8fb 8480 0001 05d8 0000 0000 0872 ...............r 00000010: 6573 6561 7263 6803 6174 7403 636f 6d00 esearch.att.com. 00000020: 00fc 0001 c00c 0006 0001 0000 0e10 0027 ...............' 00000030: 036e 7331 c00c 0a68 6f73 746d 6173 7465 .ns1...hostmaste 00000040: 72c0 0c77 64e5 4900 000e 1000 0003 8400 r..wd.I......... 00000050: 36ee 8000 000e 10c0 0c00 0f00 0100 000e 6............... 00000060: 1000 0a00 0a05 6c69 6e75 78c0 0cc0 0c00 ......linux..... 00000070: 0f00 0100 000e 1000 0c00 0a07 6d61 696c ............mail 00000080: 6d61 6ec0 0cc0 0c00 0100 0100 000e 1000 man............. 00000090: 0487 cf1a 16c0 0c00 0200 0100 000e 1000 ................ 000000a0: 0603 6e73 30c0 0cc0 0c00 0200 0100 000e ..ns0........... 000000b0: 1000 02c0 2e03 5f67 63c0 0c00 2100 0100 ......_gc...!... 000000c0: 0002 5800 1d00 0000 640c c404 7068 7973 ..X.....d...phys 000000d0: 0872 6573 6561 7263 6803 6174 7403 636f .research.att.co

  9. Properties of Ad hoc Data • Data arrives “as is” -- you don’t choose the format • Documentation is often out-of-date or nonexistent. • Hijacked fields. • Undocumented “missing value” representations. • Data is buggy. • Missing data, “extra” data, … • Human error, malfunctioning machines, software bugs (e.g. race conditions on log entries), … • Errors are sometimes the most interesting portion of the data. • Data sources often have high volume. • Data might not fit into main memory. • Data can be created by malicious sources attempting to exploit software vulnerabilities • c.f. Ethereal network monitoring system

  10. The Goal(s) • What can we do about ad hoc data? • how do we read it into programs? • how do we detect errors? • how do we correct errors? • how do we query it? • how do we discover its structure and properties? • how do we view it? • how do we transform it into standard formats like CSV, XML? • how do we merge multiple data sources? • In short: how do we do all the things we take for granted when dealing with standard formats in a fault-tolerant and efficient, yet nearly effortless way?

  11. Enter Pads • Pads: a system for Processing Ad hoc Data Sources • Three main components: • a data description language • for concise and precise specifications of ad hoc data formats and semantic properties • a compiler that automatically generates a suite of programming libraries & end-to-end applications • a visual interface to support both novice and expert users

  12. One Description, Many Tools Data Description (Type T) compiler xml translator query engine parser printer visual data browser ... programming library complete application

  13. Some Advantages Over Ad Hoc Methods • Big bang for buck: 1 description, many tools • Descriptions document data sources • the documentation IS the tool generator so documentation is automatically kept up-to-date with implementation • Descriptions are easy to write, easy to understand. • descriptions are high-level & declarative • description syntax exploits programmer intuition concerning types • Tools are robust • Error handling code generated automatically; doesn’t clutter documentation. • Descriptions & generated tools can be analyzed and reasoned about • eg: data size, tool termination & safety properties, coherence of generated parsers & printers

  14. PADS/C [PLDI 05; POPL 06] Based on C type structure. Generates C libraries. too bad C doesn’t actually support libraries .... LaunchPADS visual interface [Daly et al., SIGMOD 06] PADS/ML (Mandelbaum’s thesis) Based on the ML type structure. polymorphic, dependent datatypes Generates ML modules. better reuse & library structure functional data processing = far greater programmer productivity New framework for tool development. Format-independent algorithms architected using functors vs macros Implementation status. Version 1.0 up and running Many more exciting things to do Describe real formats: Newick tree-structured data Reglens galaxy catalogues Palm PDA databases AT&T call-detail data AT&T billing data Web server logs Gene ontologies DNS packets OPRA data More … The PADS Project

  15. Outline • Motivation and PADS Overview • Data Description in PADS/ML • Implementation architecture • The Semantic of PADS • Conclusions

  16. Base Types and Records • Base types: C (e). • Describe atomic portions of data. • Parameterized by host-language expression. • Examples: • Pint, Pchar, Pstring_FW(n), Pstring(c). • Tuples and Records:t * t’ and{x:t; y:t’}. • Record fields are dependent: field names can be referenced by types of later fields. • Example to follow.

  17. Base Types and Records Movie-director Bowling Score (MBS) Format 122Joe|Wright|45|95|79n/aEd|Wood|10|47|31124Chris|Nolan|80|93|85 Burton|30|82|71126George|Lucas|32|62|40 Tim 125 | Pint *Pstring(‘|’) * Pchar

  18. Base Types and Records Bookshelf Listing (BL) Format 13C Programming31Types and Programming Languages20Twenty Years of PLDI36Modern Compiler Implementation in ML 27Elements o f ML Programming 13 C Programming ; title: Pstring_FW(width) } { width: Pint

  19. Constraints • Constrained types: [x:t | e] . • Enforce the constraint e on the underlying type t. 125 Tim | Burton 30 | | 82 | 71 ‘|’ [c:Pchar | c = ‘|’] Pchar ptype Scores = { min:Pint; ‘|’;max: [m:Pint |min≤m]; ‘|’;avg: [a:Pint |min≤a&a≤max] }

  20. Datatypes • Describe alternatives in data source with datatypes. • Parser tries each alternative in order. n/aEd|Wood|10|47|31 124Chris|Nolan|80|93|85 122Joe|Wright|45|95|79 n/aEd|Wood|10|47|31 124Chris|Nolan|80|93|85 125Tim|Burton|30|82|71 126George|Lucas|32|62|40 pdatatype Id = None of “n/a” | Some of Pint

  21. Recursive Datatypes • Describe inductively-defined formats. 79| 31| 71| 40 pdatatypeIntList = Cons of Pint* ‘|’ *IntList | Last of Pint

  22. pdatatype(Elt) List = pdatatypeIntList = Cons of Pint* ‘|’ Cons of Elt* ‘|’ *(Elt) List *IntList | Last of Elt | Last of Pint Polymorphic Types • Parameterize types by other types. ptype IntList = PintList ptype CharList = PcharList

  23. Dependent Types • Parameterize types by values. pdatatypeIntList = Cons of Pint * ‘|’ * IntList | Nil of Pint pdatatype(Elt)List (x:char) = Cons of Elt * x * (Elt) List(x) | Nil of Elt ptype IntListBar = PintList(‘|’) ptype CharListComma = PcharList (‘,’)

  24. More Dependent Types • “Switched” datatypes: pdatatypeGuidedOption (tag: int) = pmatch tag of 0 => Zero of Pstring | 1 => One of Pint | 2 => Two of Pint * Pint | _ => None ptype source = {tag: Pint; payload:GuidedOption (tag)}

  25. PADS/ML Regulus Format: ptype Semicolon = Pcharlit(’;’) ptype Vbar = Pcharlit(’|’) pdatatype Info(alarm_code : int) = Pmatch alarm_code with 5074 -> Details of Details | _ -> Generic of (Nvp_a,Semicolon,Vbar) Plist pdatatype Service = Dom of "DOMESTIC" | Int of "INTERNATIONAL" | Spec of "SPECIAL" ptype Raw_alarm = { alarm : [ i : Puint32 | i = 2 or i = 3]; ’:’; start : Timestamp Popt; ’|’; clear : Timestamp Popt; ’|’; code : Puint32; ’|’; src_dns : SVString Nvp("dns1"); ’;’; dest_dns : SVString Nvp("dns2"); ’|’; info : Info(code); ’|’; service : Service } let checkCorr ra = ... ptype Alarm = [x:Raw_alarm | checkCorr x] ptype Source = (Alarm,Peor,Peof) Plist ptype Timestamp = Ptimestamp_explicit_FW(8, "%H:%M:%S", gmt) ptype Pip = Puint8 * ’.’ * Puint8 * ’.’ * Puint8 * ’.’ * Puint8 ptype (Alpha) Pnvp(p : string -> bool) = { name : [name : Pstring(’=’) | p name]; ’=’; value : Alpha } ptype (Alpha) Nvp(name:string) = Alpha Pnvp(fun s -> s = name) ptype SVString = Pstring_SE("/;|\\|/") ptype Nvp_a = SVString Pnvp(fun _ -> true) ptype Details = { source : Pip Nvp("src_addr"); ’;’; dest : Pip Nvp("dest_addr"); ’;’; start_time : Timestamp Nvp("start_time"); ’;’; end_time : Timestamp Nvp("end_time"); ’;’; cycle_time : Puint32 Nvp("cycle_time") } Sample Regulus Data: 2:3004092508||5001|dns1=abc.com;dns2=xyz.com|c=slow link;w=lost packets|INTERNATIONAL 3:|3004097201|5074|dns1=bob.com;dns2=alice.com|src_addr=192.168.0.10; dst_addr=192.168.23.10;start_time=1234567890;end_time=1234568000;cycle_time=17412|SPECIAL

  26. Outline • Motivation and PADS Overview • Data Description in PADS/ML • Implementation architecture • The Semantic of PADS • Conclusions

  27. Parsing With PADS data description (type T) compiler data rep (type ~ T) 01001001 00111 parser user code parse descriptor (type ~ T)

  28. pdatatype Id = None of “n/a”| Some of Pint datatype Id = None| Some of int ptype MBS-Entry = { id: Id; first: Pstring(‘|’); ‘|’; last: Pstring(‘|’); ‘|’; scores: Scores } type MBS-Entry = { id: Id; first: string; last: string; scores: Scores } Example: MBS Representation n/aEd|Wood|10|47|31

  29. Tool Generation With PADS/ML data description (type T) format- independent tool module compiler data rep (type ~ T) format-specific traversal functor 01001001 00111 parser parse descriptor (type ~ T) tools in this pattern: accumulator, debugger, histograms, clusters, format converters

  30. Types as Modules • PADS/ML generates a module for each type/description • Parameterized types ==> Functors • Recursive types ==> Recursive modules • sigh: combination of recursive modules & functors not supported in O’Caml, so we’re reduced to a bit of a hack for recursion sig type rep type pd fun parser : Pads.handle -> rep * pd module Traverse (tool : TOOL) : sig ... end end

  31. Outline • Motivation and PADS Overview • Data Description in PADS/ML • Implementation architecture • The Semantic of PADS • Conclusions

  32. Motivation • To crystallize design principles. • Example: error counting methodology in PADS/C. • To ensure system correctness. • Example: parsers return data of expected type. • As basis for evolution and experimentation. • Critical to design of PADS/ML. • To communicate core ideas. • Designing the next 700 data description languages.

  33. PADS and DDC • Developed semantic framework based on Data Description Calculus (DDC). • Explains PADS/ML and other languages with DDC. • Give denotational semantics to DDC. PADS/ML PADS/C The Next 700 DDC

  34. Data Description Calculus • DDC: calculus of dependent types for describing data. • Expressions e with type  drawn from F-omega • A kinding judgment specifies well-formed descriptions. t ::= unit | bottom | C(e) | x:t.t | t + t | t & t | {x:t | e} | t seq(t,e,t) | x.e | t e | .t | t t |  | .t | compute (e:) | absorb(t) | scan(t)

  35. Choosing a Semantics • Semantics of REs, CFGs given as sets of strings but fails to account for: • Relationship between internal and external data. • Error handling. • Types of representation and parse descriptor. • DDC • Denotational semantics of types as parsers in F-omega

  36. A 3-Fold Semantics [ {x:t | e} ]rep = [ t ]rep + [ t ]rep [  x:t.t’ ]rep = [ t ]rep * [ t’ ]rep Description Interpretations of t [ t ] t [ t ]rep[ t ]pd [ {x:t | e} ]pd = hdr * [ t ]pd [  x:t.t’ ]pd = hdr * [ t ]pd * [ t’ ]pd Representation  0100100100... Parse Descriptor Parser

  37. Type Correctness Theorem [ t ] : bits[ t ]rep * [ t ]pd Description Interpretations of t [ t ] t [ t ]rep[ t ]pd Representation  0100100100... Parse Descriptor Parser

  38. Outline • Motivation and PADS Overview • Data Description in PADS/ML • Implementation architecture • The Semantic of PADS • Conclusions

  39. Related Work • parser generator technology: • Lex & Yacc • no dependency • semantic actions entwined with data description • no higher-level tools • Parser combinators • semantic actions entwined with data description • no higher-level tools

  40. Reminder:One Description, Many Tools Data Description (Type T) compiler xml translator query engine parser printer visual data browser ... programming library complete application

  41. Parser combinators:One algorithm, One Tool parser

  42. Related Work • Other “data description” languages • Data Format Description Language (DFDL) • Binary Format Description Language (BFD) • PacketTypes [SIGCOMM ’00] • DataScript [GPCE ’02] • None have a well-defined semantics or Pads tool support

  43. Current & Future Work • Tools and Applications • Description inference. • Support for specific domains (microbiology) • Language Design • Transformation language for ad hoc data. • Description language for distributed • Describe locations, versions, timing, relationships, etc. • Theory • Analyze data descriptions for interesting properties, e.g. equivalence, data size, termination, emptiness (always fails). • Coherence of parsing & printing

  44. Summary • The PADS vision: reliable, efficient and effortless ad hoc data processing • PADS/ML: • Data description based on polymorphic, dependent datatypes • “Types as modules” implementation • Solid theoretical basis. • Visit www.padsproj.org

  45. The End Questions?

  46. Existing Approaches • C, Perl, or shell scripts: most popular. • Time consuming & error prone to hand code parsers. • Difficult to maintain (worse than the ad hoc data itself in some cases!). • Often incomplete, particularly with respect to errors. • Error code, if written, swamps main-line computation. • If not written, errors can corrupt “good” data. • Lex & Yacc • Good match for programming languages. • Bad match for ad hoc data. • Compiler converts descriptions into robust, format-specific tools.

  47. Parsing With PADS • Robust parser at the core of generated tools.

  48. Using Ad hoc Data • Can Ed Wood bowl? • Parsing only brings you part way. • Queries must be written in ML. • A lot of work. • What about a declarative query? 122Joe|Wright|45|95|79124Chris|Nolan|80|93|85125Tim|Burton|30|82|71126George|Lucas|32|62|40 n/aEd|Wood|10|47|31

  49. From Ad hoc Data To XML • XML • Encoding for semi-structured data. • Good match! • XQuery • Declarative XML query language for semi-structured sources. • Standardized by W3C, many implementations.

  50. PADX = PADS + XQuery • Galax[Fernandez, et al.] • Complete, open-source XQuery implementation. • PADX • Integrates PADS and Galax. • Supports declarative queries over ad hoc data sources.

More Related