1 / 14

Data Product Suite

Design, implement, and assimilate new data products efficiently with over 15K daily images and 1GB data files. Maintain changing grids, file formats, and resource constraints through algebraic optimization for data independence.

bettybrooks
Download Presentation

Data Product Suite

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Product Suite • Problem: • Design, Implementation, and Assimilation of new data products is expensive. • Constraints: • Efficiency • 15K+ daily images, 1GB+ data files • Maintainability • changing grids: quad cells • changing file formats: removed null values • changing resource constraints: new machines Algebraic Optimization Data Independence

  2. Data Independence We want: • Different algorithms, each dependent on data characteristics. • Changes to data characteristics mean changes to recipe • Logical equivalences are obscured… vtkExtractGeometry vtkThreshold vtkExtractGrid vtkExtractVOI vtkThresholdPoints VTK:

  3. cross restrict cross restrict Exposing Equivalences for Algebraic Optimization result(A) A: • result(B) = result(A), but cost(B) < cost(A) • Implies ‘Cross’ and ‘Restrict’ should be commutative • Difficult to determine which library routines, if any, have this property (“Documentational Semantics”) result(B) B:

  4. 3D Data Products F G H(x,y,b)  r(z>b) r(region) b(s) V(z) F r(x,y) H(x,y,b)  r(z>b) b(s) r(z) V(z)

  5. Horizontal Slice F H H(x,y,b)  r(z>b) slice b(s) V(z) r(z>b) b(s) apply H(x,y,b) <depth>

  6. Vertical Slice F H(x,y,b)  r(z>b) search b(s) V(z)  P  V P P P  V

  7. Vertical Slice H(x,y,b) search  b(s) first P V(z) H(x,y,b) search  b(s) interp P V(z) r(x,y) H(x,y,b)  r(z>b) b(s) interp r(z) V(z) search  P V(z)

  8. Plume Volume F H(x,y,b)  r(region) r(z>b) b(s) V(z) r(salt<26) area merge sum H(x,y,b)  r(ocean) apply(*) length Time dependent V(z)

  9. Another Optimization A=B r(salt<26) A merge C r(ocean) B A merge r(salt<26) r(ocean) C B A merge r(salt<26) C r(ocean) B B’

  10. Software APIs API

  11. Data Independence We want: Custom: makeSmallGrid() vtkExtractGeometry vtkExtractStructuredGrid vtkExtractThreshold vtkExtractGrid vtkExtractVOI vtkThresholdPoints0 VTK:

More Related