1 / 21

Implementing Unified Access to Scientific Data from .NET Platform

Moscow State University Department of Computational Mathematics and Cybernetics. Implementing Unified Access to Scientific Data from .NET Platform. Sergey B. Berezin Dmitriy V. Voitsekhovskiy Vilen M. Paskonov. Supported by Student Laboratory of Microsoft Technologies and RFBR grants.

galena-barlow
Download Presentation

Implementing Unified Access to Scientific Data from .NET Platform

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Moscow State University Department of Computational Mathematics and Cybernetics Implementing Unified Access to Scientific Datafrom .NET Platform Sergey B. Berezin Dmitriy V. Voitsekhovskiy Vilen M. Paskonov Supported by Student Laboratory of Microsoft Technologies and RFBR grants

  2. Different languages, common tools Viscous fluid flow visualization via vector fields and color maps (http://www.cs.msu.su) Seismic data visualization via isosurfaces (http://www.sci.utah.edu) Tensor field visualization for diffusion through biological tissue (http://www.sci.utah.edu)

  3. Viscous fluid flow visualization via vector fields and color maps (http://www.cs.msu.su) Tensor field visualization for diffusion through biological tissue (http://www.sci.utah.edu) Scientific data access requirements We need to: • Retrieve typed data object without regard where it is stored and how it is stored. • Physical data independence • Retrieve partial data when needed • Filtering & Caching • Retrieve data description • Metadata support Seismic data visualization via isosurfaces (http://www.sci.utah.edu) We don’t want to: • Rewrite existing computational software • Use existing formats • Install new system software • Use existing protocols

  4. Scientific data access today

  5. What’s so special in scientific data? • Have a complex structure; • Parameterized by … • Time • Sampling point coordinates • More complex parameters; • Stored in many files of various formats • Have very large size of individual data items • Don’t fit well to relational model! Scientific data …

  6. What is DataSet?

  7. Example: Accessing data in C# // Retrieve the DataSet object from a serverby GUID DataSet dataset = DataSet.Open("http://regatta.cs.msu.su:9111", "767c57b1-801e-4784-bbd6-287707fd0ec2"); // Fetching DataItem by name. // DataItem may be either simple or composite, it doesn’t matter DataItemxVelocity =dataset.DataItems["u-values"]; // Creating parameter corresponding to time moment = 0.0 CompositeParameter param =newCompositeParameter( newParameterValue("time", 0.0d) ); // Fetching DataItemSlice for the parameter. // It is an instance of DataItem for specified parameter value. DataItemSlice dataVelocity = xVelocity[param]; // Getting required data: velocity array for time = 0.0 ScalarArray3d data =dataVelocity.GetData() asScalarArray3d;

  8. DataRequest: communicating with server <soap:Envelope … ><soap:Body> <dataRequestdataSource="…" dataSet="767c57b1-801e-4784-bbd6-287707fd0ec2" … > <dataItemtype=“ScalarArray3d"> <dataSourcesourceName="u0000.cdf"sourceType="netCDF" sourceParameters="u" /> </dataItem> </dataRequest> </soap:Body></soap:Envelope> • The following DataRequest is sent to a server as the result of the previous example: • The following DataRequest is received from the server: <soap:Envelope … ><soap:Body> <dataRequestdataSource="…" dataSet="767c57b1-801e-4784-bbd6-287707fd0ec2" … > <dataItemtype=“ScalarArray3d"> <dataRefsourceType=“u0000.cdf" sourceParameters=“u"> <remoteurl="scp://regatta.cs.msu.su/datasource/pvm/Re1000/u0000.cdf“ /> </dataRef> </dataItem> </dataRequest> </soap:Body></soap:Envelope>

  9. Complex structures in DataSet x file with scalar array scalar array data item X,Y,Z file with scalar array scalar array data item vector array data item file with scalar array scalar array data item vector array constructor vector field data item file with spatial grid spatial grid data item data field constructor

  10. Example: Accessing composite data in C# // Retrieve the DataSet object from a serverby GUID DataSet dataset = DataSet.Open("http://regatta.cs.msu.su:9111", "767c57b1-801e-4784-bbd6-287707fd0ec2"); // Fetching DataItem by name. // DataItem may be either simple or composite, it doesn’t matter DataItemvelocity =dataset.DataItems["uvw-values"]; // Creating parameter corresponding to time moment = 0.0 CompositeParameter param =newCompositeParameter( newParameterValue("time", 0.0d) ); // Fetching DataItemSlice for the parameter. // It is an instance of DataItem for specified parameter value. DataItemSlice dataVelocity =velocity[param]; // Getting required data: velocity array for time = 0.0 Vector3dArray3d data =dataVelocity.GetData() asVector3dArray3d;

  11. DataRequest: composite data items <soap:Envelope … ><soap:Body> <dataRequestdataSource="…" dataSet="767c57b1-801e-4784-bbd6-287707fd0ec2" … > <dataItemtype="Vector3dArray3d"><!–- This is a COMPOSITE DATAITEM! --> <compositeconstructor="CompositeVectorArray"> <component><!-- u-values --> <dataItemtype="ScalarArray3d"> <dataSourcesourceName="u0000.cdf"sourceType="netCDF" sourceParameters="u" /> </dataItem> </component> <component> … </component> <!-- v --> <component> … </component> <!-- w --> </composite></dataItem></dataRequest> </soap:Body></soap:Envelope> • The following DataRequest is sent to a server as the result of the previous example execution: • The following DataRequest is received from server: <soap:Envelope … ><soap:Body> <dataRequestdataSource="…" dataSet="767c57b1-801e-4784-bbd6-287707fd0ec2" … > <dataItemtype=“Vector3dArray3d"> <dataRefsourceType="plain binary" sourceParameters=""> <remoteurl="scp://regatta.cs.msu.su/datasource/~Fdg4gBd“ /> </dataRef> </dataItem> </dataRequest> </soap:Body></soap:Envelope>

  12. Filtering • Filtering allows transfer of only required data from server to client • Filtering may be performed both by a client-side and a server-side of the system. • Examples of the filtering are cropping and thinning of large vector fields. cropping filter [0.4,0.76] x [0.4,0.76] thinning filter (0.1,0.1) 0.76 1 100 x 100 2d vectors 120KB 3000 x 3000 2d vectors 108 МB 0.4 0.76 1000 x 1000 2d vectors 12 MB 0.4 0.76 0 1 0.4 0.4 0.76

  13. Example: Filtering data in C# // Initializing the DataSet object from a serverby its GUID DataSet dataset = DataSet.Open("http://regatta.cs.msu.su:9111", "767c57b1-801e-4784-bbd6-287707fd0ec2"); // Fetching DataItem by its name. It may be either simple or composite DataItem velocity =dataset.DataItems["uvw-values"]; // Creating parameter corresponding to time moment = 0.0 CompositeParameter param =newCompositeParameter( newParameterValue("time", 0.0d) ); // Fetching DataItemSlice for the parameter. DataItemSlice dataVelocity =velocity[param]; // Creating filter "Thinner" for a type of the velocity data item // and setting up its parameters IThinner3dFilter filter = FilterFactory.GetFilter("Thinner", dataVelocity.TypeDescriptor) asIThinner3dFilter; filter.PercentageX = 0.05; filter.PercentageY = 0.05; filter.PercentageZ = 0.05; // Getting required data: thinned out velocity array for time = 0.0 Vector3dArray3d data =dataVelocity.GetData(filter)asVector3dArray3d;

  14. DataRequest: communicating with server <soap:Envelope … ><soap:Body> <dataRequestdataSource="…" dataSet="767c57b1-801e-4784-bbd6-287707fd0ec2" … > <filtername="Thinner"> <parameters> <parametername=“PercentageX” value=“0.05” type=“double” /> <parametername=“PercentageY” value=“0.05” type=“double” /> <parametername=“PercentageZ” value=“0.05” type=“double” /> </parameters> <dataItemtype="Vector3dArray3d"><!–- This is a COMPOSITE DATAITEM! --> <compositeconstructor="CompositeVectorArray"> <component><!-- u-values --> <dataItemtype="ScalarArray3d"> <dataSourcesourceName="u0000.cdf"sourceType="netCDF" sourceParameters="u" /> </dataItem> </component> <component> … </component> <!-- v --> <component> … </component> <!-- w --> </composite></dataItem> </filter> </dataRequest> </soap:Body></soap:Envelope> • The following DataRequest is sent to a server as the result of the previous example execution: • The returned DataRequest is similar in this case to the returned DataRequest from the previous example.

  15. Caching • Both server-side and client-side of the system cache the results of a successful DataRequest execution. • Server-side cache filtering results • Client-side cache retrieved data items and results of DataRequest

  16. How DataRequest is performed?

  17. Deployment Scenario • More sophisticated scenario includes development of distributed data sources that provide scientific data. • Dedicated servers will act as data processor performing data filtering and transformations • Dedicated servers will act as dataregistries allowing DataSet enumeration and querying in entire global network. • This will make possible to create dynamic data libraries of researches and enables easy data publishing • The simplest scenario is as follows:

  18. Why .NET? • Object-oriented data access requires an object-oriented platform to be built on. • High extensibility is based on CLR dynamic nature • New data types • New filters • New parsers • No built in data types, filters, parsers • .NET opens new horizons with LINQ, WPF,…

  19. Future work • Transferring files from remote server is just an example of DataProvider • Extend architecture for new types of data providers • LINQ technology will make data access from C# much more elegant. • Development of easy-to-use data management applications for the proposed approach. • Development of an innovative visualization system, highly extensible and customizable • Or integrate our approach with existing one

  20. View 1 C A Control A Control B Control C BA View 1.2 View 1.1 Control D D View 1.1.1 Future visualization system Step 1. Choose object of interest Step 2.Choose data transform Step 3. Choose visualization algorithm Example

  21. Questions? • Visit: http://microsoft.cs.msu.su/projects/uvs • Mail to: s_berezin@cs.msu.su

More Related