1 / 28

VisIVO: data visualization on the grid Scientific Multidimensional Data Exploration

VisIVO: data visualization on the grid Scientific Multidimensional Data Exploration. 4th EGEE User Forum/ OGF 25 and OGF Europe's 2nd International Event 2-6 March 2009 Catania. Ugo Becciani , A. Costa, G. Caniglia, C. Gheller M. Comparato, P. Massimino, M. Krokos, L. Zef Han.

draco
Download Presentation

VisIVO: data visualization on the grid Scientific Multidimensional Data Exploration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VisIVO: data visualization on the grid Scientific Multidimensional Data Exploration 4th EGEE User Forum/ OGF 25 and OGF Europe's 2nd International Event 2-6 March 2009 Catania Ugo Becciani, A. Costa, G. Caniglia, C. Gheller M. Comparato, P. Massimino, M. Krokos, L. Zef Han

  2. VisIVO Desktop VisIVOServer VisIVOWeb --fformat votable /home/user/demo/vizier.xml ..... --x x --y y --z z --color --colortable --colorscalar scalar0 --glyphs sphere Grid environment Pc GNU/Linux Interactive fast navigation Closely integrated, complementary and independent !

  3. VisIVOdesktop is a visualisation package developed in collaboration between INAF (Catania Astrophysical Observatory), CINECA (the largest Italian academic supercomputing center) and University of Portsmouth, with the specific object of supporting visualisation and analysis of astrophysical data. It was developed in the Italian participation in the VO-TECH project. VisIVODesktop • Free Software • Cross Platform: MacOSX Windows and Linux (Ubuntu , Debian, Fedora), • N-Dim (>3) Visualization of Astrophysical data (lut and glyphs => scalar properties of each element) • Point, Volume and Vector Visualization • Astrophysical Data Analysis • VO (Virtual Observatory) compliance (Interoperability: VOT, PLASTIC,VizieR) • Easy to use: intuitive menu drives the user.

  4. VisIVO Desktop • Input file formats • Ascii, CSV • Binary • HDF5 • FITS • Gadget • VOTable Uniform management All internal data are represented like a table, a sequence of float values (VisIVO Binary Table - VBT)

  5. Visualisations Navigation -- Zoom -- Lookup table -- Algorithms -- Data selection -- Picker op. -- Interoperability

  6. LOD Level Of Detail: while interacting with the visualized data (i.e. zooming and rotating) a simplified graphical representation is used in order to keep the manipulation interactive

  7. Data manipulation • New Fields can be created as mathematical functions of existing fields • Tables can be merged • Subsets can be extracted using different sampling techniques

  8. Vectors Vector Fields can be created from a VBT selecting any loaded fields as its three components

  9. VisIVO Server on the Grid • Non-interactive command line application that implements visualisation functionalities; its output is a static 2D image of a 3D object • Easy integration in Virtual Observatory complaint Web Services and in the Grid environment • It will provide the user with a 3D preview of huge data exploiting the powerful facilities of the Grid environment • Easy to use • Open Source code: project maintained on sourceforge. • NO LIMIT on data size !

  10. VisIVO Server Basic Architecture TVO XML Document Local or Remote (http ref) User Data VBT: VisIVO Binary Table New VBT

  11. VisIVO Importer Input: Local or Remote file: ASCII-CSV, Binary, VOTable, FLY, Gadget, TVO XML Document , Fits Table, HDF5 Op: Data format conversion: VBTs creation Output: VBTs (optimised for VisIVO Filters and Viewer). NOTE: No limit on data dimension for input data

  12. VisIVO Server VBT: VisIVO Binary Table Ascii Header (.bin.head):contains basic metadata information. float 6 16777216 little X Y Z Vx... Internal data rappresentation: float Number of fields Number of elements of each field (including mesh data for volumes) Data format identifier: little endian List of field names Binary File (.bin):a sequence of binary arrays (fields).

  13. VisIVO Importer VisIVOImporter  --fformat filetype  [--out name] [--volume] [--compx X --compy Y --compz Z] [--sizex DX --sizey DY --sizez DZ] [--bigendian] [--double] [--npoints N] filename --fformatspecifies the input file data format --out [name] (optional) output filename. Default name VisIVOServerBinary.bin --volume (optional) used to create a table that contains Volume data --compx[double] --compy [double] --compz [double] (optional)  If data is a Volume the user must enter the mesh geometry. If data size fits a cubic mesh these values are automatically computed. --sizex [double] --sizey [double] --sizez [double] (optional). Cell geometry  Default values 1.0 1.0 1.0 --bigendian(optional) Used only for big Endian Gadget and FLY input  file. Default files are little Endian. --double (optional) Used only for double data of FLY input  file. Default value is float --npoints (optional) Used only for  FLY input  file. Number of data points.

  14. VisIVO Filters Input: AVBT data file Op: It creates a new VBT or gives info on specified fields Output: a new VBT or other info NOTE: it must be applied to visualise huge data General Syntax: VisIVOFilters --op filterOpCode <options> InputFile (VBT file)

  15. VisIVO Filters Append Tables: Creates a new table appending data from a list of existing tables Coarse Volume: Produces a coarsed subvolume with plane extraction from the original volumeDecimator: Creates a subtable as a regular subsample from the input table.Extract Subregion: Creates a new table from an input table: sub-box or sphere extraction.Extract Subvolume: Produces a table which represents a subvolume from the original volumeMath. Operations: Create a new field in a data table as the result of a mathematical operation between existing fieldsMerge Tables: Create a new table from two or more existing data tablesPoint Distribution: Creates a volume using a field distribution (CIC algorithm) on a regular mesh Point Property: Assigns a property using a field distributionRandomizer: Creates a random subset from the original data tableSelect Fields: Creates a new table using one or more fields of a data table Select Rows: Creates a new table using limits  on one or more fields of a data table Show Table: Produces an ASCII table of selected fields Statistic: Creates and plots an histogram of a scalar field in the table

  16. VisIVO Viewer VisIVO Viewer is a command line application that produces 3D images from the binary internal data format table (VBT) The user must specify three fields of the table for 3D representation. The user can also specifiy the following main options: • Camera position (azimuth and elevation) • Zoom level • Transparency / Opacity • Point shape (pixel/sphere, cube, cone etc..) • Lookup table • ..........

  17. VisIVO Viewer General Syntax VisIVOViewer <options> inputFile (VBT file) --volume (for volume representation) Input: A VBT data file Op: It creates one or more view of data Output: png files LIMITS: the input VBT data file must fit the available RAM

  18. VisIVOViewer • Default Output: 4 Images: • Az=0 - Elev=0 • Az=90 - Elev=0 • Az=0 - Elev=90 • Az=45 -Elev=45 Customized Image fixing: Az, Elev, Transparency....

  19. Cometa Consortium COnsorzio Multi Ente per la promozione e l'adozione di Tecnologie di calcolo Avanzato PI2S2 Project Fundedby the Italian MUR using EU fundsforObjective 1

  20. The Sicilian Grid in one slide

  21. Available computing and storage resources • ~2500 cores AMD Opteron 2218 • 2 GB RAM per core • LSF as LRMS everywhere • Infiniband-4X at all sites gLite 3.0 middleware everywhere • 200+ TB of raw disk storage • Distributed/parallel GPFS filesystem

  22. PI2S2 infrastructure and HPC • Each PI2S2 site has a low latency Infiniband network • Intel fortran and PGI compilers are available • UI “on-board”: • share all resources with the WNs • available remote login • it is possible to compile and test codes • HPC RUN • A special hpc queue is available. It allows the CPU core reservation • It is possible to use ALL nodes of a PI2S2 site (more than 200 CPU cores)

  23. PI2S2 infrastructure and HPC • PI2S2 Grid infrastructure supports the following MPI versions each MPI version has been built with different compilers. The supported tags are : Running Using Gigabit Network • MPICH_GCC4 • MPICH_INTEL9 • MPICH_PGI706 • MPICH_SH_GCC4 • MPICH_SH_INTEL9 • MPICH_SH_PGI706 • MPICH2_GCC4 • MPICH2_INTEL9 • MPICH2_PGI706 • MVAPICH_GCC4 • MVAPICH_INTEL9 • MVAPICH_PGI706 • MVAPICH2_GCC4 • MVAPICH2_INTEL9 • MVAPICH2_PGI706 Running Using Inifiband 4x Network

  24. PI2S2 infrastructure and HPC Type = "Job"; JobType = "MVAPICH2"; Executable = “MPIparallel_exec”; NodeNumber = 200; Arguments = “arg1 arg2 arg3"; StdOutput = "test.out"; StdError = "test.err"; InputSandbox = {“mpi.pre.sh”,“mpi.post.sh”, “MPIparallel_exec”, “watchdog.sh”}; OutputSandbox = {“test.err”, “test.out” } Executable Pre e Post Processing Scripts It runs during the parallel execution

  25. VisIVOServer HPC and Visualisation on the Grid Independent job - Can be executed on the Grid - Produces images (on the grid Catalog) - Visualisation of large dataset during the data production phase Sub-process of the watchdog procedure Data Image, obtained with VisIVO Server, of a cosmological half-billion bodies simulation, running since January 9th on the Cometa Grid

  26. VisIVOWeb http://visivoserver.oact.inaf.it

  27. VisIVOWeb NEWs: Movies can be created

  28. Gabriella Caniglia Software Developer Astrophysical Observatory Catania Italy Marco Comparato Software Manager/Developer Astrophysical Observatory Catania Italy Ugo Becciani Principal Investigator Astrophysical Observatory Catania Italy Alessandro Costa Web Services & TVO Astrophysical Observatory Catania Italy Zef Han J. Software developer University of Portsmouth UK Mel Krokos Co-Principal Investigator University of Portsmouth UK • Thanks! PietroMassimino Web developer Astrophysical Observatory Catania Italy • http://visivo.oact.inaf.it • http://visivoserver.oact.inaf.it Claudio Gheller Co-Principal Investigator Cineca Bologna Italy

More Related