WP-8: Data Handling And Visualization

1. WP-8: Data Handling And Visualization Review Meeting Report Felix Hupfeld, Andrei Hutanu, Andre Merzky, Thorsten Schütt, Brygg Ullmer Zuse-Institute-Berlin

2. Overview • Formal Matters • Visualization • Tangible Devices • Remote File Access • Replica Management / File Movement • Metadata Service

3. Formal Matters • Deliverables: most right on time, some delivered earlier • Milestone 8.2 (M24): Dynamic data access techniques are designed and implemented • Replica catalog (Location independant file access) • Remote File Access (Pitfalls and GridFTP) • Visualization service • Plan for the rest of this year: • Further integration with other WPs • Implement additional features (user management)

4. Formal Matters (II) • B. Ullmer, A. Hutanu, W. Benger, H.-C. Hege. (2003). Emerging Tangible Interfaces for Facilitating Collaborative Immersive Visualization.Workshop on Collaborative Virtual Reality and Visualization, October 2003. • H.-C. Hege, A. Hutanu, R. Kähler, A. Merzky, T. Radke, E. Seidel, B. Ullmer. (2003). Progressive Retrieval and Hierarchical Visualization of Large Remote Data. Workshop on Adaptive Grid Middleware, September 2003. • T. Schütt, A. Merzky, A. Hutanu, F. Schintke, Remote Partial File Access Using Compact Pattern Descriptions, IEEE CCGrid 2004.

5. Visualization context Grid = “virtualization” of resources • Broad progress toward virtualization of hardware resources (compute power, disk space, network) • Less common: virtualization of graphics hardware, software “Virtualization” of visualization resources • Scientific visualization has demanding hardware and software requirements, unavailable to many users • Software is frequently complex and “heavy-weight” • Our approach: • Automation for core visualization operations • Provision of grid-based “visualization services” • Develop simplifying mobile services & devices

6. Visualization and mobile devices Visualization • Access to grid resources from existing viz applications • Access to existing viz applications from the grid Mobile devices • Access to viz servicesfrom mobile phones, PDAs • Development of new grid-enabled “mobile”viz devices

7. Visualization service 1D 2D 3D

8. Mobile + viz = “viz artifacts” Motivation • Advantages of stereo, collaborative interaction • Importance of the meeting room, teleconferencing • Simplifying interaction + gaining real estate by breaking out into the physical world

9. Data cards • Represent online data, user credentials, remote users, parameters, etc.

10. Interaction pads “binding pad” “placement pad” “parameter pad” “control pad”

11. Remote File Access (I) • Needed for • Visualization of remote files (intermediate results of large jobs) • Job input files (extract and transmit only data of interest) • … • Existing mechanisms • GridFTP remote partial file access (read, seek, read, seek, .. ) • I/O wrapper for local and remote access methods (to be used in GAT) But does not fit Grid requirements, because too many small remote operations!

12. Remote File Access (II)  Need to execute high-level operations on the server side • File format specific approach(HDF5) • HDF5 – structured binary file format. • Datasets, Groups, Attributes • Forwards HDF5 calls to the remote side • Plugin for the Globus GridFTP server • Generic approach (much better )  next slide

13. Remote File Access (III) • Generic approach • Compact access pattern description: Pitfalls • Speedup of several orders of magnitude achieved! • Example: Access part of a Cube: (339, 65874, 262144, (0, 511, 2048, 32, (0, 511, 2048, 32, (0, 3, 16, 32)))  just 1 (in words: one!) remote call

14. Replica Management Components • Replica catalog • Supports hierarchical data organization • POSIX filesystem like interface • Uses C++ with SOAP/GSI interface • Adaptor inside of GAT • File movement service • Used for replication • Based on GridFTP • Adaptive service (WP6) • Ranks replica locations • Optimizes GridFTP parameters

15. StorageBox Metadata Service • Unified storage for metadata of persistent Grid objects (archived files, job input/output, messages/notifications) • Any number of namespaces, configurable via pluggable access policies as public or private archives, group collaboration spaces or private annotations for public data Public Sensor Data Archive User ‘A’:Sensor Data Annotations and Private Messages User ‘B’:Sensor Data Annotations and Private Messages Project ‘X’: Collaboration Space A B

16. StorageBox Metadata Service • Portlet as user interface • Integrated with WP12’s MessageBox service • SOAP web service interface • Pluggable database backends • with PostgreSQL implementation • Portable C++ implementation • Win32, Linux, Mac OS X • Planned: • Improved query language / processing • Remote administration interface • More access policies

17. Continuing plans • Visualization, mobile devices • Further integration with portal • Implement additional viz operations • Deploy, test both services & devices w/ end users • Data management • Provide a flexible namespace for organizing files • Remote File Access: Generic vs. File Format Dependant

18. Questions?

19. Visualization and mobile devices Visualization • Access to grid resources from existing viz applications Completed: remote data access from Amira • Access to existing viz applications from the grid Demoable: grid automation, presentation of reports integrating content from gnuplot, OpenDX, Amira;grid access to streaming Amira content Mobile devices • Access to viz services from mobile phones, PDAs Completed + demoable from mobile phones, iPAQ • Development of new grid-enabled “mobile” viz devices Demoable: use of new devices to load remote content, activate and control video conference sessions

20. Videoconferencing example • 24.02.04, Berlin-Munich, surgical planning