Gfdnavi: its design and implementation with Ajax and Ruby-on-Rails

1. Gfdnavi:its design and implementation with Ajax and Ruby-on-Rails Seiya NISHIZAWA,Takeshi HORINOUCHI, Chiemi WATANABE, T. KOSHIRO, A. TOMOBAYASHI, S. OTSUKA, Y. MORIKAWA, Y.-Y. HAYASHI, M. SHIOTANI, and GFD-Dennou Davis project

2. Introduction • What’s “Gfdnavi” • A tool to archive, share, distribute, analyze, and visualize geophysical fluid data and knowledge • desktop use to data provide server • fundamental technologies • Ruby on Rails • GPhys An introduction was done by T. Horinouchi yesterday.

3. Ruby on Rails • an open source web application framework • written in Ruby • Model-View-Controller (MVC) architecture • Convention over Configuration (CoC) • Don't repeat yourself (DRY) • swift development • ActiveRecord • helper methods (HTML, JavaScript, ajax)

4. ActiveRecord (AR) • a part of Rails products • a ruby implementation of the object-relational mapping pattern • Do not need to use SQL • For better performance, SQL can be used on AR. node = Node.find(:first, :conditions=>[”id=?”,2]) # SELECT * FROM nodes WHERE id=2 LIMIT 1; path = node.path #=> “/samples” kas = node.keyword_attributes # SELECT * FROM keyword_attributes WHERE node_id=2; nodes table keyword_attribute table

5. Metadata DB used for search • name-value attributes • geospatial- and time-coordinate information • owner, groups and access mode • link among data • time-stamp, size, etc

6. 1. Name-Value attributes • attributes in data file (self-describing files) • unified access to attributes in differently formatted files with GPhys • attributes in text file • YAML format • any name-value attributes gphys_nc = GPhys::IO.open(“fname.nc”,”T”) # NetCDF gphys_nc.att_names #=> [“long_name”, …] gphys_nc.get_att(“standard_name”) #=> “air_temperature” gphys_grib = GPhys::IO.open(“fname.grib”, “TMP”) # GRIB gphys_grib .att_names #=> [“long_name”, …] gphys_grib.get_att(“standard_name”) #=> ”air_temperatrue” description: NCEP/NCAR reanalysis gfdnavi: owner: user1 other_mode: 0 rgroups: - groupA - groupB

7. YAML • a human-readable data serialization format • easier to read/write than XML Array (list) --- - 0 - 1 - 2 Hash (associative array) --- key1: value0 key0: value1 puts “Array (list)” ary = [0,1,2] puts ary.to_yaml puts “\nHash (associative array)” hash = {“key0”=>”value0”, ”key1”=>”value1”} puts hash.to_yaml Back

8. 2. Geospatial- and time-coordinate information • spatial region • rectangle in longitude-latitude section • temporal region • start time and end time global, regional, or point swath

9. 3. Owner, Groups, and Access mode • permission system like i-node • readable and writable for groups and others • Multiple groups are allowed. 4. Link among data • e.g. This data was calculated from these variables

10. Directory tree structure • nodes in the tree structure • node types: directories, variables, images, knowledges, etc • Each node can have some metadata. • inherited to children nodes variables data files groups local or opendap directories attributes description = “……..” param1 = value1 param2 = [value21,value22] attributes description = “……..” owner = userA groups = [groupA,groupB] virtual aggregated files image files

11. Analysis and Visualization draw yes Analysis model cached get cache analysis no fork a child process DB GPhys storage open data file(s) cache output calculate draw output NetCDF PNG programmable

12. Analysis model (Analysis class) • all the parameters for analysis or visualization • the form in the analysis page • instance variables of the Analysis object • It is able to construct one from the other • enable to reconstruct the analysis page from • drawn image • history list

13. Draw method and analysis function are not hard-coded. • Their definitions are in YAML files (editable) • one method in one file can create and modify via web-browser spectrum.yml :name: spectrum :description: |FFT|^2 along a specific dimension :nvars: 1 :script: | [gphys0.fft(*arg0).abs ** 2] :arguments: - :description: the dimensions for spectrum :value_type: array_string :default: [] simple coding due to GPhys

14. examples of original draw methods in a Gfdnavi server providing an ensemble forecast data

15. User Interface • bottleneck of network application • network bandwidth • machine power and system load of the server • better usability • ajax • Rails has many helper methods to write HTML and JavaScript to use ajax. • cache

16. Animation

17. Web service • local programming • cross-site use • other Gfdnavi servers • non-Gfdnavi servers • SOAP • APIs for all the analysis functions and draw methods • use the Analysis class • WSDL • (REST)

18. Summary • Metadata and Directory tree structure • attributes in self-describing data files and YAML files • inheritance • unified access to attributes with GPhys • easy and swift development with AcriveRecord (Rails) • Analysis/Visualization • programmable (with text editor or web-browser) • easy and extensible coding with GPhys • User Interface • good usability with ajax and cache • easy development with helper methods (Rails)

19. Thank you

21. MVC architecture • Model • Relational Database • ActiveRecord: a ruby object that wraps a row in a database table • View • HTML • ERuby: a templating system that embeds Ruby into a text document • JavaScript • ajax • Controller

22. Gfdnavi server G Controller User Web Browser request Web server HTML JavaScript create, update, delete analysis, draw Model ActiveRecord, VizShot View ERuby helper methods RDB GPhys DCL

24. ActiveRecord (AR) • a part of Rails products • a ruby implementation of the object-relational mapping pattern • a row in a table ↔ an instance object • a column ↔ an instance method • a reference between tables ↔ an instance method part = Part.new part.name = “Sample part” part.price = 123.45 part.save INSERT INTO parts (name, price) \ values (‘Sample part’, 123.45); name = “gearbox” part = Part.find(:first, :conditions=>[“name=?”,name]) SELECT * FROM parts \ WHERE name=‘gearbox’ LIMIT 1;

25. RDB tables • Directory tree structure • nodes (basic unit of a tree) • directories, variables, images, knowledges, etc. • Meta data • keyword_attributes • spatial_and_temporal_attributes • node_relations • User’s information • users • groups

26. carts items • instance method is used for reference between two tables with foreign key class Cart < AcriveRecord::Base has_many :items end class Item < AcriveRecord::Base belongs_to :cart end cart = Cart.find_by_user(“userA”) #=> cart whose # user = “userA” items = cart.items #=> array of # items whose cart_id = 1 items[0].name #=> “orenge” items[1].amount #=> 2

27. Directory tree structure • Tree structure is good way to organize data /…/data_top_dir/ncep.yml /ncep/UV.jan.nc (contains U and V) /UV_500hPa_vect.png /UV.jan.nc.SIGEN /UV_500hPa_vect.png.yml

28. acts_as_tree method of AR • “parent” column is assumed to have parent’s “id” • “parent” method return the parent node • “children” method return the array of children nodes class Node < AcriveRecord::Base acts_as_tree end node = Node.find(3) #=> node whose id=3 node.children #=> array of nodes # whose parent column is 3 node.parent #=> node whose id is value of parent column

29. two types of meta data • meta data in data file (self-describing file) • NetCDF: variable attributes variables: float U(level,lat,lon) ; U:long_name = “zonal velocity“ ; U:units = “m/s” ; U:missing_value = -999.f ; U:standard_name = “eastward_wind” ; U:statistic = “Long Term Mean\n” ; float V(level,lat,lon) ; V:long_name = “meridional velocity” ; V:units = “m/s” ; V:missing_value = -999.f ; V:standard_name = “northward wind” ; V:statistic = “Long Term Mean\n” ; float lon(lon) ; : snip

30. description: NCEP/NCAR reanalysis gfdnavi: owner: user1 other_mode: 0 rgroups: - groupA - groupB • meta data in YAML and SIGEN files • ncep.yml • UV_500hPa_vect.png.yml description: vector plot of horizontal velocity at 500hPa gfdnavi: owner: user2 references: - /ncep/UV.jan.nc/U - /ncep/UV.jan.nc/V other_mode: 0 rgroups: - groupA

31. keyword_attributes table

32. spatial_and_temporal_attributes table • node_relations table • UV_500hPa_vect.png (node id = 6) was drawn from U (node id=4) and V (node id = 5)

33. user and groups • root, user1(groupA,groupB), user2(groupA), user3 • access permission • owner • root: “/”, “/ncep” • user1: “/ncep/UV.jan.nc”, “/ncep/UV.jan.nc/(U|V)” • user2: “/ncep/UV_500hPa_vect.png” • permission for reading • everyone: “/” • groupA and groupB: “/ncep“, “/ncep/UV.jan.nc/(U|V)” • groupA: “/ncep/UV_500hPa_vect.png”

34. users and groups tables • nodes table The first bit is the flag for the group of id=1 (groupA), the second one is for that of id=2 (groupB), and so on. 2: writable, 4: readable, 6:both They seem to be permitted to be accessed by anyone, but it is not so, because the permission for “/ncep” is inherited.

35. Permission of a node depends on all its ancestors. • calculated in advance • saved at additional Boolean-type columns • In order to keep consistency, callbacks can be added to events (e.g. after saving) with AR.

36. When permission is changed, the columns of its children are recalculated. • Using ActiveRecord, adding such callbacks to events (e.g. after saving) is quite easy. class Node < ActiveRecord::Base after_save calculate_children_permissions # add a callback method def calculate_permissions # calculation of permission other_readable = (other_mode & 4) & parent.other_readable groups_readable = rgroups & parent.groups_readable save! # save to database table end def calculate_children_permissions # callback method children.each{|child| child.calculate_permissions } end end

37. ER diagram of RDB tables (only important tables and columns are shown)

38. Registering script • Directory tree structure on a storage is registered to the database table as it is. • The top directory to be registered can be specified. • Metadata contained in data files are registered to the database table. • Any other metadata can be registered by writing in Yaml and SIGEN files.

40. Analysis and Visualization • GPhys • IO handling • data slice • mathematical calculation • DCL (Dennou-Club Library) • plot • (Cache) • drawn plots are cached to DB

41. ajax request to get children directories click The ajax request will not be called again. Opening and closing the directory is done by changing “display” attribute of CSS. Directory tree view

43. RESTful • Representational state transfer • Uniform interface • All resource share a uniform interface like HTTP methods such as GET, POST, PUT, and DELETE. • Resource Oriented Architecture • Every resources are uniquely addressable using URI. • Statelessness • Each request is treated independently.

44. improvable • still some separate codes • web application, web service • SOAP ignore HTTP methods. • Some cache systems could not work well. • Programming with SOAP tend to be complicated. “RESTful web service + client(s)” • one unified code • Multiple output formats are still supported (HTML, XML). • acts as definition of HTTP methods • cache system friendly • simpler local programming

45. RESTful webservice • URI syntax • variable • http://host/…/ncep/UV.jan.nc/U.(html|xml) • http://host…/ncep/UV.jan.nc/children.(html|xml) # U and V • analysis • http://host/ .../{variables}/analysis/{function}.(html|xml)?{options} • drawing • http://host/ .../{variables}/draw/{draw method}.(html|xml)?{options}

46. clients • flex • ActionScript • (AIR) • (HTML + JavaScript) • (desktop ruby client) • another gfdnavi server

48. UI with ajax • asynchronous JavaScript and XML (originally) • not only JavaScript and XML, but also other pairs • update a part of the existing page • better usability • reduce bandwidth usage • Rails has many helper methods to write HTML and JavaScript to use ajax.

49. click get node list of the directory vie ajax call

50. Analysis page click check send values in the form with ajax get dimension’s information with ajax