Matthew Laurenson and Seishi Ninomiya National Agricultural Research Center Tsukuba Japan

1. Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National Agricultural Research Center Tsukuba Japan

2. Contents • Concepts • Remote Procedure Calls (RPC) • Brokers • Using a broker • Metadata requests • Data requests • Issues • Linking your own data • SOAP • New brokers and future enhancements

3. Overall Goal: To help you make “portable” software applications (DSS).

4. Current Situation: Web-based but… • Applications closely coupled to databases • Limited range of applications at each site • Duplicated development at each site

5. Connectors for local data Ag and NR Grid – a “Hotel Room” for Agricultural and Natural Resource DSS User interface in national language Room DSS DSS DSS Soils Maps Weather Elevation

6. New Zealand Maps Soils Elevation Weather

7. 日本 地図 地 地面の高さ 気象

8. 中国 土壤 交通图 气象 海拔

9. Goal of Ag and NR Grid:To provide this “hotel room” in your country, so you can use models developed by others (and they can use your models).

10. Distributed Computing Concepts • Remote Procedure Calls • Brokers to handle heterogeneous databases

11. Remote Procedure Calls (RPC) • RPCs let a program execute a procedure on a remote computer as if it were local.

12. My Computer remote=findRemote() A=5 B=6 C=remote.add(A,B) Print C ... Remote Computer procedure add(x,y) return x+y } ... Remote Procedure Call

13. Remote Procedure Calls (RPC) • RPCs let a program execute a procedure on a remote computer as if it were local. • Newer RPC mechanisms let programs send and receive data objects. • RPC mechanisms: CORBA, .NET, Java RMI, SOAP... • RMI – only Java programs can use • SOAP – used by other languages as well

14. RPC vs Local Calls • Latency (delay) • RPCs much slower than local calls • difficult to predict how long call will take • May fail • network problems • problems with other computer (“server”) • Security issues • firewalls...

15. But... • Can distribute processing and data over multiple machines • Great when multiple users must share the same dynamic data item or communicate with each other.

16. Mediated Architecture – “Brokers” Domain Databases (Heterogeneous) Consistent data access (using RPC mechanism) Applications Broker

17. MetBroker Weather Databases Applications Data Retrieval Servlet South Africa Web Browsers Data Retrieval Korea Crop Growth MetBroker Japan Disease Risk New Zealand Weather Risk USA

18. Contents • Concepts • Remote Procedure Calls (RPC) • Brokers • Using a broker • Metadata requests • Data requests • Issues • Linking your own data • SOAP • New brokers and future enhancements

19. Data and Metadata Requests Interaction between applications and brokers uses objects – data packages Metadata request Standard request DB-specific query Broker Application Database (various) Metadata result Standard result DB-specific result Metadata Data

20. Example - MetBroker Services • Weather Data • Single station • All stations in a region • Metadata (data about data) • Database details (coverage of globe, login req.) • Regions within databases • Station details (latitude, longitude, start date, end date, elements recorded, resolution...)

21. Elements of a Station Data Request Weather station Username/password Time interval Resolution Required weather elements Allow summarizing

22. Send Request to MetBroker and Receive Result  Data Request amedas 40341 rain, air temperature 1999/1/1-1999/2/1 daily OK to summarize Data Result Tsuchiura Operated 1976-present 1999/1/3 06:00-1999/2/1 00:00 10.1, -0.2, 0 12.6, 3.8, 0 11.3, -1.5, 0.3 ... To MetBroker

23. Query Results - Single Station Station & query details Station details and data sequences are encapsulated in a single result object rain air temp. solar rad.

24. Spatial Request Client Broker Temperature Rainfall Radiation

25. AMeDAS Wakayama Rainfall DB Background map dynamically retrieved from http://www.mapblast.com Spatial Request Daily max/min temperatures & rainfall for all stations in region

26. Query results - multiple stations

27. Using Brokers – Implementation Issues

28. RMI Blocked by Firewalls Wrapper Servlet ClientApplication Java RMI HTTP Broker Client-side firewalls RMI ClientApplication Java Server Machine

29. Applet Using Multiple Brokers DEMBroker Elevation data MetBroker ChizuBroker Background maps Meteorological data Spatial Risk Applet Application text Resource Server

30. Elevation data Background maps Meteorological data Application text Without Multiple Threads - Sequential DEMBroker MetBroker ChizuBroker Spatial Risk Applet Resource Server 12 seconds

31. Elevation data Background maps Meteorological data Application text With Multiple Threads – Parallel DEMBroker MetBroker ChizuBroker Spatial Risk Applet Resource Server 5 seconds

32. JavaBeans for Java GUI programs • For rapid GUI development in Java can use existing MetBeans • Transparently handle firewalls, latency... • Making a simple data retrieval application takes about 30 minutes • Used in many MetBroker demo applications

33. Easy Development of New Applications

34. Databases Linked Using “Drivers” Databases Applications Broker

35. Linking a Database to a Broker • Need to write one Java class (a “driver”) with two or three main methods • Typically use Java Database Connectivity (JDBC) drivers to query relational databases • Can reuse existing classes for Oracle, Postgres, and MySQL JDBC. • Drivers can read data from files via HTTP, or parse HTML tables. • “Test harness” available for driver development

36. How much work? • Add a new database to MetBroker • roughly working - 3 days • tested more thoroughly – 1 week • (assuming that database is readily accessible) • Make a MetBroker applet • working prototype in one day • (assuming basic science is “done”)

37. Developer Resources • Demos, documentation, downloads, pictures • http://www.agmodel.net • Mailing lists – subscribe at web site • Brokers are open source under the GNU Lesser General Public License. • Can be used in commercial applications • Any improvements must be released under same license

38. Reminders • Linking your data to the grid lets you use grid tools developed by others. • It doesn’t mean you must share your data with everyone (password protection). • The key to interoperability is the interface

39. Contents • Concepts • Remote Procedure Calls (RPC) • Brokers • Using a broker • Metadata requests • Data requests • Issues • Linking your own data • SOAP • New brokers and future enhancements

40. SOAP • An XML-based RPC mechanism • Independent of programming language • SOAP toolkits for C++, Visual Basic, Delphi, Java, ASP, PHP, Perl, Python… • Firewall-friendly • Basis of Microsoft .NET architecture • A SOAP Web service is described in standard format using a .WSDL file(Web Services Description Language)

41. Example - MetSOAP Visual Basic client application VB function calls MS SOAP Toolkit Application code VB objects SOAP over HTTP MetBroker MetSOAP RMI Delphi client application Delphi function calls SOAP over HTTP Applicationcode Borland SOAP Toolkit Delphi objects

42. SOAP Summary • Positives: • Opens brokers to a range of programming languages • Already the standard for Web-based RPC • Avoids most firewall issues • Negatives • Can’t reuse Java classes - must “mirror” in each language • Interoperability is still imperfect (but improving fast) • Inefficient for large data sets

43. Typical Array in SOAP • ...  • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">21.79</item> • <item xsi:type="xsd:float">21.79</item> • <item xsi:type="xsd:float">23.3</item> • <item xsi:type="xsd:float">21.11</item> • <item xsi:type="xsd:float">19.49</item> • <item xsi:type="xsd:float">18.48</item> • <item xsi:type="xsd:float">16.52</item> • <item xsi:type="xsd:float">14.73</item> • <item xsi:type="xsd:float">14.73</item> • ...

44. Model Deployment Options - Summary • Java applet • flexible, responsive once loaded but requires plug-in • Java servlet • simple browsers, rapid load, but clunky interface • SOAP • direct access for Delphi, VB… • Existing models (FORTAN,C,BASIC) • Use simple Java app. to retrieve data and write to a file

45. Contents • Concepts • Brokers • Remote Procedure Calls (RPC) • Using a broker • Metadata requests • Data requests • Issues • Linking your own data • SOAP • New brokers and future enhancements

46. New Brokers • DEM Broker - Digital Elevation Model data • Japan 50m resolution • World 1km resolution • ResourceServer • Internet-based service for software localization • ChizuBroker – Web map services • www.mapfan.com (Japan) • www.mapzone.co.nz (New Zealand) • SoilBroker – soils data

47. Future enhancements • Broker-side interpolation (using DEMBroker) • temporal • spatial

48. “Virtual station”Broker-side interpolation at single site Client Server Temperature Rainfall Radiation

49. Future enhancements • Broker-side interpolation (using DEMBroker) • temporal • spatial • Other brokers • Place names • Crops and crop models (prototyped) • Agrichemicals

50. Final Points • The Grid provides a stable, powerful foundation for Ag and NR software • You can now develop grid applications in your favourite programming language. • NARO is keen to help you to both develop new applications and link new databases. • Database owners around the world will be keen to evaluate your applications.