Ninf-G - Core GridRPC Infrastructure Software Forum @ OGF19

1. Ninf-G - Core GridRPC InfrastructureSoftware Forum @ OGF19 Yoshio Tanaka (AIST) On behalf of the Ninf Project

2. Goal of this session • Attendees will understand • GridRPC programming model • how to develop and run Grid applications using Ninf-G • how it is easy to develop and run Ninf-G applications • how Ninf-G is suitable for developing flexible, robust, and efficient Grid applications • flexible: allow dynamic resource allocation/migration • robust: detect various faults and recover from them • efficient: utilize hundreds to thousands of CPUs efficiently. • Interact with users • Requirements? • Bug reports? • Questions?

3. Outline • Introduction of GridRPC • GridRPC: Grid-enabled Remote Procedure Call • Introducition of Ninf-G • What is Ninf-G? • Ninf Project • How to use Ninf-G • Programming and execution of Grid-enabled applications • Examples • TDDFT long-run on PRAGMA Grid (by Yusuke) • Large-scale hybrid QM/MD simulation on Japan-US Grid testbed • Future direction

4. Introduction of GridRPC

5. GridRPC: A programming model based on RPC • What is GridRPC? • Utilize remote procedure call (RPC) on the Grid • Based on client-server model • The GridRPC API is published as a proposed recommendation (GFD-R.P 52) at the GGF • Usage scenarios • Remote library call • Executing compute-intensive tasks on a remote high performance computing resource • Task parallel processing • Executing large numbers of independent tasks on distributed computing resources Remote Library Call Task parallel processing

6. ・・・・ ・・・・ Server ・・・・ ・・・・ Func. Handle ・・・・ ・・・・ Server Client Compuer ・・・・ ・・・・ ・・・・ ・・・・ Server ・・・・ ・・・・ 遠隔実行プログラム GridRPC Model • Client Component • Caller of GridRPC． • Manages remote executables via function handles • Remote Executables • Callee of GridRPC. • Dynamically generated on remote servers. • Information Manager • Manages and provides interface information for remote executables. Client Component Remote Executables Info. Manager

7. Typical scenario 1: desktop supercomputing • Utilize remote supercomputers from your desktop computer • Reduce cost for maintenance of libraries • ASP-like approach server client arguments results Numerical Libraries Applications

8. Servers Typical scenario 2: parameter surevey • Compute independent tasks on distributed resources • eg. combinatorial optimization problem solvers • Fault nodes can be discarded/retried • Dynamic allocation / release of resources is possible Client

9. Servers Typical scenario 3: GridRPC + MPI • Coarse-grained independent parallel (MPI) programs are executed on distributed clusters • Combine coarse-grained parallelism (by GridRPC) and fine-grained parallelism (by MPI) • Dynamic migration of MPI jobs is possible • Practical approach for large-scale computation Client

10. GridRPC task parallel client/server GridRPC API dispensable good available can be dynamic easy to gridify existing apps. GridRPC v.s. MPI MPI data parallel SPMD MPI indispensable poor (fatal) unavailable static＊ well known seamlessly move to Grid parallelism model API co-allocation fault tolerance private IP nodes resources others * May be dynamic using process spawning

11. Sample Program • Parameter Survey • No. of surveys: n • Survey function: survey(in1, in2, result) • Input Parameters: double in1, int in2 • Output Value: double result[] Main Program Survey Function Int main(int argc, char** argv) { int i, n, in2; double in1, result[100][100]; Pre_processing(); For(I = 0; I < n, i++){ survey(in1, in2, resul+100*n) } Post_processing(); survey(double in1, int in2, double* result) { ・・・ Do Survey ・・・ }

12. Original Program Client Program Callee Function IDL File Interface Info. LDIF File Stub Program Build remote library (server-side operation) Callee Function survey (double in1, int in2, int size, double* result) ・・・ Do Survey ・・・ } Specify size of argument Survey Function survey (double in1, int in2, double* result) { ・・・ Do Survey ・・・ } IDL File Module Survey_prog; Define survey (IN double in1, IN int in2, IN int size, OUT double* result); Required “survey.o” Calls “C” survey(in1, in2, size, result); IDL Compiler

13. Ninfy the original code (client-side) Int main(int argc, char** argv)｛ int i, n, in2; double in1, result[100][100]; grpc_function_handle_t handle [100]; Pre_processing(); grpc_initialize(); for(I = 0; I < n; i++) { handle[i] = grpc_function_handle_init(); } For(I = 0; I < n, i++){ grpc_call_async (handles, in1,in2,100, result+100*n) } grpc_wait_all(); for(I = 0; i<n; i++){ grpc_function_handle_destruct(); } grpc_finalize(); Post_processing(); Declare func. handles Int main(int argc, char** argv) { int i, n, in2; double in1, result[100][100]; Pre_processing(); For(I = 0; I < n, i++){ survey(in1, in2, resul+100*n) } Post_processing(); Init func. handles Async. RPC Retrieve results Destruct handles

14. Ninf-G Overview and Architecture

15. Ninf Project • Started in 1994 • Collaborators from various organizations • AIST • Satoshi Sekiguchi • Hidemoto Nakada • Yoshio Tanaka • Atsuko Takefusa • Yusuke Tanimura • Hiroshi Takemiya • University of Tsukuba • Mitsuhisa Sato • Taisuke Boku • Osamu Tatebe • Tokyo Institute of Technology • Satoshi Matsuoka • Kento Aida • Tokyo Denki University • Katsuki Fujisawa

16. Brief History of Ninf/Ninf-G 1994 1997 2000 2003 Ninf-G development Ninf project launched GridRPC standared specification Standard GridRPC API proposed Release Ninf version 1 Release Ninf-G Version 1.1 Start collaboration with NetSolve team Release Ninf-G version 0.9 Release Ninf-G version 1.0

17. Brief History of Ninf/Ninf-G (cont’d) 2003 2004 2005 2006 Draft GridRPC API Ninf-G 4.2.0 Submitted experimentaldocument 1st submission to the GGF Ninf-G 2.0.0 Ninf-G 4.1.0 Ninf-G 2.1.0 Ninf-G 4.0.0 Ninf-G 2.2.0 2nd draft submission Ninf-G in NMI GridRPC API publishedas proposed recommendation GFD-R.P 52 Ninf-G 2.3.0 Ninf-G 2.4.0

18. Ninf-G: A Reference Implementation of the GridRPC API • A software package which implements the GridRPC API. • Ninf-G includes • C/C++, Java APIs, libraries for software development • IDL compiler for stub generation • Shell scripts to • compile client program • build and publish remote libraries • sample programs and manual documents • Latest version is 4.2.1

19. Features of Ninf-G Version 4 • Enables RPC via • WS GRAM • Pre-WS GRAM • Condor • Unicore • SSH • NAREGI SS • Has an interface for working with the other Grid middleware for RPC. • E.g. ARC GridFTP Server (Nordu Grid)

20. Connect back Invoke Executable Architecture of Ninf-G4 Client side Server side IDL file Numerical Library Client Invoke Server IDL Compiler Generate Globus-IO Interface Reply Interface Request Ninf-G Executable GRAM / UNICORE / Condor MDS Interface Information LDIF File retrieve