mpjdev, the low-level MPJ device

1. mpjdev,the low-level MPJ device Aamir Shafi aamir.shafi@port.ac.uk Distributed Systems Group 12th March, 2004

2. Trinity: Neo... No one has ever done anything like this.Neo: That's why it's going to work. Sequence • Introduction • History of MPJ • Overview of java.nio package • ‘mpjdev’ implementation • Introduction • Packing/Unpacking at the device level • Communication Protocols • Eager Send • Rendezvous Protocol • Quick Start Guide to mpjdev • What’s next

3. Introduction • Parallel Programming • Shared Memory • Message Passing • Message Passing • MPI • PVM • Has Java got what it takes to be a parallel programming language ?

4. Message Passing in Java (MPJ) • Specification Document • mpiJava 1.2.5 • Uses native MPI

5. Pure java MPI-like libraries • mpiJava 1.2.5 • Uses native MPI • Best candidate to be developed to MPJ • http://www.hpjava.org/mpiJava.html • DSG’s project MPJ (http://dsg.port.ac.uk/projects/mpj/) is tightly coupled to this effort • JMPI • Drawback, based on RMI • http://euler.ecs.umass.edu/jmpi/ • MPJava • Based on java.nio • Results encouraging • No source-code • DOGMA – Dead • MPP – • Basic library • http://www.mi.uib.no/~bjornoh/mtj/mpp/

6. Oracle: You're cuter than I thought. I can see why she likes you. Sequence • Introduction • History of MPJ • Overview of java.nio package • ‘mpjdev’ implementation • Introduction • Packing/Unpacking at the device level • Communication Protocols • Eager Send • Rendezvous Protocol • Quick Start Guide to mpjdev • What’s next

7. New I/O overview • Buffer classes, • Conversion among basic data types • Direct and Indirect Buffer • Scalable Applications • No more “one thread per client” • One thread, many clients • Multiplex Synchronous I/O using the selectors • SocketChannel – New Abstraction • Pipe – One way communication

8. Buffer classes in java.nio • ‘T’Buffer classes, ‘T’ being all the basic data types • Any basic data type can be copied onto the ByteBuffer, (the basic primitive) • Buffer can be, • Direct, allocateDirect(int) • Indirect, allocate(int)

9. a. ByteBuffer buffer = ByteBuffer.allocate(8);b. buffer.putInt(4);c. buffer.flip();

10. a. ByteBuffer buffer = ByteBuffer.allocate(8);b. ByteBuffer buffer = ByteBuffer.allocateDirect(8);

11. Selector • Selector provide, connecting, accepting, reading & writing facilities to the SocketChannel • OP_WRITE, OP_READ, OP_ACCEPT, OP_CONNECT • Depends on native OS facilities • Selection performs differently on Windows and Linux

12. Sequence of events for selectors

13. Taming the NIO circus • Taming the NIO circus thread • http://forum.java.sun.com/thread.jsp?forum=4&thread=459338&start=0&range=15&hilite=false&q= • OutOfMemory Exception • http://forum.java.sun.com/thread.jsp?thread=433702&forum=4&message=2136979 • Selectors taking cent percent CPU • http://forum.java.sun.com/thread.jsp?forum=11&thread=494967 • http://forum.java.sun.com/thread.jsp?forum=4&thread=494194 • J2SE 1.5.0 beta solves many problems

14. Neo: What are you trying to tell me? That I can dodge bullets?Morpheus: No, Neo. I'm trying to tell you that when you're ready, you won't have to. Sequence • Introduction • History of MPJ • Overview of java.nio package • ‘mpjdev’ implementation • Introduction • Packing/Unpacking at the device level • Communication Protocols • Eager Send • Rendezvous Protocol • Quick Start Guide to mpjdev • What’s next

15. mpjdev – Introduction • What is device ? • Sockets • Similar to ADI in MPICH • Meant for library developers (MPJ), not application programmers • jGMA can use mpjdev

16. mpjdev – Introduction • Single JVM Implementation • Processes are threads in the single JVM • Native implementation • Uses native MPI • LAPI Implementation • Eager-send • Rendezous • Buffer packing/unpacking are taken from these implementations

17. mpjev functions • Comm methods • Void send(Buffer, int dest, int tag) • Req isend(Buffer, int dest, int tag) • Req irecv(Buffer, int src, int tag) • For src, ANY_SRC • For tag, ANY_TAG • Status recv(Buffer, int dest, int tag) • Req methods • Status Wait() • Status Wait(Req[])

18. ‘mpjdev’ – Connectivity

19. Communication Protocols – Eager Send Sender Receiver Huge Memory at Receiver 1 2 3 4 5 6 Sel Sel

20. Rendezvous Protocol Receiver 1: Checks if step 2 has already posted recv, if yes, initiates step 3, else posts a req in Que Receiver 2: Checks if step 1 has posted a recv request, if yes, initiates step 3, else posts a request. NOTE :- STEP 1 & 2 needs synchronization Receiver 3: Writes back to sender that recv’r is ready to receive Receiver 4: Receives the data 1 Receiver Sender 1 Sender 1: Sender posts send(), Req stored in Queue Sender 2: Sender sends the control message asking if matching recv is posted ? Sender 3: Sender receives response confirming there is a matching recv Sender 4: Sender sends the actual data Sender Queue Recv Queue 1 1 Sel Sel

21. Packing/Unpacking of Buffers • write(t[] source, int offset,int length) • gather(t[] source, int numEls, int offs, int [] indexes) • strGather(t[] source, int srcOff, int rank, int exts, int strs, int [] indexes) • read(t[] dest, int dstOff, int numEls) • scatter(t[] dest, int numEls, int offs, int [] indexes) • strScatter(t[] dest, int dstOff, int rank, int exts, int strs, int [] indexes) • Java objects, same concepts

22. Message Format • Primary header (Eight bytes), • First four bytes –nothing • Last four byte --size of message • Primary PayLoad, • Each primitive data type is written as a section, • SECTION_HEADER (8 bytes) • First byte – Type of Message, int, float etc • Last four bytes – Number of Elements in this section • SECTION_DATA • The actual data itself • Should be multiple of 8 • Secondary header (Eight Bytes) • First Four bytes –nothing • Last Four bytes – Size of Java Object • Secondary PayLoad • Actual java Object in bytes • Can Java object be written as a section in Primary Payload ?

23. Message Format, Single Section int intArray[] = new int[2]; int[0] = 1; int[1] = 2; WriteBuffer wBuffer = new WriteBuffer(24); wBuffer.write(intArray, 0,2); wBuffer.pack(); 8 bytes 8 bytes 8 bytes X X X X Size (24) int T X X X 2(NoEls) int[0] int[1] PRIMARY_HEADER SECTION_HEADER SECTION_HEADER

24. Message Format, Multiple Section int intArray[] = new int[2]; long longArray = new long[2]; int[0] = 1; int[1] = 2; longArray[0] = 1L; longArray[1] = 2L; WriteBuffer wBuffer = new WriteBuffer(48); wBuffer.write(intArray, 0,2); wBuffer.write(longArray, 0,2); wBuffer.pack(); 8 bytes 8 bytes 8 bytes 8 bytes 16 bytes X X X X Size (24) int X X X 2(NoEls) int[0] int[1] L X X X 2 long[0] long[1] PRIMARY_HEADER SECTION_HEADER SECTION_DATA S_HEADER S_DATA

25. Control Messages • Used in Rendezvous Protocol • Format • Int – Rank of Destination • Int – Rank of Source • Int – Message Length • Int – Message Tag • Should be as small as possible

26. Morpheus: There is a difference between knowing the path and walking the path. Sequence • Introduction • History of MPJ • Overview of java.nio package • ‘mpjdev’ implementation • Introduction • Packing/Unpacking at the device level • Communication Protocols • Eager Send • Rendezvous Protocol • Quick Start Guide to mpjdev • What’s next

27. Directory Structure

28. Compiling the source files • Dependencies, • J2SE1.5.0-beta • Apache Ant (1.6.1) • If you want to compile from the source, otherwise • mpjdev.jar is lying in the mpjdev/lib folder • You need to add “mpjdev.jar” in your CLASSPATH

29. Running Examples - Config • Configuration • mpjdev/conf/mpjdev.conf • Total Number of Processes • For each Process • FULLY_QUALIFIED_NAME@PORT@RANK • Sorry – IP address won’t work at present • I never had to to write an IP 

30. Running examples • # cd mpjdev/test • # javac -classpath ../lib/mpjdev.jar:. BufferTest3.java • # java -classpath ../lib/mpjdev.jar:. BufferTest3 0 • (O is the Rank) • Tracking problems • java –version, is it 1.5. ? • Are you using IP ? • Email me

31. Writing your own programs • As an example, • Two processes • Initialize the device • CommImpl.init() • Get your ID, • CommImpl.id() • One process packs & sends data, • CommImpl.send(Buffer, int dest, int tag) • Other process reads data & unpacks • CommImpl.recv(Buffer, int src, int tag) • Finalize, • CommImpl.finish();

33. Neo: Why do my eyes hurt?Morpheus: You've never used them before. Sequence • Introduction • History of MPJ • Overview of java.nio package • ‘mpjdev’ implementation • Introduction • Packing/Unpacking at the device level • Communication Protocols • Eager Send • Rendezvous Protocol • Quick Start Guide to mpjdev • What’s next

34. A Few experiments • No tests, yet  • Bottlenecks, • Transfer of huge messages seems inefficient • Queues, java.util.Vector is not good enough • Would like to see how many processes, can selector handle ? • Potentially thousands • Need bench-marks, keeping in view java issues and bottlenecks, and test them • Write our own

35. Future • Badly need a runtime • Shell scripts starting jobs using ‘ssh’ or ‘rsh’ ? • MPJ, • Group communications • I thought, I would have completed the whole MPJ, by now • Looking for my PhD project, in the same domain Morpheus: Unfortunately, no one can be told what the Matrix is. You have to see it for yourself.

36. Suggestions ?