RMesa (Remote Mesa) Kutty S Banerjee Emmanuel Agu

1. RMesa (Remote Mesa) Kutty S Banerjee Emmanuel Agu

2. Remote Mesa • OpenGL Implementation on Mobile Clients • Split the OpenGL pipeline b/w Mobile Host and Server “adaptively” • Granularity is each stage of GL pipeline • Provide support for remote fetching of meshes

3. Mesa3D • RMesa built on top of Mesa3D • Software implementation of OpenGL. • Started by Brian Paul,lots of contributors • Technically, Mesa is OpenGL like API • Latest stable version is 6.0 Major Number of Mesa =(Minor Number of OpenGL)+1 • E.g., Mesa 6.0 = OpenGL 1.5

4. Pipeline Splitting

5. OpenGL Pipeline OpenGL Pipeline Overview Diagram OpenGL Reference Manual (Addison-Wesley Publishing Company)

6. Detailed OpenGL Pipeline

7. Split Pipeline? • Client weak • Borrows resources from powerful graphics server • Saves Battery Life • Renders more pixels on client • Adaptively distributes stages on client-server

8. Pipeline Splitting Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Client Blocks Client Pipeline Shipping vertices to server Fetching processed vertices from server Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Server Pipeline

9. Pipeline Splitting in RMesa • OpenGL calls classified: • State Changers (glNormal,glColor) • Vertex feeders (glVertex,glRect) • Buffers (glFinish, glFlush) • Special (Display Lists, Vertex Arrays, Textures) • Backend of these calls in Mesa is “Pipeline Factory” • RMesa modifies Pipeline Factory!!

10. Running Mesa Pipeline • Between glBegin and glEnd pair vertices • collected • When pipeline run, vertices flushed down • pipeline • NOTE!! New vertices created in pipeline • E.g., Clipping!! • In this case glColor merely changes the • GL State. glBegin(GL_TRIANGLES); glColor3f(1.0,0.0,0.0); glVertex3f(0.2,0.5,0.8); glColor3f(0.0,1.0,0.0); glVertex3f(0.9,0.3,0.5); glColor3f(0.0,0.0,1.0); glVertex(1.4,1.2,1,3); glEnd(); Within the draw() method Function draw()

11. Vertex Information • Vertex Data • Fog • Coordinates • Color • Color (Secondary) • Edge Flag • Normal • Texture Coordinates • Vertex & Vertex Data required by client • & server!! • Optimization? Don’t need • to ship Fog, Color, Secondary Color, • Texture Coordinates, Normal for • Projection Stage • But Projection Matrix required to be • shipped!! • All state information, matrices, stack • stored in GLContext!! Maintained by • all GL Implementations. FCENT

12. Networking Inside RMesa • Winsock 2.0 & BSD Sockets • Win32 and Linux (n/w abstracted) • RPC • Win32 RPC (OSF – Linux) • Marshalling ? Slow? • Creating interface for entire GL library - troublesome • Sockets vs RPC? • Do self marshalling • Tailor made for RMesa

13. Performance Metrics • Speed? RMesa vs Mesa • On Desktops • On Mobile Clients • Battery Life • Image resolution

14. Summary!! ` • RMesa: • Client performs “Primitive Assembly” stage. • Ships data over to server for projection stage and blocks • Server receives • Projection Matrix • Vertex Coordinates • Server performs the projection operation . • Ships modified coordinate values back to client!! Primitive Assembly User Defined Clipping Projection Stage This is where in the OpenGL pipeline we are talking of!!

15. RMesa Adaptive

16. RMesa -Adaptive • Maps stages on clients capabilities • A wrist watch maps all stages to server • Receives 2D image • RMesa on a laptop with GeForce FX Go5700 can itself be a server!!(Me not in sales at nVidia)

17. Constraints – Mobile Client • Will rendering drain my battery? • CPU fast? Can network to server be faster? • Memory enough? Need to store on server? • Graphics card – enough geometry power? • Is network fast? • Mobile Constraints: • Battery Life • CPU power • Memory • GPU limitation • Network congestion

18. Intelligence Agency • Features: • Static Reads: • CPU power • Memory • Graphics Hardware • Architecture • Dynamic Reads: • Battery Life • Network congestion • CPU utilization • Intelligraph reads client’s capabilities • Static Reads: One time read by the agency of • the client, done at startup!! • Dynamic Reads: Periodic surveillance of the • client’s configurations and changes • Maximum Polygons rendered without server • support • Empirical decision - Intelligraph

19. Intelligraph and RMesa

20. Intelligraph and RMesa Intelligraph Max vertices client can render alone Stage Mapper How to increase this number by stage mapping? Mesh Simplification RMesa Feeds Vertices to RMesa Pipeline RMesa – Interaction with Intelligraph

21. Fetching Remote Files

22. Rendering Remote Files • Remote File Render Process: • Client queries server for a list of files containing 3D data. • Selects a file and informs the server to fetch it. • It can also order server to render the polygons in the file instead of fetching the file. • Application stage provides vertices to geometry • stage. • Allows the application stage to be remote (Retained • mode graphics)!! • Client can fetch file from server and then with the • apply stage mapping to the polygons so obtained. • Extreme case – Client fetches 2D image from • server file!! • RMesa provides extensions to OpenGL calls to • support Remote Rendering of Files!!

23. Rendering Remote Files Application OpenGL Extension OpenGL Extension RMesa RMesa Client Server File Server

24. RMesa for Programmers

25. Configuration Files • clientRMesa Config file: • StageMapping • ModelView=client • Projection=server • Lighting=client • Texture=server • Fragment =client • Rasterization=client • . . • . . • Server • Server Address • Server Port • Intelligraph Address • Intelligraph Port • The clientRMesa.config file helps in locating • the RMesa server and Intelligraph. • Stage Mapping – details client server stages • Note: Temporary solution!! • The configuration file needs to be in the same • directory as that of the executable. • serverRMesa.config provides port number • of server

26. Compile Link with RMesa • Compile as usual • Link with OpenGL (RMesa Implementation) • clientRMesa.config in current directory • InitRMesa() – to use extended features • Without InitRMesa – regular single machine OpenGL appln

27. RMesa and java3D • java3D for OpenGL • Uses RMesa OpenGL transparently!! • Java applications abstracted completely!!

28. Project Status • Status Bar: • Milestones accomplished • Geometric pipeline split • Mapping of stages b/w • client-server • Socket & RPC implementation of n/w • Work Under Progress • Texture & Lighting • Rasterization Stage splitting

29. UbiCom Project • Augmented Reality:(AR) • Model synthetic objects in real world scene • Viewer sees through special glasses • Optical AR and Video AR • Requirement for low latency graphics scene modeling • Constraints on processing power, memory, graphics hardware on client mobile device (glasses) • Dynamic LOD • Adaptive Resource Contract • Optical AR- Virtual objects are merged into scene • via half transparent mirror • Video AR- Real world scene is tracked and modeled • in 3D and virtual objects immersed into it. More real • but slow!! • Head moves very fast therefore scene modeling • speed requirements are extremely high. • Done using client – server model • Reads in the client power and chooses a particular • Simplification model. Done dynamically!!

30. WireGL Project • WireGL product of Stanford Graphics • Client – server model • Client sends GL commands • Server does geometry & rasterization on a single node!! • O/p of Rasterization send to Image Composition and lastly display • Nodes form cluster • Soft First

31. WireGL & RMesa : State Tracking • Each server connects to many clients • How much data should client send server each time? • How does server keep pace with client’s GL Context? • State Tracking!! • Involves Tree Hierarchy and context switching. Slows server

32. WireGL & RMesa State Tracking • WireGL uses “Tree Hierarchy” on client • Server has “Context Switching” • Slows down and increases latency • RMesa state at the client. • “LookAhead State” shipped to server • No Context Switching on server

33. AnyGL Project • Derivative of WireGL • Separates Geometry and Rasterization • State Tracking present but involves more sophisticated “Timestamp Protocol” • Uses hardware for Image reassembly called Lightning-2 • Also provision for software reassembly

34. RMesa (Remote Mesa) Version 1.0 Kutty S Banerjee Emmanuel Agu