Presentation Transcript

1. Integrating JavaFX with Native Technologies Stephen Northover (Oracle)Felipe Heidrich (Oracle)
2. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
3. Program Agenda FX Architecture: Overview Native Code in Glass and Prism Integrating OpenGL with JavaFX Performance: The DiamondBenchmark
4. Why might you need Natives? Access operating system specific operations Use Win32, Cocoa and GTK API Integrate pre-existing native code Improve performance Native code can be faster (tight loops etc.) Access faster platform specific operations OpenGL 3.x or greater, DirectX 11 or greater
5. JavaFX Architecture: Glass and Prism
6. JavaFX Architecture: Threading start(Stage) UI-Thread (Glass, FX) Sync User Code User Code User Code Sync Render-Thread (Prism) Render Code Render Code
7. JavaFX Architecture: Synchronizing Render-Thread (Prism) UI-Thread (Glass, FX) Sync
8. Glass Architecture Glass contains shared Java code (ie. View) Glass contains platform specific Java code (ie. WinView) Glass contains platform specific native peers (GlassView.cpp) Native peers written in C/C++/Objective-C Native peers contain native handles in their data structures No API to get the native handles from Glass (*) (*) An official API might be similar in concept to JAWT
9. What are the Native handles? Windows objects (C++, COM objects) HWND, HHOOK, IUnknown … Mac and iOS objects (Objective-C objects) NSApplication, NSWindow, NSView … GTK and Lens objects (C objects) GtkWidget *, GdkPixbuf *, ..
10. Where are the Native Handles? Glass objects and native handles are not 1-to-1 View lightweight on Windows but not on Mac No HWND on Windows, NSView on Mac … sometimes In a Browser, on Mac, no NSView, only a CALayer Need to write different code for different platforms Glass uses different implementation languages Glass uses different native objects to wrap handles
11. Problem: Get the Handle of a Stage Get the Glass object from the FX object Quantum exists between Glass and FX No API: Glass/Quantum objects are not public Get the Native object from the Glass object No API: Native code structure is subject to change No API: Must include Glass header files / fragments
12. Use Reflection: Java or Native Code staticlonggetHandle (Stage stage) { Object ws = stage.impl_getPeer(); try { Method m1 = ws.getClass().getDeclaredMethod("getPlatformWindow"); m1.setAccessible(true); Object w = m1.invoke(ws); Method m2 = w.getClass().getSuperclass().getDeclaredMethod("getNativeWindow"); return(long) m2.invoke(w); } catch (Throwableth) {} return0; }
13. Use Reflection: Java or Native Code NOT API ! staticlonggetHandle (Stage stage) { Object ws = stage.impl_getPeer(); try { Method m1 = ws.getClass().getDeclaredMethod("getPlatformWindow"); m1.setAccessible(true); Object w = m1.invoke(ws); Method m2 = w.getClass().getSuperclass().getDeclaredMethod("getNativeWindow"); return(long) m2.invoke(w); } catch (Throwableth) {} return0; }
14. Example: Open a Native Dialog on a Mac
15. Prism Architecture Prism contains shared Java code (ie. BaseShaderContext) Prism contains platform specific Java code (ie. ES2Conext) Prism contains shared native code (ie. GLContext.c) Prism contains platform specific native code (ie. MacGLContext.c) Native code is written in C and C++ (a little Objective-C) Native code contains native handles in their data structures No API to get the native handles (same as Glass)
16. Prism Architecture: Graphics Pipelines Prism Software D3D ES2 Java2D Mac Linux FB iOS …
17. Where are the OpenGL Prism handles? OpenGL is a state machine GL state is in a hidden platform specific “GL context” Example: glScissor(x, y, width, height) // no GL context param State can be queried when running in the Render-Thread Query state using OpenGL calls and/or platform specific calls State must be initialized when running in another thread Share state (textures) with Prism using platform specific calls
18. Platform Calls to Share GL State NSOpenGLContext>>initWithFormat:shareContext: BOOL WINAPI wglShareLists( HGLRC hglrc1, HGLRC hglrc2 ); GLXContextglXCreateContext(Display *dpy, XVisualInfo *vis, GLXContextshareList, Booldirect);
19. Native Calls to Share GL State EGLContexteglCreateContext(EGLDisplay display, EGLConfigconfig, EGLContextshare_context, EGLintconst*attrib_list)
20. To fill a shape with an image. Use existing picture box, DO NOT delete and create new picture box. Right click on the shape. At the bottom of the submenu select “Format Shape” Select “Fill” at the top of the “Format Shape” dialog box. Select “Picture or Texture fill” from the options. And select “File” under the “Insert from” option. Navigate to the file you want to use and select “Insert” On the “Format” tab, in the Size group, click on “Crop to Fill” in the Crop tool and drag the image bounding box to the desired size DELETE THIS INSTRUCTION NOTE WHEN NOT IN USE Integrating OpenGL with JavaFX
21. Java OpenGL Libraries JOGL (Java Binding for the OpenGL API) https://jogamp.org/jogl/www/ JLWGL (Light Weight Java Game Library) http://www.lwjgl.org/ Eclipse Platform Generator http://www.eclipse.org/swt/macgen.php Not an official Eclipse component but useful
22. Problem: Draw OpenGL content in JavaFX Solution 1: Draw OpenGL content to a byte buffer Upload the byte buffer to an Image (UI-Thread) Solution 2: Draw OpenGL content as part of Prism Provide an OpenGL object (texture) for Prism to draw Why not just draw directly? (more on this later)
23. Solution 1: Draw using a Byte Buffer Advantages Easy to understand and implement Does not access internals of JavaFX and Prism Disadvantages Too slow when fps requirements are strict Uses more memory than drawing as part of Prism
24. Use OpenGL calls to get the bytes glGetTexImage(GL_TEXTURE_2D, 0, GL_RGBA, GL_UNSIGNED_SHORT, buffer); glReadPixels(0, 0, WIDTH, HEIGHT, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
25. Use FX code to write the bytes PixelWriter pw = image.getPixelWriter(); PixelFormat<ByteBuffer> pf = PixelFormat.getByteBgraInstance(); pw.setPixels(0, 0, WIDTH, HEIGHT, pf, buffer, WIDTH*4);
26. Demo: Draw a Torus using a Byte Buffer
27. Solution 2: Draw as part of Prism Advantages Simple to understand/explain Fastest possible solution (least overhead) Disadvantages Must run / coordinate with Render Thread Uses Prism internals and relies on representation
28. Threading, Prism and OpenGL Application code runs in the UI-thread Prism code runs in the Render-thread OpenGL threading rules: OpenGL is not thread safe (it is thread aware) Multiple threads are possible (each in their own apartment) Platform specific code needed to create/share GL contexts We’ve seen these calls before
29. In what thread should your code draw? Draw in the Render-Thread Must be careful not to conflict/hammer Prism state Draw in another thread (even the UI-Thread) Must create a GL context for that thread (apartment threading) Must draw to a GL object (texture) for Render-Thread to draw Cannot issue GL commands from “wrong thread” Must synchronize around access to GL object (texture)
30. Drawing in the Render Thread Risky because Prism drawing model might change Must save/restore any GL state that you set Slow to save and restore state one at a time Doesn’t sound that hard but easy to get wrong How to integrate with Prism effects? Transforms? Transforms are done in software (might change) You won’t draw in the right place or the right way
31. OpenGL Integration: The Prototype Draw in the Render Thread Create your own GL context (platform specific) Save/Restore Prisms GL context Draw to a GL object (Texture) Allow Prism to create the texture (but you draw to it) Prism then draws the texture (translations, effects work correctly) Dispose the texture using Prism when no longer needed
32. Drawing in the Render Thread Take part in the UI-Thread/Render-Thread Sync Subclass an FX Class Prototype uses Pane Reimplementimpl_createPeer() to return your NGNode Subclass an NGNode (your NGNode) Prototype subclasses NGRegion ReimplementNGNode.renderContent(Graphics g)
33. Drawing in the Render Thread NOT API ! Take part in the UI-Thread/Render-Thread Sync Subclass an FX Class Prototype uses Pane Reimplementimpl_createPeer() to return your NGNode Subclass an NGNode (your NGNode) Prototype subclasses NGRegion ReimplementNGNode.renderContent(Graphics g)
34. Override renderContent() @OverrideprotectedvoidrenderContent(Graphics g) { // Save the Prism GL context, initialize the user GL context Texture texture = getPrismTexture(); NSOpenGLContextprismContext = NSOpenGLContext.currentContext(); NSOpenGLContextuserContext = getUserContext(prismContext, texture); // Set the user GL Context userContext.makeCurrentContext(); // Execute GL commands in the user GL context, and flush (or nothing draws) TorusLWJGL.resizeTorus(WIDTH, HEIGHT); TorusLWJGL.drawTorus(); glFlush(); // Restore the Prism GL Context, draw the texture prismContext.makeCurrentContext(); g.drawTexture(texture, 0, 0, WIDTH, HEIGHT, 0, 0, WIDTH, HEIGHT); }
35. Implement getPrismTexture() // Cache a single texture (released in setPrismTexture()) Texture prismTexture; Texture getPrismTexture() { if(prismTexture == null) { ResourceFactoryf = GraphicsPipeline.getDefaultResourceFactory(); prismTexture= f.createTexture(PixelFormat.INT_ARGB_PRE, Texture.Usage.DEFAULT, Texture.WrapMode.CLAMP_NOT_NEEDED, WIDTH, HEIGHT); prismTexture.makePermanent(); } returnprismTexture; }
36. Implement getUserContext() NSOpenGLContextgetUserContext(NSOpenGLContextprismContext, Texture prismTexture) { if(userContext == null) { userContext= (NSOpenGLContext) newNSOpenGLContext().alloc(); NSOpenGLPixelFormatpixelFormat = (NSOpenGLPixelFormat)newNSOpenGLPixelFormat().alloc(); pixelFormat.initWithAttributes(newint [] {0, }); userContext= userContext.initWithFormat(pixelFormat, prismContext); userContext.makeCurrentContext(); try {GLContext.useContext(userContext, false);} catch (LWJGLException e) {e.printStackTrace();} intuserFBO = glGenFramebuffersEXT(); glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, userFBO); intprismTextureID = com.sun.prism.es2.SetID.getID(prismTexture); glFramebufferTexture2DEXT(GL_FRAMEBUFFER_EXT, GL_COLOR_ATTACHMENT0_EXT, GL_TEXTURE_2D, prismTextureID, 0); } returnuserContext; }
37. Important: Follow Threading Rules Application code runs in the UI-Thread renderContent() is called in the Render-Thread Do not access UI-Thread state Copy state from UI-Thread FX object to Render-Thread NGNode It is easy to forget to do this but this leads to instability / crashes
38. Demo: Draw Torus inside Prism
39. The Way Forward Implement GLNode for JavaFX Provide minimal API (GL context, Prism texture) Enough API to implement simple GL integration Implement JOGL and LWJGL binding for JavaFX Provide appropriate drawing mechanism for the library Example: JOGL Autodrawable NOTE: Only ES2 platforms supported (non-Windows)
40. To fill a shape with an image. Use existing picture box, DO NOT delete and create new picture box. Right click on the shape. At the bottom of the submenu select “Format Shape” Select “Fill” at the top of the “Format Shape” dialog box. Select “Picture or Texture fill” from the options. And select “File” under the “Insert from” option. Navigate to the file you want to use and select “Insert” On the “Format” tab, in the Size group, click on “Crop to Fill” in the Crop tool and drag the image bounding box to the desired size DELETE THIS INSTRUCTION NOTE WHEN NOT IN USE The DiamondBenchmark
41. The Diamond Benchmark Strict requirements: Must handle thousands of small objects (“diamonds”) Objects are constantly moved and updated Multi-monitor, large area needs to be drawn Performance critical (must render at 60 fps) Examples: Sea urchins on the ocean floor, bees swarming around a hive, targets being tracked by missile defense system
42. First Version: Use Nodes Node for each diamond Use standard FX coding conventions Subclass of Region Use properties for variables Result: Too slow, animation is not smooth … why? Too many nodes, too much data, too many updates
43. Example: DiamondBenchmarkFX
44. Second Version: Use Canvas Lightweight object for each diamond Use standard FX coding conventions Subclass of Object (instead of Node) Use properties for variables Result: Too slow (slower than using Nodes) … why? Canvas uses immediate mode drawing (no nodes) Prism caches node drawing (can’t cache immediate mode)
45. Example: DiamondBenchmarkCanvas
46. Third Version: Use Custom OpenGL Lightweight object for each diamond Use optimal Java coding conventions Use optimal OpenGL conventions / data structures Use LWJGL for convenience (recode in C and compare) Result: Meets requirements, animation is smooth ... why? Uses an optimal OpenGL drawing strategy Uses platform specific OpenGL extensions
47. Example: DiamondBenchmarkLWJGL
48. How Prism draws the Diamonds A Diamond is a simple path (5 vertices) There is a path for each diamond node The diamond is transformed on the screen Prism renders each path separately Each path requires: Vertex coordinates, texture coordinates, color data etc. Rendering gets slower as the number of paths increase
49. A Faster way to draw the Diamonds Use OpenGL to do instanced drawing Put 5 path vertices on the graphics card and leave them Use glDrawArraysInstanced() from OpenGL 3.1 to draw Transfer the minimal amount of data (just x and y) Use a vertex shader to apply the transform
50. Prism versus Custom OpenGL “By knowing the content being rendered, an OpenGL developer can use techniques that Prism, as a general purpose graphics engine, does have the context to use.” “At the price of portability, an OpenGL program can take advantage of extensions and platform specific mechanisms that are not available to Prism.”
51. Summary Need an API like JAWT for Glass Reflection and private header files don’t cut it Need first class support for OpenGL in JavaFX Drawing to a texture looks promising Low level hacks / undocumented calls don’t cut it OpenGL can be used to improve performance Use of platform / GL version specific features Lots of drawing / updates (you won’t beat Prism otherwise)
52. Questions?
53. Main Title Goes Here Insert Presenter’s Name HereInsert Presenter’s Title Here
54. Program Agenda Topic 1, Arial, 24 pt Topic 2, Arial, 24 pt Topic 3, Arial, 24 pt Topic 4, Arial, 24 pt Topic 5. More than 5 topics, add second agenda slide.
55. Slide Title: Arial, 28 PTTwo-Line Max Subtitle: Arial, 20 pt, One-Line Max Bullets are sentence case. Use Arial, 20 pt font. Sub-bullets are Arial, 18 pt font. Keep bullets short. One idea per bullet. No more than five bullets. NOTE: Arial is the ONLY font that should be used in the Oracle corporate presentation template.Times and other serif fonts are not acceptable. To ensure that slides are properly formatted to this template, see pages 7 and 8 for instructions. Times Arial
56. Slide Title: Arial, 28 PT, One-Line Subtitle: Arial, 20 pt, One-Line Max Bullets are sentence case. Use Arial, 20 pt font. Sub-bullets are Arial, 18 pt font. Keep bullets short. One idea per bullet. No more than five bullets. NOTE: Arial is the ONLY font that should be used in the Oracle corporate presentation template.Times and other serif fonts are not acceptable. To ensure that slides are properly formatted to this template, see pages 7 and 8 for instructions.
57. Graphic Section Divider
58. Safe Harbor Statements One of the following slides must be used if your presentation covers material affected by Oracle’s Revenue Recognition Policy To learn more about this policy, e-mail: Revrec-americasiebc_us@oracle.com
59. The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract.It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
60. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
61. To fill a shape with an image. Use existing picture box, DO NOT delete and create new picture box. Right click on the shape. At the bottom of the submenu select “Format Shape” Select “Fill” at the top of the “Format Shape” dialog box. Select “Picture or Texture fill” from the options. And select “File” under the “Insert from” option. Navigate to the file you want to use and select “Insert” On the “Format” tab, in the Size group, click on “Crop to Fill” in the Crop tool and drag the image bounding box to the desired size DELETE THIS INSTRUCTION NOTE WHEN NOT IN USE Image Section Divider
62. ORACLE PRODUCT LOGO Announcement ALL CAPS, ARIAL 44 PT, 4-LINE MAX DELETE IF NOT IN USE
63. ORACLE PRODUCT LOGO Title, Arial, 28 PT, Two-Line Max BODY/STATEMENT COPY, ALL CAPS, 24 PT, HIGHLIGHT TEXT IN BLUE, USED FOR EMPHASIS DELETE IF NOT IN USE
64. Case Study Example with Photograph Solutions Loremipsum dolor sit amet, consecteturadipiscingelit. Fusce a sagittisorci. Maecenas vehiculalorempharetraipsumsuscipitutvenenatisrisusfacilisis. Donecvelaugue vitae urnaaliquetcursus. Aliquamrutrumiaculis.
65. Case Study Example with Screenshot Loremipsum dolor sit amet, consecteturadipiscingelit. Fusce a sagittisorci. Maecenas vehiculalorempharetraipsumsuscipitutvenenatisrisusfacilisis. Donecvelaugue vitae urnaaliquetcursus. Aliquamrutrumiaculis. Solutions
66. Case Study Example with Graphic Loremipsum dolor sit amet, consecteturadipiscingelit. Fusce a sagittisorci. Maecenas vehiculalorempharetraipsumsuscipitutvenenatisrisusfacilisis. Donecvelaugue vitae urnaaliquetcursus. Aliquamrutrumiaculis. Solutions
67. Case Study Example with Partner Logos Loremipsum dolor sit amet, consecteturadipiscingelit. Fusce a sagittisorci. Maecenas vehiculalorempharetraipsumsuscipitutvenenatisrisusfacilisis. Donecvelaugue vitae urnaaliquetcursus. Aliquamrutrumiaculis. Solutions
68. THIRD PARTY COMPANY LOGO Insert Author Name Here Position Title, Company Name “This slide format serves to call attention to a quote from a prominent customer, executive, or thought leader regarding a particular topic.” DELETE IF NOT IN USE
69. Classic Duke
70. Vertical Bar Chart Contextual information about the chart Bullet copy Bullet copy Bullet copy
71. Horizontal Bar Chart Contextual information about the chart Bullet copy Bullet copy Bullet copy
72. Stacked Chart Contextual information about the chart Bullet copy Bullet copy Bullet copy
73. Line Chart Contextual information about the chart Bullet copy Bullet copy Bullet copy
74. Pie Chart Contextual information about the chart Bullet copy Bullet copy Bullet copy
75. Tables
76. Tables
77. Tables
78. Tables
79. Tables
80. Tables
81. Tables