ID 710C: Developing Smart Screens for Industrial Applications

1. ID 710C: Developing Smart Screens for Industrial Applications QNX Software Systems Patrick Shelly Field Application Engineer 13 October 2010 Version: 1.1

2. QNX Software Systems Overview

3. Agenda • QNX Overview • HMI Technology Architecture • Adobe Flash as a next-gen embedded HMI tool • Challenges with HMI frameworks in embedded • Reliability • Performance • Integrating the HMI to real-time components • ActionScript extensions • Persistent Publish-Subscribe • Blending multiple apps into a seamless interface

4. QNX Core Product Offerings Development and productivity tools Aviage Middleware Portfolio Runtime Product Portfolio Services QNX Neutrino RTOS (Micro Kernel) QNX Photon microGUI Instrumented kernel File systems Multicore Adaptive partitioning High availability Connectivity and networking Integrated Java QNX Aviage multimedia solution QNX Aviage HMI suite (including Flash Lite 3) QNX Aviage acoustic processing Voyager web browser Support Custom engineering Professional Services Training QNX Momentics IDE Multicore visualization Performance optimization Memory and system analysis Performance and footprint optimization Photon application builder Target information & remote debug Libraries and GNU tools

5. QNX Runtime Product Portfolio Multimedia Suite Web Browser Acoustic Processing HMI Suite QNX Aviage Middleware Advanced Runtime Module Java Adaptive Partitioning Multi-core Instant Device Activation Extended Networking (security) QNX Neutrino Extensions QNX Neutrino RTOS Runtime Core OS Core Networking Core File Systems Peripheral Hardware QNX Neutrino RTOS High Availability 2D/3D Graphics Photon TDP/QNET

6. QNX Standards and Certifications

7. QNX HMI Architecture

8. 2D native HTML Video 3D native QNX HMI Technology Architecture OpenGL Application OpenVG Application WebKit engine Video Render Flash Services and widgets Scroll List Buttons Soft keyboard NavBar Transitions Themes ... Sliders Application launcher Loading Settings State Resize ... Native OS Services Media Browser SQL PPS Remote Graphics Layers ... Native and EGL interface Screen Management HW & Virtual Layer Management HW Accelerated Screen Transitions Vertical Stacks & Demos Medical Demo Industrial White Goods Demo Industrial SMA Demo Generic Demo QNX CAR QNX HMI Technology Core Documentation Core Applications Media player App store Docs Browser Widget Examples Application framework (ActionScript 3 based) Flash Adobe Flash OS interfaces HW Acceleration Composition Manager Input Touch, keyboard, mouse Graphics Engine OpenGL ES 1.1 & 2.0 OpenVG Operating System QNX RTOS 8

9. QNX HMI Technology Components: Flash Flash Lite 4 Hardware accelerated OpenVG (on supported platforms) ActionScript3 support Debugging capabilities X-Ray AS3 compatible extensions Webkit ASE – Browser mm-render ASE – video, audio playback PPS ASE –data distribution services QDB ASE – database access etc Native video decode/render for supported formats and platforms Complete HMI framework provided by QNX Apps, widgets, services, lifecycle management 9

10. QNX HMI Technology Components: Webkit and Multimedia • Webkit • HTML5 support • Includes new additions to HTML language for video, audio, geolocation etc • Uses Skia standard for 2D drawing • Current Webkit version 525; updating to version 533 • Improved components for user interaction • High performance panning and zooming • HTML rendering engine; browser app provided in Flash • Multimedia • Based on Aviage 1.2.1 • Providing audio/video playback without needing full framework • New hardware encode/decode support for new targets • Streaming or file-based video playback

11. QNX HMI Technology Components: HMI Composition Flash OpenVG Composition HTML OpenGL Video • Combine and overlay graphics from different render engines • Flash acts as a “Window Manager” • Based on standards • Action Script, OpenGL, OpenVG, OpenKode, HTML, JavaScript, POSIX

12. QNX HMI Technology Components: Tools and Documentation Tools QNX Momentics Tool Suite plug-in for Flash Builder 4 Sample Applications Reference Applications Youtube, audio and video player, browser, weather Widget use demonstration Documentation ActionScript extension API PPS documentation Installation guide Sample applications Developers guide API documentation 12

13. Adobe Flash in Embedded • 1 billion mobile devices in 2009 • 400+ device models

14. Why Adobe Flash? • Flash is the de facto HMI standard • Flash player installed on*: • 98% of US Web users • 99.3% of all Internet desktop users • Over one million graphics designers worldwide • Flash realizes the Java promise of Write Once, Run Anywhere • Class library dependency problems do not exist as in Java • Content developed for web or PC can run on embedded without change • Embedded FlashLite player uses less memory and provides faster rendering • Flash is ideal for HMI creation • Graphical environment by nature • Used by industrial designers to design look & feel • Ability to create “executable specifications” *Numbers are estimates from Adobe, and depend on demographical & statistical info

15. From Design to Deployment, no Detours Flash HMI Development • Designer creates HMI in Adobe Flash on PC • .swf file from design process runs directly on the target system • Done. Standard HMI Development • Designer creates HMI in Adobe Flash on PC • .swf file from design process run by software engineers on PC as mock-up of system • Engineers attempt to duplicate behavior by recreating screens and graphics • Design all state logic from scratch • Use C/C++ code or another toolkit • Graphical assets not directly reusable • Engineers debug and get C/C++ code running on target • Design difficulties encountered in HMI are sent back to designers for rework with real hardware constraints • Engineers make adjustments to software & graphics for new design • Pass running system by design for verification • Errors in HMI duplication reworked and recoded • Repeat steps 4 – 8 as necessary • Done.

16. Adobe CS3/CS4 Development Environment • Timeline controls “movies” with object motion, state control, and interaction • Many pre-scripted actions (effects, transitions, tweens, etc.) • Both scripted (code) and non-scripted (timeline) behavior • Graphical resources are primarily vector graphics • Can include videos (.flv) • Very well suited for Internet or connected development • remote connection, datasets • web services • wsdl, xml, urls, etc.

17. Challenges with HMI Frameworks Performance

18. Flash Lite in Embedded • Adobe Flash Lite 3 • Highly optimized implementation of the Flash runtime • Specifically for embedded market • Requirements in line with embedded • 32-bit data bus, 200MHz • 380KB core player DLL size (without video codecs) • RAM requirements: • For user interface: 4–6MB • For standalone content: 2–4MB (excludes video) • For web browsing: 32MB • Advantages • Web/desktop Flash content to mobile and back again, with minimal effort • Vector graphics adjusts to different screen sizes without loss of quality • Vector graphics creates very small content files, compared to bitmaps

19. How do you squeeze more speed out of Flash? • Real-time Optimizations • Use hardware acceleration • Optimize Flash movies • Move to native code

20. Non-Real Time Behavior • Jerky HMI (Flash unable to keep up with frame rate) • May need to raise priority of Flash Player thread or lower priority of other system activities • Can move portions of HMI into separate native applications and use hardware graphics layers to merge into one display surface • Can use Adaptive Partitioning to reserve processing time for HMI • What about inability to meet real-time deadlines for other components? • Won’t happen if thread priorities correctly set • Robust OS should include: • Automatic priority inversion protection • Priority-based graphics synchronization

21. Graphics Hardware Acceleration • QNX Aviage HMI Suite adds OpenVG hardware acceleration to Adobe Flash • OpenVG drawing accelerated by graphics hardware • Transparent application—no changes to flash code • Improvement depends on type of operation • Some operations 14 times faster! (Continuous rescale of transparent vector image) • Some operations slightly slower (alpha-blended bitmaps, which aren’t supported hardware) • Improvement is expressed in either faster frame rate, less CPU used, or both • Overall improvement is ~150%

22. Minimize Software Alpha Blends • Use a minimum of alpha blends (transparency) • Byte operations per pixel: 6 loads, 6 multiplies, 3 adds, 3 stores • Separate alpha channel (in addition): 1 load, 1 subtract • α + • (255-α) = • α + • (255-α) = • α + • (255-α) =

23. Minimize Gradients • Gradients require interpolation between multiple colors • Determination of colors on either side of the pixel • Calculation of interpolation percentage • Multiply percentage against each of two colors • Adobe provides complicated gradients • Radial, linear (multiple point), bitmap, rainbow, angled, etc. • Choose the simplest gradient that looks acceptable • Use bitmap caching if your platform supports it

24. Minimize Framerate • Use the lowest acceptable frame rate • 40-60 gives very fluid animation • 20-25 provides acceptable • 15 typical “lower-end” web recommendation • 10-12 may be best target for embedded • Frame rate determines speed of “movie” playback, and number of tweens (e.g. between key frame) generated per second • More frames = more CPU, and more RAM/Flash • Using multiple SWF movies (one that calls another), the framerate used is the caller’s • Can lead to unanticipated (i.e. higher than expected) frame rate • Easiest solution: use consistent frame rate across project

25. Integrating the HMI to Real-time ComponentsActionScript Extensions

26. Persistent Publish Subscribe (PPS) • Subscriber and Publisher • Can be unknown from each other at compile time • Can be dynamically substituted at runtime • Can have one-to-one, one-to-many, many-to-one relationships • Do not use direct calling API; available to all languages, scripts, shells • Can be in different languages, virtual machines, sandboxes • Are asynchronous from each other • Publisher • Can be configured how to persist data • Can support on-demand publishing (pull rather than push) • Persists attribute data over reboots • Subscriber • Are notified when data changes • Do not poll (block until delta update received) • Can receive notification on changes within a publishing tree Temperature Monitor Motor /fs/pps/heat, ethercan, pll Heat sensor EtherCAN pll

27. PPS Benefits

28. Blending Multiple Apps into a Seamless InterfaceGraphical Composition

29. Combining Flash and Native Applications • Flash as the HMI Manager • Flash normally lives in a browser environment • For embedded applications, use Flash as the main graphical environment • Control of display pane • Foremost HMI layer • Launch and control multiple native apps • Applications blended with OpenKODE composition manager index.html main.swf main.swf Browser Flash Plug-in Standalone Flash Player OS OS

30. Example HMI Decomposition

31. OpenKODE Window Transforms • OpenKODE supports multiple ways to merge window content • Positioning • Rotation • Scaling • Alpha blend • Chroma key

32. Why Develop an HMI Using Multiple Processes? • Different safety/criticality requirements • 99.999% uptime required for some pieces; lesser guarantee required on rest of system • Separate into multiple processes for validation • Require process isolation • Create sandbox around untrusted code • Take full advantage of existing software blocks • Browser, 3D visualization, etc.

33. Smart Meter Reference Implementation

34. Smart Energy Reference Design – Menu Tree Weather Info Home Screen Lighting Control Video Surveillance

35. Smart Energy Reference Design – More Screens Energy Usage Detailed Monthly Energy Usage Editing Energy Rules Energy Management

36. System Configuration Network Weather, Flickr, Pandora, etc. Zigbee Sensors Temperature & Humidity Powerline Communications Dimmer Controllers Insteon Server Smart Meter Zigbee Gateway Load Switches IP Camera Utility Simulator

37. Reference Design Software Architecture PPS Zigbee Sensors PPS PPS PPS PPS HVAC Controller Insteon Lighting Load Switches Smart Meter Flash Action Script Flash Lite 3 Player Flash Player Extensions PPS SQL SQLite Energy Usage Database Power Scheduler PPS SQL C/C++ Drivers & Applications PPS MPG4 Mediaplayer

38. Thank You