Designing High-Performance, Low-cost Embedded Systems

1. Designing High-Performance, Low-cost Embedded Systems Jason Clarke, Field Application Engineer

2. QNX Highlights • Over 25 years of building brand value through reliability and innovation • Millions of installations worldwide • Mission- and life-critical environments • Standards compliance, protected IP • Platform for leading-edge solutions • Most reliable RTOS • Fully integrated development tools • Expert services for embedded systems • Leader in innovative embedded technology • Multitasking RTOS running with MMU support • Founding member of Eclipse Consortium • POSIX certified RTOS • Long time member of the RapidIO Trade Association

3. Freescale and QNX

4. Extensive Freescale Support Host Processors MPC603, MPC603e, MPC604, MPC604e MPC740, MPC745, MPC750, MPC750ex, MPC755 MPC7400, MPC7410, MPC7450, MPC7451, MPC7455 MPC5200 PowerQUICC Family MPC821, MPC823e MPC850, MPC855T MPC860, MPC860T(P) MPC860DE (DT, DP, EN, SR) MPC8240, MPC8245, MPC8255, MPC8260 MPC 8270/8280, MPC8540/60 Sandpoint, 82xx ADS, MVP, Discovery, Atlantis, Multiple Vendor Boards (Artesyn, Force, MCG, Embedded Planet, and others) Boards

5. QNX Neutrino RTOS:The Most Reliable Realtime Operating System

6. Kernel space Application Application Device Drivers TCP/IP Stack File-system User Space Application Application Kernel space Device Drivers TCP/IP Stack File-system Microkernel Architecture Benefits  • Realtime Executive • No MMU and no protection • Applications, drivers, and protocols are all in Kernel space  System wide corruption  • Monolithic Kernel • (NT / Unix / etc) • MMU with partial protection • Applications are protected Contained (re-startable)  System wide corruption  • TRUE Microkernel • (QNX Neutrino) • MMU with full protection • Applications, drivers, and protocols are protected Contained (re-startable) File-system Application  Contained (re-startable) Micro Kernel Device Drivers TCP/IP Stack

7. Development and Testing Immediate Isolation of Bugs Faster Develop Test Cycle Allows for Parallel Debugging with Limited Targets Memory Protection Forces Use of APIs (Cleaner & Better Code) POSIX Programming Through Out System, Even in Drivers High Code Reuse Easier Driver Level Development Regression of Component as Completed Release and Management Immediate Fault Recognition and Recovery High Availability Falls Out of Architecture Makes System Upgrades and Patches Easy to Deliver Upgrades and Patches can be done Live, Without Downtime System Can Scale With Out Software Re-Architecture Microkernel Architecture Benefits TIME TO MARKET SERVICEABILITY

8. Hard Realtime Performance • Multiple concurrent scheduling algorithms • FIFO, Round Robin, Sporadic • Prioritized pre-emptable threads • 256 priority levels • Fully pre-emptable and deterministic kernel • Prioritized and nested interrupts • Interrupt handlers can schedule a user thread or run custom interrupt code Priority 50 Thread C Thread B Priority 18 Thread A Priority 12 Thread B Thread A IRQx Handler x IRQ y (higher) Thread B scheduled Handler y

9. /dev/tcpip /dev/ser1 Network Driver Process Manager Flash FileSystem Graphics Driver Audio Driver File System Message-Passing Bus Microkernel Application Application Inter-Process Communication • Processes communicate by sending messages Shared Memory - Large data sets and hardware access • Using messages cleanly decouples processes • POSIX calls built on messages fd = open(“/dev/tcpip”, ,,,) read, write, stat, devctl, … close • Other POSIX calls as well • realtime signals • pipes and POSIX mqueues • mutexs, condvars, semaphores • barriers, sleepon • reader/writer locks

10. Scalability

11. Route Manager File System Ethernet Driver Thread A Thread B Thread C Thread D Thread E QNX Neutrino Realtime Scheduler (OS) Thread E Thread A Thread B Thread C Thread D Priority Thread B FutureCPU CPU CPU FutureCPU Cache Cache Cache Cache High-Bandwidth CPU Bus Memory Symmetric Multiprocessing • Multiple processors sharing common hardware • Common memory bus and address space • Access to all peripheral devices and interrupts • OS manages tasks running on processors – true concurrency • Transparent to application programs • No application software changes needed • Automatic thread (~) scheduling across all CPUs

12. Internet Flash File System NetworkingStack MessageQueues Message-Passing Bus Microkernel Application Message Bridge (Ethernet, fabric, interconnect…) Database Flash File System Microkernel Application fd = open(“/etc/log_file.dat”,…); write(fd, …); Transparent Distributed Processing • Applications/servers can be network-distributed without special code • Message queues • File systems • Services • Databases • … fd = open(“/net/$HOSTNAME/etc/log_file.dat”,…); write(fd, …);

13. High Availability

14. MTBF MTBF + MTTR What is High Availability MTBF: mean time between failure MTTR: mean time to repair Availability =The probability that a system or subsystem will perform its intended function at a given instant of time. AVAILABILITY = • 99.999% Availability = Fewer than 5.25 minutes of Annual Downtime • As MTTR approaches 0, Availability reaches100%

15. Critical Process Monitor • Critical Process Monitor (HAM) monitors components and sends notification of component failure • Heartbeat services detect component ‘hang’ • Core file on crash can be created for debugging and analysis • Recovery from crash can be: • Controlled shutdown or system restart • Restart of only the failed subsystem (driver)

16. System Upgrades

17. BOOT IMAGE FLASH FILESYSTEM IPL BOOT IMAGE 1 BOOT IMAGE 2 FLASH FILESYSTEM IPL Flash Layout (8260ADS) • Boot Image • Contains Kernel • Requires only Flash Filesystem to be in Image • Flash Filesystem • Fault Tolerant POSIX Compliant Filesystem • Once Filesystem is Loaded Everything Else Can be Loaded from the Filesystem, Even Drivers • IPL (Initial Program Loader) • Sets Up Board and Loads Boot Image • Sits at Reset Vector

18. Application 1 Application 2 Application 1.1 Server System Upgrade • Add New Features or Processes on the Fly • Download New Binary into Filesystem or Ram • Load New Binary into RAM Network Driver Process Manager Flash FileSystem Graphics Driver Microkernel • Replace Existing Processes Without Reboot • Download New Binary to Filesystem • Remove Process Running in RAM • Load New Binary From the Filesystem

19. Server V1.1 /dev/service Zero Down Time Upgrade Server V1.0 Client /dev/service New Client • New version of the server attaches to the same name • New clients connect to new server • Old server exits when all old clients are gone

20. QNX Momentics:The Most Productive Tools

21. Momentics is Built on Eclipsewww.eclipse.org QNX sits on the Board and leads the C/C++ project.

22. Choose your host Windows, Linux, Solaris, QNX Neutrino IDE Tools C, C++ Code Developer Command Line GCC Tools (GCC v3.3.1) Source Debugger Choose your target ARM, MIPS, PPC, SH4, XScale, x86 Board Support Packages (Binary) Target System Tools Driver Development Kits Memory Analysis Choose your language C, C++, Java Application Profiler System Builder Application Builder Choose your BSPs BSPs for many popular boards and reference platforms System Profiler Code Coverage Choose command line or IDE IDE and command line tools interoperate Choose 3rd party tools Growing ecosystem of tools vendors supporting Eclipse QNX Momentics Tool Suite – At at a Glance Other Tools and Kits Development Host OS: Windows, Linux, Solaris, QNX Neutrino

23. Code Development Tools C, C++, Embedded C++ Power wizards Code editors Source navigator Local history Code Coverage Use coverage data to validate test plan Multi-run coverage amalgamation Runtime launch integration into IDE Source Debugger Concurrently debug multiple C and C++ applications Target System Tools System builder Target agent Detailed information on all processes and threads Application Builder Create QNX Photon microGUI applications Drag and drop widgets Source control tools Makefile structures Build tools Launch configuration wizard QNX Momentics Plug-ins • Target navigation view • Target system information tool

24. QNX Momentics Plug-ins - Continued • Performance Optimization Tools • Application Profiler • Statistical sampling • Call counting and call pair tracking • Visual call graph • Shared-library aware • Memory Analysis • Detect double free, non-initialized use • Overrun/underrun and memory leaks • Kill/freeze/debug/ignore on error • System Profiler • Software logic analyzer • Analyze events from instrumented kernel

25. Code Editors Identify keywords, syntax, and matching brackets at a glance Use the outline view to jump to any point in your source file Insert bookmarks and to-do’s Set breakpoints before you compile Simply double-click to build for any target Identify compile errors and warnings at a glance Hover over any function to view its arguments and required headers. Hover over any variable to view its type Track all errors and to-do’s from a central task list

26. Source Debugger Use the debug toolbar to start, stop, or restart processes, and to define breakpoints Click here to view breakpoints, variables, watch points, memory, registers, etc. Track each thread independently or trace control from thread to thread Jump directly to any point in your source Edit source directly from the debugger view Simply double-click to set breakpoints Hover over any variable to view its contents

27. Application Profiler View results by total time, percentage of total time, call count, etc. Utilize call pairing to highlight your program’s execution structure, then use the information to make your code more efficient Identify at a glance which functions need optimization Pinpoint which source lines consume the most CPU time

28. Memory Analysis Detect invalid pointer references and jump directly to the source Identify the location and nature of any memory error Inspect a call stack trace leading back to the origin of the error View memory leaks and allocations as they occur

29. Instrumented Kernel • The instrumented kernel logs events which are filtered and stored into buffers which are captured and analyzed. System calls Interrupts Process/thread creation On/Off filters Static event filters User defined filters Events Microkernel Event buffers State changes E1 E2 E3 E4 E5 E6 System Profiler Network Capture File

30. System Profiler Zoom in on a time range, select processes of interest, and create custom views Annotate log files with custom bookmarks Search for specific event types, and jump directly to any event View event details at a glance, including owner, type, and event data Examine summary statistics of thread states, including duration

31. QNX Momentics:System Profiler Example

32. Case: High Resource Contention • Problem definition: • Application is taking too long to send required number of messages • System has multiple threads, but all require access to a shared hardware resource – protected by a mutex • Approach: • Search on mutex blocking to see if there is high contention

33. Server Function static pthread_mutex_t global_mutex; void costly_function(void) { int i; /* faked up hardware access */ for ( i = 0; i < 100; i++ ) { nanospin_ns( 500 ); } /* The do something with the result */ delay(1); } void *server_thread(void *arg) { int rcvid, chid; chid = (unsigned)arg; while(1) { rcvid = MsgReceive( chid, NULL, 0, NULL ); pthread_mutex_lock( &global_mutex ); /* protect access to hardware */ costly_function(); pthread_mutex_unlock( &global_mutex ); MsgError( rcvid, EOK ); } return NULL; }

34. Initial Timeline View

35. Use Condition Statistics View

36. Configure Conditions

37. Add Search Conditions

38. Condition Statistics View

39. Minimize the Mutex static pthread_mutex_t global_mutex; void costly_function(void) { int i; /* faked up hardware access */ pthread_mutex_lock( &global_mutex ); /* protect access to hardware */ for ( i = 0; i < 100; i++ ) { nanospin_ns( 500 ); } pthread_mutex_unlock( &global_mutex ); /* The do something with the result */ delay(1); } void *server_thread(void *arg) { int rcvid, chid; chid = (unsigned)arg; while(1) { rcvid = MsgReceive( chid, NULL, 0, NULL ); costly_function(); MsgError( rcvid, EOK ); } return NULL; }

40. Ahhh….That’s Better!

41. QNX Momentics:System Profiler Evolution

42. System Profiler – CPU Activity

43. Porting to QNX

44. VxWorks Porting Guide • Available as free download on www.qnx.com • Porting VxWorks 5.x to QNX Neutrino v6.x • Most in-depth VxWorks porting guide available, with over 80 pages of technical content, including: • Architectural comparisons between VxWorks and QNX Neutrino • Memory accessibility differences • IPC differences with code examples for: • Message queues • Shared memory • Watchdog timers • Exception handling • Startup code • Hardware input/output, including device driver discussion • Networking • Build environments • Recommended porting methodologies

45. In Conclusion… • QNX technology is proven in mission-critical and life-critical environments • Scales from small single board computer to massive high performance systems using Neutrino’s unique SMP and Distributed capabilities • High-availability framework protects against software failures • Architecture allows for easy and flexible upgrade strategy • Integrated tool suite accelerates development

46. Contact QNX • George Logaras (Regional Sales Manager) • T : (613) 271-9330 • E : glogaras@qnx.com • Jason Clarke (Field Applications Engineer) • T : (613) 591-0931 • E : jclarke@qnx.com • Online • Info@qnx.com • www.qnx.com