Implementing a World of GUIs Using the ARM CortexM3.

1. Implementing a World of GUIs Using the ARM CortexM3. Dave Wilson Luminary Micro, Inc.

2. GUIs for Embedded Systems • We are not talking about running Linux or Windows on a low-cost consumer device! • We are talking about using CPU-rendered, on-screen graphical controls in conjunction with touch or button input to control your application.

3. GUI Benefits • Customizability • Product features can be defined by the software. • Offer new features without needing hardware changes. • Visually appealing and intuitive • Show only the information the user needs at that time. • Less display clutter.

4. The Good Old Days In yesteryears, fewer UI options were available to embedded device developers. • Graphics-capable displays were expensive. • Microcontrollers were too slow to run an application and handle graphics processing. • Graphics took too much storage for use on a small device. • Writing graphical interface code was time consuming and difficult.

5. LCD Displays Are Too Expensive • High demand for mobile phones and the availability of older-generation LCD manufacturing capacity has driven small panel LCD display prices sharply downwards. • 128x64, 4bpp OLED ~$4.00 • 128x128, color CSTN ~$3.60 • QVGA color touchscreen ~$14 Source: iSuppli/Stanford Resources, http://www.usdc.org/resources/DisplayTrends_summer2004/biggrowthahead.htm

6. Graphics Need Too Many MIPs Consider a worst case: • 320x240x16bpp using an 8bit parallel interface, 2 cycle GPIO operations, 15Hz refresh. • This requires about 14MIPS. • 28% of the CortexM3 CPU bandwidth at 50MHz. • 72% of the CPU is left for application use. • BUT - we are not in the video business! • Updates are seldom at 15Hz and seldom full screen. • Interface is often 16bits wide with single cycle GPIOs • 15Hz overhead drops to about 9.5% in this case.

7. To offer a different example… • Full software rendering. • Comparable resolution to today’s small displays. • “DOOM” ran on a 25MHz 80486. • Less than half the power of a CortexM3. “DOOM” from id Software State of the art 3D rendering on a PC in 1994.

8. Bitmapped images are huge! • A 320x240 16bpp image takes 150KB! • But… • The display panels we are looking at include integrated frame buffers. • Draw directly into the display panel buffer. • But what about application images? • Use the CPU power to your advantage: • Store images in a compressed format. • Saves about 80% of the space for typical buttons and text. • Use palletized image formats and convert on-the-fly. • Describe the image using 16 colors rather than 65536.

9. Graphics library software is big • True if you want Bezier curves, non-rectangular clipping regions, support for 15 image formats, pie charts, metafile generation, complex pens, customizable line endcaps, double byte fonts….. • You don’t need the Windows GDI or OpenGL to implement a very usable, compelling graphical user interface. • Luminary Stellaris Graphics Library compiles to about 10KB of code and needs about 400bytes of RAM workspace. • ASCII fonts are between 1.5KB and 6KB depending upon glyph size.

10. GUI Software Aims • Develop a minimum subset of graphics primitives that you will need for your interface. • Keeps code size to a minimum. • Keeps work to a minimum! • Isolate the control definitions and appearance from the code that manages them. • Allows rapid prototyping. • Get the application working with basic controls then spend time making them pretty. • Be consistent in how controls are defined and used. • Learn one set of concepts and use them in different areas. A widget-based architecture makes a lot of sense.

11. Widget (noun) • “A coil-like device placed in the bottom of cans and bottles of beer to aid in the generation of froth.” • “A component of a graphical user interface with which a user interacts.” • A widget is a control – something on the screen that you read, press, slide, click, toggle or otherwise fiddle with to set or get application information. • Widgets are instantiated as instances of a widget class (don’t worry – C++ is not needed!). • Widgets encapsulate the drawing of the control and its interaction with the user. Interaction with the application is via messages sent to an application callback function.

12. Widget Types • Container • A convenient way to group other widgets together and, optionally, place them on a background rectangle. • Canvas • A “static” control used to display text, image or color. Canvas widgets ignore user input. • Pushbutton • A circular or rectangular area that the user can press to initiate some action. May be outlined color or contain an image. • Radio Button • A clickable control which is grouped with its siblings to allow “one-of-many” selections to be made. • Checkbox • A Boolean indicator that can be clicked to toggle its state. • Slider • A draggable horizontal or vertical control that reports its position. • ListBox • A scrollable collection of text strings optionally allowing the user to select and highlight one.

13. Layered Architecture A layered UI management architecture allows maximum flexibility and reuse potential. Application Widget Manager Input Driver (e.g. buttons or touchscreen). Widget Classes Widget Classes Widget Classes Widget Classes Graphics Functions Display Driver Display Driver(s) Input H/W Display H/W

14. Display Driver A minimal interface to abstract the display hardware from the graphics rendering software. • Initialize the display hardware. • Report display dimensions. • Translate colors from 24bpp RGB to a display-specific value. • Plot a pixel. • Draw horizontal and vertical lines. • Copy a line of pixels from CPU memory to the display. • Fill a rectangle with a given color. • Flush any outstanding operations to the screen.

15. Display Drivers Without Hardware • Display drivers may also be written for offscreen rendering. • Allows double-buffering for smooth animation. • Allows composition of images before display. • Luminary Stellaris Graphics Library includes offscreen display drivers for 1bpp, 4bpp and 8bpp formats, for example. • These drivers offer the same interface as their hardware-supporting siblings (apart from being supplied a frame buffer image on their Init() call).

16. A Graphics Library for GUIs A very visually appealing GUI can be generated with a small set of graphics primitives: • Straight lines • Filled and outlined rectangles • Filled and outlined circles • 1, 4 & 8bpp images • Bitmap font rendering • Clipping rectangle The Luminary Stellaris Graphics Library offers this support and is free for use on Stellaris microcontrollers.

17. Graphics Context • All graphics functions take a context pointer as first parameter. • Graphics operations can take a large number of parameters: • Background and Foreground Colors • Fonts • Clipping rectangle • Display driver to use • These parameters are used frequently but seldom changed. • Rather than passing them as parameters on each call, they are stored in a “graphics context” structure. • Context parameters are changed via access functions: • GrContextBackgroundSet() • GrContextClipRegionSet() • Context parameters remain active until the application changes them.

18. “Hello World” // Initialize the display driver and turn on the backlight. Formike240x320x16_ILI9320Init(); Formike240x320x16_ILI9320BacklightOn(); // Initialize the graphics context. GrContextInit(&sContext, &g_sFormike240x320x16_ILI9320); // Fill the screen with dark blue then outline in white sRect.sXMin = 0; sRect.sYMin = 0; sRect.sXMax = GrContextDpyWidthGet(&sContext) - 1; sRect.sYMax = GrContextDpyHeightGet(&sContext) - 1; GrContextForegroundSet(&sContext, ClrDarkBlue); GrRectFill(&sContext, &sRect); GrContextForegroundSet(&sContext, ClrWhite); GrRectDraw(&sContext, &sRect); // Say hello using the Computer Modern 40 point font. GrContextFontSet(&sContext, &g_sFontCm40); GrStringDrawCentered(&sContext, "Hello World!", -1, GrContextDpyWidthGet(&sContext) / 2, GrContextDpyHeightGet(&sContext) / 2, 0);

19. I can draw stuff. Now what? • Graphics primitives offer the tools needed to draw the user interface. Now we need to generate usable controls. • The Widget Manager and Widget Classes turn graphics primitives and user input into working controls. • Each widget is defined in terms of a structure defining properties of the control and a message callback function. • The structure contains a standard widget header used by the Widget Manager and a class-specific section.

20. Defining Widgets Each widget is defined using a C structure: typedefstruct __Widget { long lSize; struct __Widget *pParent; struct __Widget *pNext; struct __Widget *pChild; const tDisplay *pDisplay; tRectangle sPosition; long (*pfnMsgProc)( struct __Widget *pWidget, unsignedlong ulMessage, unsignedlong ulParam1, unsignedlong ulParam2); } tWidget; typedef struct { tWidget sBase; // Header structure common to all widgets unsigned long ulStyle; // TrivialButton-specific style information void (*pfnOnClick)(tWidget *pWidget); // Click handler for this TrivialButton } tTrivialButton;

21. Widget Tree Structure WIDGET_ROOT pParent/pChild CHILD1 NULL pNext pParent/pChild pParent GRANDCHILD1 GRANDCHILD2 NULL pNext pNext pParent/pChild pParent/pChild pParent GGRANDCHILD1 GGRANDCHILD2 NULL GGRANDCHILD3 NULL pNext pNext pNext NULL NULL NULL

22. An Example How do we implement this simple interface as a widget tree? • 2 Radio buttons • “Choice 1” & “Choice 2” • 1 Push button • “Done” • 3 Containers • “What will it be?” border • Blue border behind “Done” • Overall black background

23. Determining Parent/Child Relationships • Using the right parent/child relationship is vital to ensure correct repainting and handling of user input. • The widget tree is painted starting at WIDGET_ROOT and working downwards – parents are painted before children. • Parents can be thought of as being below their children on the screen. • User input is handled in the opposite direction – clicks are first passed to children and, if not processed, passed up to the parents. • Widgets “on top” get a chance to process input before those underneath. • Think in terms of planes - the further back the widget, the closer it is to WIDGET_ROOT in the widget tree.

24. Drawing Planes Great- Grandchildren Grandchildren Child Draw Later Draw Earlier

25. Example Widget Tree WIDGET_ROOT NULL Background Container NULL Radio Container “Done” Container NULL “Choice 1” Radio Button “Choice 2” Radio Button NULL “Done” Pushbutton NULL NULL NULL NULL

26. Handling Repaint WidgetPaint(WIDGET_ROOT); 1 WIDGET_ROOT NULL 2 Background Container NULL 6 3 Radio Container “Done” Container NULL 4 5 7 “Choice 1” Radio Button “Choice 2” Radio Button NULL “Done” Pushbutton NULL NULL NULL NULL

27. Handling User Input • Pointer drivers generate up, down and (x,y) position messages in response to touchscreen or pointing device input. • A touchscreen or pointer driver is attached to the Widget Manager by setting WidgetPointerMessage as the driver callback function. • When a pointer down message (WIDGET_MSG_PTR_DOWN) is received, the Widget Manager passes it to all widgets, children first. • Each widget checks to see if the pointer is within its area and, if so, claims the message by returning “true” from the message procedure. • All further pointer move messages (WIDGET_MSG_PTR_MOVE) are sent to this widget until pointer up (WIDGET_MSG_PTR_UP) is received or the widget is removed. • Note that the order that pointer messages are passed to widgets is different from the order in which the widgets are painted. The widgets “on top” (children) need to see the input first before we offer it to those which lie behind (parents).

28. User Input Processing Order 7 WIDGET_ROOT NULL Search skips to the bottom of the tree 6 Background Container NULL 5 3 Radio Container “Done” Container NULL 1 2 4 “Choice 1” Radio Button “Choice 2” Radio Button NULL “Done” Pushbutton NULL NULL NULL NULL

29. What does the application have to do? • Define the widget tree. • Provide handler functions for each widget. • void OnDoneBtnPress(tWidget *pWidget); • Initialize the display driver. • Formike240x320x16_ILI9320Init(); • Initialize the touch screen/input driver. • TouchScreenInit(); • Pipe input events to the widget manager. • TouchScreenCallbackSet(WidgetPointerMessage); • Process messages. • while(1) { WidgetMessageQueueProcess(); }

30. Intelligent Display Module • 240x320 16bpp touchscreen display with LED backlight • 10/100 Ethernet • Relay • microSD slot • 4 ADC inputs • Speaker

31. IDM Example Security keypad application Buttons are arranged randomly when the keypad is activated. Entry of the correct code closes a relay connected to the door lock mechanism. A web server allows the code to be changed.

32. Oscilloscope • Uses a text- and menu-based widget system built on top of the Luminary Stellaris Graphics Library. • Runs on an EK-LM3S3748 evaluation board with 128x128 CSTN display. • User input via 4 directional rocker switch with “press to select.”

33. Conclusion • Using the CortexM3, today’s low cost color displays and software such as Luminary Micro’s Stellaris Graphics Library, visually appealing, high function graphical user interfaces are both affordable, usable and straightforward to develop.

34. Getting hold of the code www.luminarymicro.com/products/software_updates.html Software Documentation Look under C:\DriverLib\boards\rdk-idm for main examples.

35. Questions?