RTAI: a Linux extension to hard real time applications

1. Dipartimento di Ingegneria Aerospaziale, Politecnico di Milano, via La Masa 34, 20156, Milano, Italy RTAI: a Linux extensionto hard real time applications

2. Presentation overview • Operational view of the hard real time definition. • Appropriate perspective of why custom off the shelf (COTS) solutions are usable for high demanding local/distributed real time systems. • Advantages in adopting a free open source distributed real time operating system. • The RTAI solution: • overview of the core system (RTHAL/ADEOS) and services; • description of the implementation and features of RTAI middleware layer (net_rpc); • presentation of RTAI-Lab: a comprehensive middleware tool to develop and monitor local/distributed RTAI applications. • DIAPM and non-DIAPM RTAI applications. • Concluding remarks.

3. Soft and hard real time • Computer Science literature definitions: • a soft real time (SRT) system is characterized by its ability to execute a task according to a desired time schedule on the average (average determinism) video display • an hard real time (HRT) system embodies guaranteed timing, cannot miss deadlines and must have bounded latencies (strict determinism) plant controlled by a digital controller • Strict determinism is possible only with dedicated CPU, i.e. DSP and DSP like microcontrollers, having a guaranteed interrupt latency in the range of a single/few execution cycles.

4. GPCPUs have non deterministic latencies and jitter due to: virtual memory and MMU; multi level caches; bus arbitration from intelligent I/O subsystems; high depth piped execution and speculation. GPCPUs can meet high requirements at unprecedented cost/performance figures(mega/giga FLOPS capabilities). Even higher performances using low cost Multi Processors and networked GPCPUs. Hard real time using COTS hardware Is it viable the solution of adopting COTS hardware, like general purpose CPUs (GPCPUs), for HRT systems?

5. Worst case latencies in the range of few s if the controller is run standalone on a GPCPU. Typically the controller has to work cooperatively in a prioritised fully preemptable multitasking execution environment, e.g. for data visualisation, monitoring, logging and communication. Such a mix can lead to non deterministic latencies (jitter) up to 30 s (30% of the sampling period). A system having such a jitter can be still considered HRT? High precision profiling machine example MIMO controller at 10 KHz with some form of adaptivity, force control and active vibration suppression loop forever acquire data (sensors/commands) compute control action output data (actuators/logs) wait for the next sample (latency computation)

6. Hard real time specifications • The finishing and tolerances of our machined items verify our top expectations, around the clock, seven days a week. • A comparison against any equivalent dedicated DSP implementation will show no difference in results. • Our real time specifications must be related to the actual system bandwidth and only indirectly to the sampling rate. • Our 10 KHz controller will likely be controlling a 1 KHz bandwidth system for which a worst case 3% timing uncertainty will produce measurements and command errors that are guaranteed to be filtered, down to an acceptable noise level, by the inherent system low pass attenuation. • No clear cut between hard and soft real time but simply a continuous transition from low to high bandwidths.

7. Hard real time using FOSS • Real time operating systems (RTOS) running on GPCPUs will afford equivalent performances because these are dominated by the architectural features of GPCPUs. • At this point FOSS RTOSes can have an edge, with the added advantage of full code sources availability. • Is it possible to meet the required HRT specifications within an existing truly FOSS operating system such as Linux? • Linux is a non preemptable UNIX like OS, with superb performances as a general purpose computational/desktop/server OS, but with limited soft and hard real time capability. However, it offers a very good base to which HRT can be added. • This goal is achieved easily setting a strict separation of hard and non hard real time processes, while affording efficient communication and synchronization services to allow an effective interaction between the two environments. RTAI follows such a concept.

8. A Real Time Hardware Abstraction Layer (RTHAL) is installed that gathers all the pointers to the time critical kernel internal data and functions into a single structure so that they can be dynamically substituted by RTAI when HRT is required. Linux is changed to use what pointed in RTHAL for its operation. The Adaptive Domain Environment for operating Systems (ADEOS) nanokernel is installed that implements a pipeline scheme into which every domain (OS) has an entry with a predefined priority. RTAI is the highest priority domain which always processes interrupts before the Linux domain, thus serving any hard real time activity either before or fully preempting anything that is not hard real time. RTAI core system: RTHAL/ADEOS RTAI is integrated into Linux through a kernel patch and a series of add on programs (loadable modules) expanding the Linux kernel to hard real time. The patch can be RTHAL or ADEOS based.

9. RTAI schedulers • Fully preemptable. • Can be scheduled from within interrupt handlers so that Linux has no chance of delaying any RTAI hard real time activity. • Allow to symmetrically work inter-intra both user and kernel space by using the same set of APIs everywhere. • Optimised for kernel uniprocessor (UP), kernel symmetric multiprocessors (SMP), kernel multi uniprocessor (MUP), integrated UP/MP kernel/user space processing (LXRT). • Possibility of choosing between a base periodic timing, with a fixed assigned time resolution tick, and an arbitrary timing, allowing scheduling a task at the resolution of the available clock by firing a oneshot timer at the time instant imposed by the highest priority task waiting on the timed list. • Different scheduling policies: FIFO, Round Robin, Early Deadline First.

10. RTAI services FIFO communications Mailboxes Basic task management Direct intertask messages Binary/resource semaphore User space interrupts Bits synchronization Distributed services (net_rpc) User space interrupts Tasklets and timers

11. Distributed RTAI: net_rpc • Small (20 KBs) and effective real time middleware layer to allow native usage of any of the RTAI APIs on distributed local/remote control nodes. • The integration is provided by just adding a node/port identifier in front of any RTAI function call. Thus any application can be run on a single machine or on many networked machines without changing a single line of source code. • Lean implementation of the RPC idea. • The scheme uses a stub (buddy, proxy, agent) task to execute the remote APIs. • Stubs and ports are statically connected to guarantee maximum determinism. • All services are executed in a strict synchronous request_service/return_result mode, asynchronous usage is also possible. • Port requests/releases are not real time operation.

12. Distributed RTAI: net_rpc (cont.) • Ports are owned by a task and a task can own multiple ports having different identifiers. • Capability of dynamically expanding the services at run time. • Actual available support of low level net service through UDP, with dedicated connections and switches. • Sharing Linux networking possible but forgetting real time performances. • Ethernet UDP protocol is non strictly deterministic but using a dedicated high speed hardware guarantees acceptable performances for low to medium bandwidth, say up to 2 KHz, distributed controllers. • Higher bandwidth at an increased cost (GigaEthernet or SCI adapter) or using the free open source project RTNet. • Security?

13. RTAI-Lab • Provides a common structured framework to design, build, run and monitor locally/remotely any suite of RTAI-based single/multitasking real time applications, either specifically coded in a high level procedural language (C/C++), or automatically generated by proprietary packages (MATLAB/Simulink/Real-Time-Workshop), and/or fully open source projects such as SCILAB/Scicos/CodeGen. • Allows two separate systems, the host and the target(s), to communicate for changing/tuning parameters on the fly and/or monitoring/logging signals. • The host is the machine where RTAI-Lab is executing in soft real time; the target(s)is/are the machine(s) where the local/distributed generated code runs in hard real time.

14. Matlab case Simulink model (model.mdl) Real-Time Workshop build process 1) Target Language Compiler - rtai.tlc (model.c, model.h) 2) Template Makefile - rtai.tmf (model.mk) RTAI-Lab Graphical User Interface Real time executable (model) Making process make, gcc

15. DIAPM control applications (1) Tracking control of a flexible manipulator • Structural flexibility of the links. • Integrated active damping of structural vibrations. • Few sensors (angular potentiometers at the joints and strain gauges along the links) and actuators (direct drive brushless motors) to limit weight and costs.

16. DIAPM control applications (2) Fatigue test system • High flexibility in programming arbitrarily complex coordinated multi-load histories. • Load signals generation, test results logging, display and recording • Optimisation of test conditions and controller parameters

17. DIAPM control applications (3) MPWM control of the vibrations of a large space structure • Modular beam-like (19 m) plastic tubes-made (75 Kg) truss suspended from the ceiling by means of six soft springs. • Equipped with six pairs of on/off air jet thrusters operating within the horizontal plane, and six colocated capacitive accelerometers. • Multi PWM deadbeat predictive control law.

18. DIAPM control applications (4) Active control of combustion instabilities on a Rijke tube

19. DIAPM control applications (5) Reduction of vibrations transmitted to helicopter airframes using active gearbox struts • Agusta A109 mockup • Active gearbox struts provided with surface bonded piezoelectric patches • Feedforward multi-input multi-output FXLMS control algorithm.

20. Non-DIAPM control applications • Marble and granite automatic cutter machine • Ball bearing rings shape control • Laser cutting machine • Robot control • Universal measuring machines • Electric power conversion • Electroerosion machines • Plastic deformation machines

21. Concluding remarks • Since its very beginning RTAI has grown in an impressive way in robustness, services and programming tools. • RTAI is a FOSS development effort with a team of developers worldwide. • RTAI numbers many academic and industrial applications and its usage is more and more expanding. • RTAI might still lack some of the bells and whistles of commercial RTOSes, but it does not fear any performance challenge. • Bugs are always there: with FOSS, if nobody cares, you are free to loose some sleeping and fix bugs yourselves (feeling proud to contribute your fixes back to help the community).