Real Time Operating Systems

1. Real Time Operating Systems It’s not Rocket Science! Ron Smith

2. Schedulability It’s Not Rocket Science! Ron Smith

3. Maj Smith’s experience - the Electronic Support Trainer (EST) Case • 70+ periodic (and asynchronous) tasks • Scheduled according to RM • no “special” deadlines assigned • asynchronous scheduled as “background” • Execution budgets allocated and maintained such that: • Usystem < 65% • After four long years of difficult multi-team development • never ending intermittent software (schedulability) errors • never-ending hardware / software finger pointing Ron Smith

4. My own experience - the Air Defence Anti-Tank System (ADATS) Case • No RTOS • 10 Parallel heterogeneous multi-processor systems • System performance and “hand” scheduling by cycle counting • Counting lines on a Pulse Code Modulation Output • Footprint for memory and schedule exceeded • Had to remove functionality such as playback to meet spec. • After several long years of difficult multi-team development • never ending intermittent software control errors • never-ending hardware / software finger pointing Ron Smith

5. NASA (the rocket science folks!)Mars Pathfinder Case • 3 periodic tasks • Single processor, fixed priority scheduling (DM) • WindRiver’s VxWorks real-time operating system (RTOS) • Task 1 – Bus Manager • high frequency, high priority • Task 2 – Earth Communicator • medium frequency / priority, high execution time • Task 3 – Geological Data Gatherer • low frequency, low priority • How much simpler could it be??? Ron Smith

6. NASA (the rocket science folks!)Mars Pathfinder Case • Introduce fault-tolerance • a watchdog timer was used to reset the system in the event that the computer / software locks up • essential design feature (no going to Mars to reboot) • “watched” for hang ups on the highest priority task • Introduce inter-task communication – • a shared resource (memory) was used to pass data from the data gatherer (task 3) to the communicator (task 2) via the bus manager (task 1). Ron Smith

7. NASA (the rocket science folks!)Mars Pathfinder Case “The Mars Pathfinder mission was widely proclaimed as "flawless" in the early days after its July 4th, 1997 landing on the Martian surface. … But a few days into the mission, not long after Pathfinder started gathering meteorological data, the spacecraft began experiencing total system resets, each resulting in losses of data. The press reported these failures in terms such as "software glitches" and "the computer was trying to do too many things at once". ”1 Ron Smith

8. NASA (the rocket science folks!)Mars Pathfinder Case • Wasn’t Pathfinder tested on earth !? • Extensive tests were conducted on earth • However, “… on Mars the task execution patterns were different”2 • the data communications task worst-case execution was much worse when communicating between Mars and Earth than Earth and Earth • Even still, “… the JPL engineers actually created a priority inversion situation during testing, but did not manage to analyze their recorded data well enough so to conclude that priority inversion is indeed a bug in their system. In other words, their test runs were sufficient, but their analysis tools were not.”1 Ron Smith

9. References 1. A clear and concise account “Priority Inversion and the Mars Pathfinder” by the folks at Temporal Solutions http://www.time-rover.com/Priority.html 2. Another more detailed scientific perspective “Missed it! - How Priority Inversion messes up real-time performance and how the Priority Ceiling Protocol puts it right.” by N.J. Keeling http://www.omimo.be/magazine/99q4/index994.htm Ron Smith