Embedded Port Operating System (EPOS)

1. Embedded Port Operating System (EPOS) Fred Kuhns (fredk@arl.wustl.edu, http://www.arl.wustl.edu/~fredk) Applied Research Laboratory Department of Computer Science and Engineering Washington University in St. Louis

2. A domainU domainV domainW X Decrypt all traffic from domain U Z Encrypt all traffic between domain U and W Y B X X The Big Picture CS523 – Operating Systems

3. Simple Message Format source address pr_id; params; Protocol: destination address Message Header protocol params node_id; app_id Source: node_id; app_id Destination: application data Message Body application data CS523 – Operating Systems

4. Advanced Services Router Line Card • Line card (LC) • Transmission interface (TI) • Routine packet processing (RPP) • Advanced services processing (ASP) • Interconnection network: interconnects ports and centralized control processor. • Control Processor (CP): manages system Control Processor TI RPP ASP Interconnection Network CS523 – Operating Systems

5. Port Processor Switch Interface Message Classifier Transmission Interface Protocol Processing, Forwarding /dev/null Application Specific Processing Embedded Port Processor CS523 – Operating Systems

6. ControlProcessor Switch Fabric Switch Core PP PP PP PP PP PP Port Processors LC LC LC LC LC LC Line Cards ASR Block Diagram CS523 – Operating Systems

7. RPP classifies packet and performs core IP processing If active, send to ASP. ASP may delete, modify pkt, modify attributes or inject new traffic. IP datagram arrives at an input port • Result of classification: • Action: fwd, drop, active, monitor • Attributes: forwarding, QoS, Active RPP forwards to output port IP datagram Hdr Hdr Hdr Hdr Hdr Line Card Control Processor Hdr TI RPP Hdr • Attributes • fwd: output port, next hop • QoS: scheduling params • active: • Attributes • fwd: output port, next hop • QoS: scheduling params • active: special processing ASP returns 0 or more packets to RPP. ASP Interconnection Network • Attributes • fwd: output port, next hop • QoS: scheduling params • active: special processing • Attributes • fwd: output port, next hop • QoS: scheduling params • active: output port queues packet for delivery to next hop. Example Processing CS523 – Operating Systems

8. PE PE PE ... ... MP MI CI Control Bus SRAM DRAM ASP Architecture • Processing Engine (PE): contains a processing core, small amount of local memory and register files for multiple thread contexts • Memory Interface (MI): provides access to external memory • Control Interface (CI): Provides access to the external control bus. • Management Processor (MP): Centralized manager for a given port. CS523 – Operating Systems

9. ... ... ... ... Manager Threads ... Simplified Model of System Processing Engine Processing Engine Local Memory Local Memory Thread Contexts Thread Contexts program text local data program text local data … … register file register file ALU ALU Bus Interface Bus Interface Memory Module Bridge Shared Memory cache MMU Application Specific Data Proc. Input Queues Output Queues IPC Local Memory Management Processor System Tables CS523 – Operating Systems

10. Issues to Consider • Application processing requirements • IPC requirements • scheduling and performance requirements • resource requirements: • CPU: job duration, periodicity, aperiodic, sporadic • Memory: must state be preserved across jobs, if so how much • Models: abstractions • Protection and Security • Computational model • Threads versus FSM; events versus algorithmic • multiprogramming – concurrent activities • Secondary storage requirements • Memory model: • sharing versus isolation • dynamic or static allocations; virtual memory; relocatable CS523 – Operating Systems

11. Overview • Event driven model: • “1st class” events: Packet arrival, timer expire, message arrival • Events are periodic, sporadic or aperiodic • New “Jobs” must undergo admissions test • Hierarchical scheduling: Logical CPUs allocated to application, then app (or sched library) allocates jobs to CPUs • Must bound overall delay and provide throughput guarantee for periodic and sporadic. Aperiodic is best-effort. • Preemptive CPU scheduling • Isolation between applications (app == processes) • shared global resources within task • threads == flow == related sequence of jobs • Support per thread memory allocation with protection • Virtual memory (segmented, paged). • CPU Local memory used as cache for resident sets. CS523 – Operating Systems

12. Overview • Protection and Security • packet/flow authentication/authorization • support IPv6 mechanisms in OS? • Performance/Efficiency requirements • Job duration generally short (usecs) would imply deferred preemptive model • CPU allocation to app may be on larger time scale, preemptive scheduling • Context switch between jobs must be cheap • Can HW help? CS523 – Operating Systems

13. Help from HW • Programmable clock interrupts, but good if OS is not dependent on short period interrupts • bind job with event queue and a SW controlled “Released” flag • lightweight threads: Explore CS523 – Operating Systems

14. Solutions? • Application creates threads • user threads may be bound to kernel threads and input queues • kernel threads may be used for running system activities • real-time scheduling, MP support • CPU – threads are schedulable entities • Memory • access protection but permit explicit sharing • assume dynamic installation and removal of programs • local memory mapped to tasks address space • global memory (slow) also mapped to address space (same vm address range for all tasks?) CS523 – Operating Systems

15. What you will do first • Design and implement a user space threading library • Scheduling framework supports multiple scheduling policies: • Priority-driven, preemptive versions of FIFO, Round Robin • extensible to support other work conserving and non-work conserving algorithms • Interface must include the follwoing: • wth_exit(...), wth_create(...), wth_this(), wth_yield() and wth_join(). CS523 – Operating Systems