More on RSVP implementation

1. More on RSVP implementation • I/O subsystem • Logging

2. Why break down work • Say that for each incoming packet I have to do 3 pieces of work A, B, C • For example (A = parse packet and check for errors, B = create session state, C = create output packet and send it out) • Option 1: Do A+B+C in one big function/handler • If something slows down C (for example packet pacing forces me to send packets out slowly) then everything slows down • I will have to drop incoming packets • All processing is done at the pace of the slowest component • I may have a lot of CPU available but I still drop incoming packets!

3. Why continued • Option 2: Do A, B, C in separate handlers, with queues of packets between them • Then if C is slow, I can still continue accepting packets and doing A and B. • If C is tooo slow then the queue between B and C will fill up and will have to slow down B and eventually A • But for transient bursts of packets this will work better • I will be able to accept and process packets • Packet will be queued internally until C can clear the backlog

4. Example: packet recv/processing • Each incoming packet will trigger some action • I do not want to perform this action inside the packet read handler • May take variable time, trigger other actions etc and make my code complex • Isolate packet reception from packet processing • Read packet • Do some basic checks, parse • drop early if errors or I am overloaded • Put it in some queue • A separate handler will pick it up from the queue and process it

5. Example: processing/packet tx • I do not want to mix processing with sending packets • Processing handlers will generate packets for sending • queue the packets to be sent and a different handler will actually send them

6. Packet I/O • Use a raw kernel socket to send IPPROTO_RSVP packets • I do not want to BLOCK! • Because I loose control on the protocol operation • I make sure I read only when I have data to read • The select call should handle that • I set the socket non-blocking so I get a EWOULDBLOCK when sending • I want to build the IP header in RSVP • Because I may need to set the ROUTER ALERT option • Use sendmsg() call • I want large socket buffers • So the kernel will buffer large bursts of packets

7. Where is the select() • In Quagga this is handled by the thread library • When setting up a handler, I can specify which socket it is waiting on • thread_add_read(…, handler, …, socket) • The system will call the handler when there the socket is ready for read • If I want to read again, I have to call thread_add_read() after each time my handler gets called • Each Quagga process has a hidden loop where it keeps checking sockets through a select() call • The big select() loop • In lib/thread.c

8. Receiving Packets • In a Linux kernel I receive all packets on a single RSVP socket • Must be able to determine which interface the packet came from • The kernel will give me the ifindex of the interface • I can not tell which interface the packet arrived until I have received it • I can not enforce real interface fairness without the kernel’s help • If one interface receives so much traffic that will overrun socket buffers packets for other interfaces will get lost • After I receive the packet I have some control

9. Receive Packet scheduling • Do not get stuck in reading packets • Read only a limited amount of packets • If there are more, reschedule the read handler • Queue incoming packets in a per-interface queue • Process interfaces in a fair way • Process a limited amount of packets from each interface queue • Visit interfaces in a round-robin fashion • Have a list of interfaces with packets to read

10. Sending Packets • I can send a packet out of a given interface • Queue packets in a per-interface send queue • I do not want to get stuck in sending packets • Send only a limited amount of packets each time • If there are more packets reschedule the write event handler • Process interfaces in a fair way • Send a limited amount of packets from an interface • Visit interfaces in a round robin fashion • List of interfaces with packets to send

11. MSG_DONTROUTE • Sometimes I need to bypass routing • There may be ECMP paths and I need to send the packet out of one interface • Just need to send the packet to a directly connected neighbor • Setting the option results in a TTL of 1 in the packet • Packet will not be forwarded further • In RSVP some (but not all) packets will be sent with DONTROUTE on • Only when I have a strict ERO

12. Packet priorities • Protocol packets may have different priorities • HELLOs, LS-Updates in OSPF • Even RSVP has HELLOs • I have to prioritize • Put in a separate global or per-interface rx queue • give them preference when processing them • Put in a separate global or per-interface tx queue • give them preference when sending • But be careful • A malicious or buggy neighbor may send me tons of HELLOs • I do not want to starve lower priority packets • Should start dropping HELLOs too • Kernel should prioritize too

13. Logging • I want to be able to see what is going on • May help me debug a problem • If the problem is persistent • Not very useful for transient problems or crashes • Where do I log? • To screen • Can be very slow, somebody needs to watch • To file • Faster, but the log files may overflow • What to log? • Must be careful, logging is expensive • Too much logging will slow down system

14. Example logging: syslogd() • Used in Unix systems • A separate daemon that collects messages sent by the syslog() call in the std C library • Server and client speak the syslog protocol • Over TCP or UDP or named files • Server and client can be on different machines • Lot’s of security issues of course • 8 priority levels • From EMERG to DEBUG • Quite active, it even has its IETF working group!

15. Logging • Logging is tricky • Will change timing characteristics of the programs • Bugs may not appear anymore when logging is on • Logging also can change the flow of execution • May even cause a thread to block • The syslog() call writes to some file descriptor and can block • Levels • From very severe errors • That must be logged • To debug messages that are rarely used • May not want to allow customers to enable the very low internal logging • Will reveal internal implementation information • These are usually hidden commands

16. Logging • Need to ensure that the logging system is robust to overload and attacks • If there are too many incoming messages to be logged must drop some • Force the sender to drop them or • Make him block so that it slows down • But do I want to have a process block on a syslog() call? • Let the process do its own prioritization? It knows best what messages are important • Need to tell the user when events are dropped • Dropped events cause very confusing logs • Fairness between clients • Logging can be DoSed or fed seriously malformed input • Check the link in the class page for a discussion on the security of syslog()

17. Logging • Even if I have a properly designed logging system • It is still static • I can debug only persistent problems • An interesting idea: • Dynamic logging: when things start going bad turn on logging • How to know when things are bad? • Low memory • Impossible conditions happen • Too many errors happen • Which debugging to turn on • Has to be related to the problem

18. Logging in Quagga • zlog(), zlog_debug() etc • Frontend to syslog() or • Write to a file • Each process has its own knobs for what to log • Keeps a bitmap of events to log and checks each time there is a log call • Sometimes the check may be more complex • For example I want to log only HELLO packets that come from a certain IP address • Then for each incoming packet I will have to check its type and the source IP address to decide if I will log it

19. Other ways to get information • “show” commands • Can use them to show information about the system • Some versions of the commands can show some very useful internal information • Number of various errors • Availability of resources • I/O status • Again better hide from customers