CS 498 Lecture 10 Interaction Between Kernel and User Interface tc

1. CS 498 Lecture 10 Interaction Between Kernel and User Interface tc Jennifer Hou Department of Computer Science University of Illinois at Urbana-Champaign Reading:iproute2/tc

2. Compiling tc • Download from class website • Compile • Set the KERNEL_INCLUDE and LIBC_INCLUDE in Makefile • Follow instructions in README • No need for install • tc/tc

3. Running tc • Add a queuing disciple • ./tc qdisc add dev eth0 root red limit 600KB min 150KB max 450KB avpkt 1000 burst 200 probability 0.02 bandwidth 10Mbit • Check: • ./tc –s qdisc • /sbin/lsmod • Remove the queuing discipline • ./tc qdisc del dev eth0 root • /sbin/rmmod sch_red • Check again

4. Output of tc –s qdisc • [root@tiger tc]# ./tc -s qdisc qdisc red 800a: dev eth0 limit 600Kb min 150Kb max 450Kb Sent 868 bytes 6 pkts (dropped 0, overlimits 0 empty_arrive 0) marked 0 early 0 pdrop 0 other 0 • Line 1: parameters of the qdisc • Line 2: generic statistics for tc • Line 3: statistics for specific queuing discipline

5. In What Follow, We Discuss • How to add a dummy parameter • How to add a dummy queuing statistics variable • How to add a dummy statistics for tc

6. Data Structure for RED • In both • /usr/include/linux/pkt_sched.h and • /usr/src/linux-(version) /include/linux/pkt_sched.h • struct tc_red_qopt { __u32 limit; /* HARD maximal queue length (bytes) */ __u32 qth_min; /* Min average length threshold (bytes) */ __u32 qth_max; /* Max average length threshold (bytes) */ unsigned char Wlog; /* log(W) */ unsigned char Plog; /* log(P_max/(qth_max-qth_min)) */ unsigned char Scell_log; /* cell size for idle damping */ unsigned char flags; #define TC_RED_ECN 1 };

7. Data Structure for RED (con’t) struct tc_red_xstats { __u32 early; /* Early drops */ __u32 pdrop; /* Drops due to queue limits */ __u32 other; /* Drops due to drop() calls */ __u32 marked; /* Marked packets */ };

8. Data Structure for tc struct tc_stats { __u64 bytes; /* NUmber of enqueues bytes */ __u32 packets; /* Number of enqueued packets */ __u32 drops; /* Packets dropped because of lack of resources */ __u32 overlimits; /* Number of throttle events when this * flow goes out of allocated bandwidth */ __u32 empty_arrive; __u32 bps; /* Current flow byte rate */ __u32 pps; /* Current flow packet rate */ __u32 qlen; __u32 backlog; };

9. Locations of Data Structures • Kernel include file • /usr/src/linux-(version)/include/linux/pkt_sched.h • User include file • /usr/include/linux/pkt_sched.h

10. struct newred_sched_data { /* Parameters */ u32 limit; /* HARD maximal queue length */ u32 qth_min; /* Min average length threshold: A scaled */ u32 qth_max; /* Max average length threshold: A scaled */ u32 Rmask; u32 Scell_max; unsigned char flags; char Wlog; /* log(W) */ char Plog; /* random number bits */ char Scell_log; u8 Stab[256]; /* Variables */ unsigned long qave; /* Average queue length: A scaled */ int qcount; /* Packets since last random number generation */ u32 qR; /* Cached random number */ psched_time_t qidlestart; /* Start of idle period */ struct tc_newred_xstats st; u32 dummy; };

11. Adding a Dummy Parameter • Add member dummy in struct tc_newred_qopt in both files • Add member dummy in struct newred_sched_data in kernel net/sched/sch_newred.c file • Input the parameter • Output the parameter

12. Input/Output Parameter • [root@tiger tc]# ./tc qdisc add dev eth0 root newred limit 600KB min 150KB max 450KB avpkt 1000 burst 200 probability 0.02 bandwidth 10Mbit dummy 234 • [root@tiger tc]# ./tc -s qdisc qdisc newred 8009: dev eth0 limit 600Kb min 150Kb max 450Kb dummy 468 Sent 1316 bytes 8 pkts (dropped 0, overlimits 0 empty_arrive 8) marked 0 early 0 pdrop 0 other 0 dummy 12345678

13. Flow of Control oftc qdisc add dev eth0 root main() in tc.c filter qdisc: do_qdisc() in tc_qdisc.c class add: tc_qdisc_modify q->parse_qopt() rtnl_open() rtnl_talk()

14. Open a rtnetlink Connection • struct rtnl_handle rth; • if (rtnl_open(&rth, 0) < 0) { fprintf(stderr, "Cannot open rtnetlink\n"); exit(1); }

15. Send a Request to Kernel • if (rtnl_talk(&rth, &req.n, 0, 0, NULL, NULL, NULL) < 0) exit(2);

16. Request Message Format struct tc_red_qopt

17. Data Structure of nlmsghdr • struct nlmsghdr { __u32 nlmsg_len; /* Length of message including header */ __u16 nlmsg_type; /* Message content */ __u16 nlmsg_flags; /* Additional flags */ __u32 nlmsg_seq; /* Sequence number */ __u32 nlmsg_pid; /* Sending process PID */ };

18. Data Structure for a qdisc • In tc/q_newred.c struct qdisc_util newred_util = { NULL, "newred", red_parse_opt, red_print_opt, red_print_xstats, };

19. Finding the Right qdisc • struct qdisc_util *get_qdisc_kind(char *str) • Search in the qdisc_list first, return if found for (q = qdisc_list; q; q = q->next) if (strcmp(q->id, str) == 0) return q; • snprintf(buf, sizeof(buf), "q_%s.so", str); dlh = dlopen(buf, RTLD_LAZY) • snprintf(buf, sizeof(buf), "%s_util", str); q = dlsym(dlh, buf);

20. Input Parameter to Kernel: tc Part • int red_parse_opt(…) } else if (strcmp(*argv, "dummy") == 0) { NEXT_ARG(); if (get_size(&opt.dummy, *argv) ) { fprintf(stderr, "Illegal \"dummy\"\n"); return -1; } ……. addattr_l(n, 1024, TCA_RED_PARMS, &opt, sizeof(opt)); • rtnl_open(&rth, 0) • rtnl_talk(&rth, &req.n, 0, 0, NULL, NULL, NULL)

21. Input Parameter to Kernel: Kernel Part • static int red_change(struct Qdisc *sch, struct rtattr *opt) • q->dummy = ctl->dummy; • q->dummy <<= 1;

22. Output Parameters from Kernel • [root@tiger tc]# ./tc -s qdisc qdisc newred 8009: dev eth0 limit 600Kb min 150Kb max 450Kb dummy 468 Sent 1316 bytes 8 pkts (dropped 0, overlimits 0 empty_arrive 8) marked 0 early 0 pdrop 0 other 0 dummy 12345678

23. tc qdisc show (or tc –s qdisc) main() in tc.c filter qdisc: do_qdisc in tc_qdisc.c class In lib/libnetlink.c show: tc_qdisc_list rt_open rtnl_dump_request rtnl_dump_filter send_msg recv_msg print_qdisc q->print_qopt print_tcstats q->printxstats

24. Output parameters From Kernel: Kernel Part • red_dump() opt.dummy = q->dummy; RTA_PUT(skb, TCA_RED_PARMS, sizeof(opt), &opt);

25. Output Parameters From Kernel: tc Part • red_print_opt() fprintf(f, " dummy %u", qopt->dummy);

26. Adding a Dummy qdisc Stat • Define it in struct tc_newred_xstats (in both kernel and tc header file pkt_sched.h): __u32 dummy; • Initialize it in red_change() • Perform operations on this variable when enqueue/dequeue/… • Output it

27. Output a Dummy qdisc Statistics • In kernel: nothing to change • In tc: q->printxstats: red_print_xstats() fprintf(f, " marked %u early %u pdrop %u other %u dummy %u", st->marked, st->early, st->pdrop, st->other, st->dummy);

28. Adding a Dummy tc Statistics • [root@tiger tc]# ./tc -s qdisc qdisc newred 8009: dev eth0 limit 600Kb min 150Kb max 450Kb dummy 468 Sent 1316 bytes 8 pkts (dropped 0, overlimits 0 empty_arrive 8) marked 0 early 0 pdrop 0 other 0 dummy 12345678

29. Adding a Variable in struct tc_stats struct tc_stats { __u64 bytes; /* NUmber of enqueues bytes */ __u32 packets; /* Number of enqueued packets */ __u32 drops; /* Packets dropped because of lack of resources */ __u32 overlimits; /* Number of throttle events when this * flow goes out of allocated bandwidth */ __u32 empty_arrive; __u32 bps; /* Current flow byte rate */ __u32 pps; /* Current flow packet rate */ __u32 qlen; __u32 backlog; };

30. Updating the tc Statistics • In function red_enqueue() if (sch->stats.backlog == 0) { sch->stats.empty_arrive ++; }

31. Output the tc Statistics • Kernel: nothing to change • tc: print_tcstats() in tc_qdisc.c fprintf(fp, " Sent %llu bytes %u pkts (dropped %u, overlimits %u empty_arrive %u) ", (unsigned long long)st->bytes, st->packets, st->drops, st->overlimits, st->empty_arrive);