1 / 34

The New MSR Plugin Download Environment

The New MSR Plugin Download Environment. First: The Old Way. Compile plugin code on NetBSD Copy plugin directory onto File System destined for SPC Boot SPC Log in to SPC requires user space and serial line or telnet access Use modload(1) on SPC to load plugin

mathewsj
Download Presentation

The New MSR Plugin Download Environment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The New MSRPlugin DownloadEnvironment

  2. First: The Old Way • Compile plugin code on NetBSD • Copy plugin directory onto File System destined for SPC • Boot SPC • Log in to SPC • requires user space and serial line or telnet access • Use modload(1) on SPC to load plugin • links plugin against kernel in /netbsd on SPC • Use modstat(1) on SPC to see list of loaded plugins

  3. The New Way • Compile plugin code on NetBSD • Verify plugin has no undefined symbols • There is no link against running kernel anymore • msr_lkm.h provides macros to needed kernel fucntions and structures • We can add more as we need them. • Use pluginDownload utility from CP (Linux) to download plugin to SPC • Not sure I’ve even tried it on NetBSD… • Use sendcmd utility from CP to configure plugin, filters, etc • Use pluginStat utility from CP to view list of loaded plugins

  4. pluginDownload • Uses gnu binary utilities cross-linker • Runs on Linux, builds a NetBSD binary • Used to: • prelink to get size information • final link to relocate • Uses new AAL5 control data channels to download plugin directly into kernel memory. • uses sendcmd API to: • allocate kernel memory for plugin • set up AAL5 data channel • call plugin entry function to complete load operation • Side Note: sendcmd API • should be expanded to include all of sendcmd, but right now we have about 15 “commands”.

  5. The rest of the slides are old notes

  6. Plugin Loading Steps • Compile Plugin code on NetBSD • Do one of these two, currently Linux seems to work better for later steps: • Pre-link Plugin on NetBSD to get size info ready • ld -e example -T 0 -o pl_combined combined.o • Pre-link Plugin on Linux to get size info ready • ld-cross –o pl_combined.o –e _example –Ttext 0 combined.o

  7. Plugin Loading Steps • On Linux CP: open file (pl_combined.o), get size information and allocate MSR/SPC memory for plugin • sendcmd –p # -c rp_pcu –s allocate –d <size> -d <sym_size> -d <sym_symsize> • sendcmd code in kernel calls msr_lkmiotcl(LMRESERV, <ptr> , FWRITE) • <ptr> points to a (struct lmc_reserv) which passes in: • size = a_text + a_data + a_bss (all from a.out header) • but we don’t need to load the bss since it is all 0 • sym_size = a_syms + stb.st_size – N_STROFF(info_buf) • a_syms is symbol table size • string table is at the end so, the string table size is full file size (stb.st_size minus string tab offset) • sym_symsize = a_syms • and returns: • address of block of memory allocated in kernel (resrvp->addr) • slot in lkm table (resrvp->slot) • address of symbols (resrvp->sym_addr) • msr_lkmioctl() returns 0 if successful, non-zero if there was an error. • sendcmd will return • Kernel address (PlAddr) for plugin to be loaded at • Slot number • Address of location of symbols (what do we use this for?) • This is used for adjusting the string table pointers before they are loaded (see modload.c for details) • useful command: • objdump

  8. Plugin Loading Steps (continued) • On Linux CP: re-link plugin to relocate to PlAddr • ld-cross –o plugin –e _example –Ttext <PlAddr+0x20> combined.o • relocates to plAddr+0x20 and sets ‘_example’ as the entry point • the +0x20 is because we are currently downloading the a.out header • This sets the a_entry field in the a.out header of the file • file plugin is now ready for downloading • On Linux CP: request two data channels to MSR Kernel • One for plugin text and data: • sendcmd –p # -c dchan -s get -d <PlAddr> -d <size> • Another for the plugin symbols (sym_addr was returned by allocate command earlier) • sendcmd –p # -c dchan -s get -d <sym_addr> -d <sym_size> • sendcmd returns the dchan# in its responses • On Linux CP: download file plugin into MSR/SPC memory: • pluginDownload -f plugin –S <sym_addr> -d <dchan#1> -d <dchan#2> -V <vci> -L <lkm slot> • This needs to adjust the string table before downloading. Thus it needs the sym_addr which was returned by the allocate sendcmd above • text and data go to dchan#1 and symbol and string table to go dchan#2

  9. Plugin Loading Steps (continued) • On Linux CP: Finish the loading of the plugin • sendcmd –p # -c rp_pcu –s load -d <entry_addr> -d <size> -d <sym_size> • I don’t think we actually need the PlAddr here. • I think in this command (or an earlier one) we do need to send the entry symbol address • We are also sending the amount of data sent for the plugin (size) and symbols (sym_size) • the msr_lkm code will use these two values to update • curp->offset • curp->sym_offset • This sendcmd causes a call to msr_lkmioctl(LMREADY, <data>, FWRITE) • We may also need to do some other stuff. See page of ioctl’s used by modload. • On Linux CP: release the data channel • sendcmd –p # -c dchan -s release -d <dchan#> • When all done with plugin: • On Linux CP: unload and deallocate the plugin • sendcmd –p # -c rp_pcu –s unload –d <id> • We should never have to use the LMLOADBUF and LMLOADSYMS cmds

  10. Data Transfer • Options for the download phase, CP side: • hack up modload from NetBSD and build it on Linux • modload uses ioctl’s to load the module into the local kernel • we want to send AAL5 frames • modify Stage3 of AAL5_download • Stage3 already processes NetBSD a.out files • Need to do one of these: • change Stage3’s interaction with Stage2, i.e. have it not expect any acks • add a Stage2 like thing to the MSR kernel for accepting plugin frames • Merge modload and Stage3 • modload does some manipulations to get the string table correct • Stage3 has all the code to send AAL5 frames and wait for ACK/NACK • Options for the downloading phase, MSR side: • Build a new AAL5 control data channel structure • Uses AAL5 frame buffer pool that the IP fwding channels use • Re-uses some of the frame processing code in kernel • Arriving frames are copied into previously allocated memory associated with the data channel. • It would be nice if APIC descriptors pointed directly at the allocated memory, but that breaks the current model and would make it hard to build in safeguards against corruption.

  11. Index(dchan#) 0 1 . . . DCHAN_MAX Data Transfer (continued) MSR Kernel Dchannel Table

  12. Data Transfer (continued) MSR Control Data Channel RATM AAL5 Frame Format (as it appears in the MSR Kernel) Shim Reserved space (8 Bytes) APIC Rx desc points here flags (8bits) dchan#(8bits) seq# (16bits) framelength (16bits) pad (16bits) flags: 1: Data 2: Ack 4: Nack AAL5 Frame Data (<= 1992 Bytes) 2016 B 2000 B 1992 B AAL5 Trailer (8 Bytes)

  13. External Symbols in Plugin • We will implement a structure to contain pointers to Kernel symbols that Plugins are allowed to use. • This structure will surely grow over time as we discover more kernel functions that we want to expose to Plugins • When a plugin is loaded, it will be given the pointer to this structure in the kernel via the call to its entry function • The plugin will therefore not need to link against the running kernel to resolve symbols. • This strategy will achieve two goals: • Put bounds on what plugins can do in the kernel • Make the linking and downloading of plugins easier

  14. External Symbols in Plugin (continued) • Current kernel functions used in example plugin: • malloc related: • free() • via FREE() macro from sys/malloc.h • malloc() • via MALLOC() macro from sys/malloc.h • MSR Related: • msr_printf() • PCU Related: • pcu_deregister_class() • pcu_free_all_instances() • pcu_register_class() • LKM Related: • lkm_nofunc() • lkm_dispatch() • via DISPATCH() macro from sys/lkm.h • lkm_exists()

  15. The rest of the slides are supporting notes some from other places

  16. Data Transfer • Options for the download phase, CP side: • hack up modload from NetBSD and build it on Linux • modload uses ioctl’s to load the module into the local kernel • we want to send AAL5 frames • modify Stage3 of AAL5_download • Stage3 already processes NetBSD a.out files • Need to do one of these: • change Stage3’s interaction with Stage2, i.e. have it not expect any acks • add a Stage2 like thing to the MSR kernel for accepting plugin frames • Options for the downloading phase, MSR side: • When we get a request to load a plugin of size <size> • sendcmd –p # -c rp_pcu –s allocate –d <size> • Allocate an APIC buffer of that size • associate the plugin load descriptor with that buffer • Tie the descriptor to the plugin load VCI • Go • Once we have the plugin in the allocated buffer: • in response to a • sendcmd –p # -c rp_pcu –s load –d <PlAddr> -S “example • use calls to lkmioctl() to do the rest

  17. Implementing modload in MSR • Issues: • Where will we download plugin from? • presumably just from the CP • Will the plugin on the CP already be pre-linked with the current MSR kernel? • if not, we may have Linux vs. NetBSD problems. • if so • how do we plan to keep plugins and MSR kernels in sync • how do we know the address where the plugin will go in the kernel • Dynamic scheme: • allocate memory on demand in the kernel for a new plugin • Static scheme: • predefined slots at predefined addresses in kernel • Each plugin is built to fit in a specific slot at a predefined address • Still need to resolve symbols with actual kernel • if we allow this at all…

  18. Implementing modload in MSR • Option 1: Use sendcmd utility to execute modload on SPC • Download plugin binary to a file on MSR • e.g. /var/run/rp/plugin/fileABC.o • and run modload • e.g. modload –o /var/run/rp/plugin/fileABC.o –e example fileABC.o • Something like: • sendcmd –p # –c rp_pcu –s load –S “fileABC.o” –S “example” • uses proposed new option –S for including strings • causes file “fileABC.o” to be downloaded to MSR and loaded as kernel module named “example” • Requires presence on MSR file system of: • /sbin/modload • /usr/bin/ld • up to date symbol version of kernel being executed

  19. Implementing modload in MSR • Option 2: Prelink on CP, use sendcmd to download directly into memory • Link plugin against copy of kernel on CP • ld -A netbsd.MSR -e _<entry> -o <outfile> -T <addr=0> <module> • Reserve Kernel memory: • sendcmd –p # –c rp_pcu –s allocate –d <size> • returns kernel load address • Relink at kernel load address (LA) • ld -A netbsd.MSR -e _<entry> -o <outfile> -T <addr=LA> <module> • Open relinked module and load it into kernel • sendcmd –p # –c rp_pcu –s load –S <module> • uses proposed new option –S for including a strings • Adjusting symbol table entry pointers and load it. • What does this actually do in modload(1) • sync(); • What does this actually do in modload(1) • Call the module’s entry function to test it. • Post-install if called for... • ioctl(lkm_fd, LMSTAT, ...)

  20. Implementing modload in MSR • Option 3: Prelink on CP, Plugin must use kernel function pointer array to access predefined, limited number of kernel functions, then use sendcmd to download directly into memory • plugin code uses local pointer for function pointer array which will be reset later when kernel calls entry function • Reserve Kernel memory: • sendcmd –p # –c rp_pcu –s allocate –d <size> • returns kernel load address • Relink at kernel load address (LA) • ld -A netbsd.MSR -e _<entry> -o <outfile> -T <addr=LA> <module> • Open relinked module and load it into kernel • sendcmd –p # –c rp_pcu –s load –S <module> • uses proposed new option –S for including strings • Call the module’s entry function to test it. • kernel will provide the actual location of the function pointer array • Post-install if called for... • ioctl(lkm_fd, LMSTAT, ...)

  21. Implementing modload in MSR (continued) • Current ld commands issued by modload on NetBSD: • modload links it the first time so it can get size information: • ld -A /netbsd -e _example -o /var/run/msr/example -T 0 combined.o • modload links it the second time to resolve symbols and relocate: • ld -A /netbsd -e _example -o /var/run/msr/example -T 100000 combined.o • Equivalent command on Linux: • First do this on NetBSD (‘pl_’ implies pre-linked) • ld -e _example -o pl_combined -T 0 combined.o • This is just for relocating, everything else should be done on NetBSD: • ld-cross –o plugin –e _example –Ttext 0x100020 pl_combined.o • Linux and NetBSD seem to be off by 0x20 on their location of the text segment, still investigating that. • After this command, the file plugin should be ready for downloading

  22. modload ioctl’s • We need to know exactly what each of these does: • ioctl(LMRESERV) • reserves kernel memory • reserves lkm slot for module • reserves kernel memory for the symbols • ioctl(LMLOADBUF) • loads module into memory, uses and affects lkm struct curp • ioctl(LMLOADSYMS) • loads symbol table into memory , uses and affects lkm struct curp • ioctl(LMLOADSYMS) • loads string table into memory , uses and affects lkm struct curp • ioctl(LMREADY) • if DDB is turned on, adds modules symbols to the DB symbol table • clears modules BSS space • calls entry procedure of module , uses and affects lkm struct curp • ioctl(LMSTAT) • post install retrieval of stats from module • ioctl(LMUNRESRV) • release slot for module if something goes wrong after LMRESERV.

  23. (From Crossbow Tutorial) NetBSD Kernel Topics IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous

  24. Loadable Kernel Modules • Mechanism in NetBSD to dynamically load code into running kernel • NetBSD System Utilities: • modload(8) • modunload(8) • modstat(8) • Module Types supported: • System Call modules • When unloaded, returns to original system call • Any system call can be replace • Take care when replacing ioctl(2) since LKM uses it to load/unload modules!! • Virtual File System modules • Device Driver modules: Block and character device drivers • Execution Interpreters: For binaries not normally usable by OS • Miscellaneous modules • No defined interfaces • Up to user/developer to provide hooks to get to the code • This is what Router Plugins uses

  25. Loadable Kernel Modules: modload • Open /dev/lkm • ioctl’s will be performed on the open file descriptor • Prelink module, open it and calculate size info • system(“ld -A /netbsd -e _<entry> -o <outfile> -T <addr=0> <module>”); • Reserve Kernel memory: • ioctl(lkm_fd, LMRESERV, size_info) • returns kernel load address • Relink at kernel load address • Open relinked module and load it into kernel • ioctl(lkm_fd, LMLOADBUF, ...) • Adjusting symbol table entry pointers and load it. • sync(); • Call the module’s entry function to test it. • Post-install if called for... • ioctl(lkm_fd, LMSTAT, ...)

  26. Overview (From MSR Tutorial) • Introduction to hardware environment • APIC core processing and buffer management • Overview of SPC kernel software architecture and processing steps • Plugin environment and filters • Command Facility

  27. Packet Classification & Plugins • Classification provides and opportunity to bind flows to registered plugin instances. • General classifier - Network Management • classification using 5-tuple • <saddr, sport, daddr, dport, proto> , • Prefix match on address, exact match port and proto • 0 is a wildcard for all fields • input and output ports • filters added/removed via the command facility

  28. General Match Classifier: Linear search of {src_addr, dst_addr, src_port, dst_port, proto}. General Classifier options: {First, Last, All} Rule Actions: {Deny, Permit, Active}. Rule flags {All, Copy, Stop} Invoke instance handler Search i1 i1 i1 i1 i1 i1 i1 i1 i1 i1 i1 i2 i2 i2 i2 i2 i2 i2 i2 i2 i2 i2 i3 i3 i3 i3 i3 i3 i3 i3 i3 i3 i3 i4 i4 i4 i4 i4 i4 i4 i4 i4 i4 i4 i5 i5 i5 i5 i5 i5 i5 i5 i5 i5 i5 Rule 1 Rule 2 Exact Match Classifier: Hash {src_addr, dst_addr, src_port, dst_port}, then linear search for flow spec. Exact Match Classifier options: None. Rule Actions: {Deny, Permit, Active, Reserve}. Rule flags {Pinned, Idle, Remove} Rule 3 hash Flow Table Rule 4 Call packet handler for bound instance with pointer to IP packet (struct ip *). Rule 5 flow flow Rule 6 handle_packet(inst, pkt, flags) { /* Plugin may read and/or * modify content but not * delete it unless COPY. * On return the framework * forwards packet */ ... return;} Rule 7 Shim instance->handle_packet(instance, packet, flags) Rule 8 AAL5 Frame Version H-len TOS Total length Identification flags Fragment offset Rule 9 pkt (struct ip *) TTL protocol Header checksum Rule 10 Source Address Destination Address Instance 1 {Active} Options ?? IP data (transport header and transport data) AAL5 padding (0 - 40 bytes) Send packet to exact match classifier Flow entry to plugin has a one-to-one relationship. CPCS-UU (0) CPCS-UU (0) Length (IP packet + LLC/SNAP) CRC Flow Bound to a Plugin Plugin Environment Plugin plugin plugin DQ/ In Queuing ... Exact Match: active processing same as general match. The AAL5 length is and IP header checksum are calculated so plugin does not have to perform these operations. NM Filter Flow Classifier/ (channel map) ... Ingress/ Egress ? Frame/Buffer and IP Processing Route Lookup (Shim, FIPL, Simple, cache) DRR/ Out Queuing ...

  29. Invoke instance handler Rule 1 i1 i2 i3 i4 i5 Rule 2 i1 i2 i3 i4 i5 Rule 3 i1 i2 i3 i4 i5 Rule 4 i1 i2 i3 i4 i5 Rule 5 i1 i2 i3 i4 i5 Rule 6 i1 i2 i3 i4 i5 Rule 7 i1 i2 i3 i4 i5 Rule 8 i1 i2 i3 i4 i5 Rule 9 i1 i1 i2 i2 i3 i3 i4 i4 i5 i5 Rule 10 i1 i2 i3 i4 i5 General Match Classifier Notes • General Match Classifier: Linear search of • {src_addr, dst_addr, src_port, dst_port, proto} • General Classifier options: {First, Last, All} • Rule Actions: {Deny, Permit, Active}. • Rule flags {All, Copy, Stop} Search

  30. hash Flow Table flow flow Instance 1 {Active} Flow entry to plugin has a one-to-one relationship Exact Match Classifier Notes • General Match Classifier: Linear search of • - {src_addr, dst_addr, src_port, dst_port, proto}. • Exact Match Classifier options: None. • Rule Actions: {Deny, Permit, Active, Reserve}. • Rule flags {Pinned, Idle, Remove}

  31. Active Processing Environment Class A “plugin x” Class B “plugin y” Class C “plugin z” Instance 1 {Active} Instance 1 {Deny} Instance 1 {Active} Instance 2 {Active, All} General/Exact Match Classifier Rule N RuleP • Plugin instance maps to at most one rule/filter. • General classifier: rule maps to at most 5 instances. • Exact match classifier: rule maps to at most 1 instance.

  32. create_instance() Called by PCU framework in response to receiving command. struct my_inst { inst_t base; subclass defs }; Creating an Instance Class A classid = 100 inst_t *create_instance(class_t *, inst_id) Return reference to instance create class instance Instance of Class A - (Base Class extended by Developer) <Fields defined by the Base Class> class_t *class inst_t *next inst_id id fid_t bound_fid void (*handle_packet) (inst_t *, ip_t *, flag32_t); void (*bind_instance) (inst_t *); void (*unbind_instance) (inst_t *); void (*free_instance) (inst_t *); int (*handle_msg) (inst_t *, buf_t *, flag8_t, seq_t, len_t *) <Class Specific Data> ...

  33. Plugin Class Specific Interface • All plugins belong to a class. At run time a class (i.e. plugin) must be instantiated before it vcan be referenced. • Plugin is passed its instance pointer (like c++) as the first argument. • Developer may extend the base class (struct rp_instance) to include additional fields which are local to each instance. • Plugin developer must implement the following methods: • void(*handle_packet)(struct rp_instance *, struct ip *, u_int32_t); • void(*bind_instance)(struct rp_instance *); • void(*unbind_instance)(struct rp_instance *); • void(*free_instance)(struct rp_instance *); • int (*handle_msg)(struct rp_instance *, void *, u_int8_t, u_int8_t, u_int8_t);

  34. Plugin Framework Enhancements • Integrated with Command framework • send command cells to PCU: • create instance, free instance, bind instance to filter, unbind instance • Send command cells to particular plugin instances • Send command cells to plugin base class • Enhanced interface to address limitation noticed in crossbow: • instance access to: plugin class, instance id, filter id • pcu reports describing any loaded classes, instances and filters

More Related