1 / 20

Device Layer and Device Drivers

Device Layer and Device Drivers. COMS W6998 Spring 2010 Erich Nahum. Device Layer vs. Device Driver. Linux tries to abstract away the device specifics using the struct net_device Provides a generic device layer in linux/net/core/dev.c and include/linux/netdevice.h

ita
Download Presentation

Device Layer and Device Drivers

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Device Layer and Device Drivers COMS W6998 Spring 2010 Erich Nahum

  2. Device Layer vs. Device Driver • Linux tries to abstract away the device specifics using the struct net_device • Provides a generic device layer in linux/net/core/dev.c and include/linux/netdevice.h • Device drivers are responsible for providing the appropriate virtual functions • E.g., dev->netdev_ops->ndo_start_xmit • Device layer calls driver layer and vice-versa • Execution spans interrupts, syscalls, and softirqs

  3. Device Interfaces Higher Protocol Instances Network devices (adapter-independent) dev.c dev_open dev_close napi_schedule dev_queue_xmit Network devices interface net_device_ops netdev_ops->ndo_open Abstraction from Adapter specifics netdev_ops->ndo_stop netdev_ops->ndo_start_xmit pcnet32.c pcnet32_open Network driver (adapter-specific) pcnet32_interrupt pcnet32_stop pcnet32_start_xmit

  4. Network Process Contexts • Hardware interrupt • Received packets (upcalls) • Process context • System calls (downcalls) • Softirq context • NET_RX_SOFTIRQ for received packets (upcalls) • NET_TX_SOFTIRQ for delayed sending packets (downcalls)

  5. Softnet • Introduced in kernel 2.4.x • Parallelize packet handling on SMP machines • Packet transmit/receive is handled via two softirqs: • NET_TX_SOFTIRQ feeds packets from network stack to driver. • NET_RX_SOFTIRQ feeds packets from driver to network stack. • The transmit/receive queues used to be stored in per-cpu softnet_data. • Now stored in specific places: • Receive side: in device packet rx queues • Send side: in device qdiscs

  6. Device Driver HW Interface • Driver talks to the device: • Writing commands to memory-mapped control status registers • Setting aside buffers for packet transmission/reception • Describing these buffers in descriptor rings • Device talks to driver: • Generating interrupts (both on send and receive) • Placing values in control status registers • DMA’ing packets to/from available buffers • Updating status in descriptor rings Driver Memory mapped register reads/ writes Interrupts

  7. Packet Descriptor Rings TX Descriptor Ring RX Descriptor Ring • Descriptors contain pointers, status bits • Driver allocates packet buffers TXQ Tail Packet Buffer SendErr Free Packet Buffer Sent Free Packet Buffer Packet Buffer RXQ Head Send RecvOK Send RecvOK Packet Buffer Packet Buffer Send RcvErr TXQ Head Packet Buffer Packet Buffer Free RecvCRC Free RecvOK Packet Buffer Packet Buffer RXQ Tail Free Free Packet Buffer Packet Buffer Packet Buffer Packet Buffer

  8. NIC IRQ • The NIC registers an interrupt handler with the IRQ with which the device works by calling request_irq(). • This interrupt handler is the one that will be called when a frame is received • The same interrupt handler may be called for other reasons (depends, NIC-dependent) • Transmission complete, transmission error • Newer drivers (e.g., e1000e) seem to use Message Sequenced Interrupts (MSI), which use different interrupt numbers • Device drivers can release an IRQ using free_irq .

  9. Packet Reception with NAPI • Originally, Linux took one interrupt per received packet • This could cause excessive overhead under heavy loads • NAPI: “New API” • With NAPI, interrupt notifies softnet layer (NET_RX_SOFTIRQ) that packets are available • Driver requirements: • Ability to turn receive interrupts off and back on again • A ring buffer • A poll function to pull packets out • Most drivers support this now.

  10. Reception: NAPI mode (1) • NAPI allows dynamic switching: • To polled mode when the interrupt rate is too high • To interrupt-driven when load is low • In the network interface private structure, add a struct napi_struct • At driver initialization, register the NAPI poll operation: netif_napi_add(dev, &bp->napi, my_poll, 64); • dev is the network interface • &bp->napi is the struct napi_struct • my_poll is the NAPI poll operation • 64 is the weight that represents the importance of the network interface. It is related to the threshold below which the driver will return back to interrupt mode.

  11. Reception: NAPI mode (2) • In the interrupt handler, when a packet has been received: if (napi_schedule_prep(&bp->napi)) { /* Disable reception interrupts */ __napi_schedule(& bp->napi); } • The kernel will call our poll() operation regularly • The poll() operation has the following prototype: • static int my_poll(struct napi_struct *napi, int budget) • It must receive at most budget packets and push them to the network stack using netif_receive_skb(). • If fewer than budget packets have been received, switch back to interrupt mode using napi_complete(& bp->napi) and reenable interrupts • Poll function must return the number of packets received

  12. HW interrupt invokes __do_IRQ __do_IRQ invokes each handler for that IRQ: action->handler(irq, action->dev_id); pcnet_32_interrupt Acknowledge intr ASAP Checks various registers Calls napi_schedule to wake up NET_RX_SOFTIRQ Receiving Data Packets (1) dev.c napi_schedule pcnet32.c ‘‘hard“ IRQ pcnet32_interrupt irq/handle.c __do_IRQ interrupt

  13. Immediately after the interrupt, do_softirq is run Recall softirqs are per-cpu For each napi struct in the list (one per dev) Invoke poll function Track amount of work done (packets) If work threshold exceeded, wake up softirqd and break out of loop Receiving Data Packets (2) .. ipx_rcv arp_rcv ip_rcv dev.c ptype_base[ntohs(type)] netif_receive_skb soft IRQ pcnet32.c pcnet32_poll dev.c net_rx_action softirq.c do_softirq Scheduler

  14. Driver poll function: may call dev_alloc_skb and copy pcnet32 does, e1000 doesn’t. Does call netif_receive_skb Clears tx ring and frees sent skbs netif_receive_skb: Calls eth_type_trans to get packet type skb_pull the ethernet header (14 bytes) Data now points to payload data (e.g., IP header) Demultiplexes to appropriate receive function based on header type Receiving Data Packets (3) .. ipx_rcv arp_rcv ip_rcv dev.c ptype_base[ntohs(type)] netif_receive_skb soft IRQ pcnet32.c pcnet32_poll dev.c net_rx_action softirq.c do_softirq Scheduler

  15. Packet Types Hash Table packet_type 0 ptype_base[16] type: ETH_P_ARP dev: NULL arp_rcv() func ... packet_type A protocol that receives only packets with the correct packet identifier list packet_type 1 type: ETH_P_IP A protocol that receives all packets arriving at the interface dev: NULL ip_rcv() func ... list . . . 16 packet_type packet_type packet_type type: ETH_P_ALL ptype_all type: ETH_P_ALL dev dev func func ... ... list list

  16. Transmission Overview • Transmission is surprisingly complex • Each net_device has 1 or more tx queues • Each queue has a policy associated with it • struct Qdisc • Polices can be simple • e.g., default pfifo, stochastic fairness queuing • Policies can be very complex • e.g., RED, Hierarchical Token Bucket • In this section, we assume PFIFO.

  17. Queuing Ops • enqueue() • Enqueues a packet • dequeue() • Returns a pointer to a packet (skb) eligible for sending; NULL means nothing is ready • pfifo – 3 band priority fifo • Enqueue function is pfifo_fast_enqueue • Dequeue function is pfifo_fast_dequeue

  18. dev_queue_xmit Linearizes skb if nec Checksums if nec Calls q->enqueue if avail If not, calls dev_hard_start_xmit dev->q->enqueue(pfifo) Checks queue length Drops if necessary Adds to tail otherwise Sending a Packet Direct (1) dev.c dev_queue_xmit sched_generic.c dev->qdisc->pfifo_fast_enqueue __qdisc_run Syscall or soft IRQ qdisc_restart dev->qdisc->pfifo_fast_dequeue dev.c dev_hard_start_xmit pcnet32.c pcnet32_start_xmit

  19. __qdisc_run Calls qdisc_restart until error Enables tx softirq if nec Qdisc_restart Dequeues a packet Finds tx queue Calls dev_hard_start_xmit dev_hard_start_xmit Invokes dev->xmit Frees the skb pcnet32_start_xmit Puts skb in tx descriptor ring Sending a Packet Direct (2) dev.c dev_queue_xmit sched_generic.c dev->qdisc->pfifo_fast_enqueue __qdisc_run Syscall or soft IRQ qdisc_restart dev->qdisc->pfifo_fast_dequeue dev.c dev_hard_start_xmit pcnet32.c pcnet32_start_xmit

  20. do_softirq invoked net_tx_action is the action for NET_TX_SOFTIRQ net_tx_action Frees packets posted to completion queue Invokes __qdisc_run on all output qdiscs if possible Sets bit in qdisc to run again if necessary Sending a Packet via SoftIRQ softirq.c do_softirq dev.c net_tx_action sched_generic.c __qdisc_run soft IRQ qdisc_restart dev->qdisc->pfifo_fast_dequeue dev.c dev_hard_start_xmit pcnet32.c pcnet32_start_xmit

More Related