Bandwidth management and optimization

Bandwidth management and optimization BCrouter 14-16 March 2006 Dirk JanssensICTS – K.U.Leuven

Introduction into introduction BCrouter is an ongoing network project • Not all features are already implemented or ready for 3th party deployment Constructive feedback • What do you expect from a good solution • Try to fulfill as many expectations as possible

Introduction Problem Expectations BCrouter solution BCrouter solution Components Example network setups Integration Security considerations BCrouter Introduction Components Commands and logging Routing and Netfilter setup Quota/Bandwidth exceptions BCpolicer Introduction Design principles Policing alternatives Complete design Case study: KotNet Development Current status Future Wish list Overview

Introduction: problem • Bandwidth usage rises rapidly • Increasing Internet population • ‘Richer’ content (HTML,Flash,…) • P2P download applications • Video/music streaming • Bandwidth availability is limited • Expensive uplink • No alternatives • Expensive hardware

Introduction: problem • Majority of bandwidth • used by minority of users • Minority of users • cause network congestion • cause problems for other users • Example: K.U.Leuven KotNet • Student network across region of Leuven • 20 000 active students • 5% of users caused 50% of used bandwidth

Introduction: problem • Users are anonymous • Only known by IP address • Very easy to change IP address to be anonymous • Everyone can (ab)use the network • What to do if external complaints come in? • User awareness is needed • Let the user take responsibility of his own network usage • Give the user a ‘personal credit’ he can use (network quota) • Notify/block the user if his/her PC acts ‘strange’ and give instructions • Answer: User authentication • Makes it possible to map every action on the network to an individual person • Prevents unauthorized access • Makes it possible to use ‘personal’ network settings and actions

Introduction: expectations • Login system • Each user must authenticate him/herself before using the network • No extra software or configuration needed on client hosts • Bandwidth regulation • Works for all protocols and traffic • Prevent that a minority of users take away all the bandwidth for the majority of users • Allow exceptions to certain (educational) sites • E.g. OS security updates, e-learning site… • Maximize responsiveness for interactive traffic • E.g. Slow down bulk traffic, but don’t touch SSH unless really needed • Every user and/or IP can have its own personal bandwidth settings • E.g. Different settings for a lab computer and personal PC • Distribute the individual bandwidth over the individual active network connections

Introduction: expectations • Volume quota • Every user and/or IP is only allowed to use a certain fixed amount of traffic • Learns the user how to manage his Internet behavior • Slow down traffic when a user and/or IP generates too much traffic • Every user and/or group and/or IP can have its own personal quota settings • E.g. personal vs. lab PC, limited guest accounts... • A user and/or IP is never blocked from the network (real-time small band) • If a user and/or IP who is on 'small band' stops downloading for a few minutes, the user immediately can use a limited amount of traffic again at normal speed.

Introduction: BCrouter solution • Why? • Didn’t find another solution that fulfills all the expectations • No open source projects • Commercial black boxes not really an option • It’s interesting, fun and challenging  • High performance needed • Old quota/login system was maxed out • Network usage still increases

Introduction: BCrouter solution • Features • User login system • ‘Unlimited’ number of users • Users can login multiple times at different location • Group based routing • ‘Unlimited’ number of user groups possible • Every group has its own independent routing and policy • Bandwidth regulation and volume quota • Individual user/group and IP address based settings with no performance impact • Prevent network congestion by dynamically regulating maximum bandwidth • Powerful quota and bandwidth exception possibilities • User friendly • No user side configuration needed • Nice user webpage with information and history information • Automatically redirect to login site for login

Introduction: BCrouter solution • Quota/bandwidth limiting to both user and IP • Example 1: • Assign user: • Quota of 1 Gigabyte • Refill the quota at rate of 1 Gigabyte/month • Maximum speed: unlimited • Assign IP: • Quota of 10 Mbyte • Refill the quota at rate of 5 Kilobytes/second • Maximum speed: 20 Kilobytes/sec • Result: • User settings to determine the maximum volume a user can download each month • IP settings to limit the ‘real-time’ bandwidth usage

Introduction: BCrouter solution • Quota/bandwidth limiting to both user and IP • Example 2: • Assign user: • Unlimited quota • Maximum speed: 50 Kilobytes/second • Assign IP: • Quota of 10 Mbyte • Refill the quota at rate of 5 Kilobytes/second • Maximum speed: 20 Kilobytes/sec • Result: • If a user logs in multiple times, the sum of all logins cannot exceed the maximum user speed. The speed is divided across the hosts that are logged in.

Introduction: BCrouter solution

Solution: components • Frontend • Login server • Redirect server • Backend • User database server • Log/History server • “BCrouter” router

Solution: components • Login server • Serves secure web pages to the users • Login page • Statistics page • Technical information page • … • Contacts the user database server for validating user accounts • Contacts the history server to gather historical information about logins and/or quota • Contacts BCrouter to check current quota and/or login status and performs login/logout

Solution: components • Redirect server • Redirects HTTP requests to the login page on the Login server • Gets all the traffic that requires a login from non-logged-in hosts • Redirect done by a webpage (not TCP level) • Separate dedicated host because can get DoS • Real time network anomaly detection • Detect virus/worm before login… even for 1st time users • Coupled to automatic user blocking system

Solution: components • User database • Contains all known users • Contacted by the login server • Can be any type of server • LDAP • Radius • Custom type of authentication • …

Solution: components • Log/history server • Receives logs from BCrouter • Parses received log files • Store processed information in a database • Historical login information • Historical account information • Database contacted by the login server • Possibility to use data mining techniques to detect suspicious user behavior

Solution: components • BCrouter • Implements the core functionality • Linux based solution • Sends detailed quota reports and issued commands to the log server • Contacted by the login server • Get quota information about user and/or IP • Get login status of user and/or IP • Perform login and logout operations

Solution: internet router setup • Assumptions • A few 1000’s of users • Limit by log/history server • Manage the internet connection • Auto redirect to login website • Minimize the used Internet bandwidth

Solution: internet router setup Internal backbone network BCrouter User database Log/History server Login server Web cache NAT Firewall Redirect server Internal management network Internet

Solution: main router setup • Assumptions • A few 1000’s of users • Limit by log/history server • Manage the entire network • Auto redirect to login website • Central DHCP server is used to distribute IP addresses • Minimize the used Internet bandwidth

Solution: main router setup Internal net Internal net Internal net Internal net BCrouter User database Log/History server Login server Redirect server Internal management network Web cache NAT Firewall DHCP Internet DNS

Solution: setup remarks • Webcache and NAT are between BCrouter and Internet • BCrouter needs to ‘see’ the user IP address • Otherwise not possible to make user and IP distinction • Advantage: • Transparent web caching is possible • Disadvantage: • Cached contents are also accounted and speed limited

Solution: integration • Suitable for each network? • Ethernet based networks • BCrouter does not support any routing protocols (RIP,EIGRP…) • BCrouter can also play a Cisco Netflow probe • High performance • Gigabit speeds with dual CPU system • Redundancy (still in development) • Possible to have backup BCrouter in hot standby

Solution: integration • Scalability • BCrouter server • Supports virtual unlimited users • Tested up to 50 000 users (1 Gigabyte RAM) • Handles up to 60 000 login/logout operations per second • Supports virtual unlimited IP addresses • Tested up to 200 000 IP’s (1 Gigabyte RAM) • Supports up to 300 000 packets/sec (1.5 Gigabit) • Dual Xeon 3.6Ghz • Clustering (Not yet implemented) • Possible to use multiple BCrouter servers • Each server handles a part of the given network segments • Inter-BCrouter communication to exchange quota changes

Solution: integration • Quota/bandwidth exceptions? • Yes… very powerful exception capabilities • Exception flags • IP speed limit • User speed limit • IP accounting • User accounting • No login required • Exceptions can be made for hosts or even entire networks (both local and/or internet)

Solution: integration • Quota/bandwidth exceptions examples: • Default: • Login required • Accounting to both user and local IP • Obey both user and local IP speed limits • Local host A does not have to login to access the Internet, but still uses IP quota and speed settings • E.g. Embedded device that can’t login and needs network access • Traffic from Internet host B is always possible from any local host and is never accounted, but local host IP speed limits are obeyed • E.g. Website with security patches • Any combination of exception flags is possible in either direction for any host/network

Solution: security considerations • Account abuse • Example • User A powers off his PC without logging out • Malicious user X takes IP of user A • X continues to work with credentials of user A • Solution: Auto logout • Possibility 1: BCrouter performs logout after X minutes of inactivity • Possibility 2: Ping probes • Possibility 3: DHCP server • Login server checks if IP that wants to login has been issued by the DHCP server. Refuse login with static IP • Use very short DHCP lease times (e.g. 15 minutes) • Run script every few minutes that logs out inactive DHCP leases • DHCP based auto-logout is preferred

BCrouter: introduction • Let’s take a look at the core element: BCrouter • Components of BCrouter • Commands and logging • Routing and Netfilter setup • Quota/Bandwidth exceptions

BCrouter: introduction Internal net Internal net Internal net Internal net BCrouter User database Log/History server Login server Redirect server Internal management network Web cache NAT Firewall DHCP Internet DNS

BCrouter: components • ‘open’ black box • Linux operating system • User space • DHCP forwarder • Syslog daemon • BCrouter daemon • Network configuration script • Kernel space • BCpolicer module

BCpolicer BCrouter: components User space Management interface DHCP forwarder BCrouter daemon Syslog daemon Kernel space Kernel logging Netfilter framework Input interfaces Output interfaces

BCrouter: components • DHCP forwarder • Forward broadcast DHCP DISCOVER to a central DHCP server • Dhcp-fwd • http://www.nongnu.org/dhcp-fwd/ • Very simple application • User space application running in chroot jail • Listens in ‘promiscuous mode’ on specified interfaces

BCrouter: components • Syslog daemon • Send logs to a remote log server for remote processing • Syslog-ng • http://freshmeat.net/redir/syslog-ng/10178/url_homepage/syslog-ng • Very powerful options (filtering, multi logserver…) • Logs both user space as kernel logs

BCrouter: components • BCrouter daemon • Provides a network-based console to the BCpolicer kernel module • Simple Perl script (Forking TCP server) • Allows simultaneous management access • Listens on a network socket (telnet port 23) • Communicates with the kernel module

BCrouter: components • Network configuration script • Provides entire interface, routing and Netfilter configuration and setup • Shell script • Executed at boot time

BCrouter: components • BCpolicer kernel module • Receives login/logout commands and performs accounting and routing decisions • Core element of BCrouter (ipt_bcpolicer) • Works entirely in kernel space • Loadable module which implements an iptables target

BCrouter: commands & logging • Commands • Login/logout • login [username] ip [x.x.x.x] reason [text] • logout [username] reason [text] • logout [ip] reason [text] • Query information • show user ip [x.x.x.x] • show ip user [username] • show quota ip [x.x.x.x] • show quota user [username] • Configuration • conf ip … • conf user … • export all • Miscellaneous • show uptime

BCrouter: commands & logging • Commands example bcrouter1#export all bcrouter1#login user kuleuven/u0022948 ip 10.91.91.1 reason login demo 200 OK: login - 1142031358930300 - kuleuven/u0022948 (1) on 10.91.91.1 (login demo) bcrouter1#show ip user kuleuven/u0022948 204 OK: show ip user - 10.91.91.1 bcrouter1#show user ip 10.91.91.1 203 OK: show user ip - kuleuven/u0022948 bcrouter1#login user kuleuven/u0022948 ip 10.91.91.2 reason 2nd login 200 OK: login - 1142031429848045 - kuleuven/u0022948 (1) on 10.91.91.2 (2nd login) bcrouter1#show ip user kuleuven/u0022948 204 OK: show ip user - 10.91.91.2,10.91.91.1 bcrouter1#export all conf user kuleuven/u0022948 …. conf ip 10.91.91.1 … login user kuleuven/u0022948 ip 10.91.91.1 reason recovering statefull info conf ip 10.91.91.2 …

network segment Time of log Log sequence number Name of segment Traffic counters Bytes and packets Download and upload Accounted and not accounted Dropped and accepted Number of active IP’s host Time of log Log sequence number IP address Username (–none- if no login) Traffic counters Bytes and packets Download and upload Accounted and not accounted Dropped and accepted BCrouter: commands & logging • Logging • Log commands and responses • Log network/host statistics

BCrouter: routing & Netfilter Routing with BCrouter is done by a BCPOLICER target in the PREROUTING mangle table that alters the fwmark value of the packet and uses this value as selector for policy based routing.

BCrouter: routing & Netfilter • Use Linux networking capabilities • IEEE 802.1Q support (VLAN technology) • Used to limit the number of physical interfaces • Policy based routing (Routing rules) • Used for implementing user groups • Netfilter/Iptables framework • Used for host exception lists

VLAN 1 VLAN 2 dot1Q ‘trunk’ containing VLAN 1,2,3 VLAN 3 VLAN enabled Device VLAN enabled Device BCrouter: routing & Netfilter • IEEE 802.1Q support • Virtual LAN technology (VLAN) • Operates on the data link layer (OSI layer 2) • Adds 4 extra bytes to existing ethernet header • Allows multiple LAN’s over 1 physical wire (trunking) • Each VLAN id has its own interface device • E.g. eth0.5 indicates VLAN id 5 on physical interface eth0 • ‘vconfig’ tool

Bandwidth management and optimization