100 likes | 241 Views
Communications Infrastructure Committee. Update for LAN Managers Friday, May 27, 2011. Initial Building List. OT – Done* Clark - Done* Shepardson - Done* ARBL - Done A/Z - Done NESB – In process Engineering VTH Atmospheric Science * Single GE for now.
E N D
Communications Infrastructure Committee Update for LAN Managers Friday, May 27, 2011
Initial Building List • OT – Done* • Clark - Done* • Shepardson - Done* • ARBL - Done • A/Z - Done • NESB – In process • Engineering • VTH • Atmospheric Science • * Single GE for now IAC Update
10G Connectivity Proposals • One upgrade in process, as part of a funded NSF proposal • Evaluating 5 additional requests for high-speed connectivity • Goal is to make a funding decision very soon IAC Update
Operational Management Discussion • Issues and Difficulties • Responding to problems/outages • Typical examples of problems • Edict from VPIT • Open Discussion IAC Update
Issues and Difficulties • Switch Consistency (Type and Configuration) • Documentation (Change mgmt., ports, vlans) • Security (Physical and Virtual) • Resources • Hardware inventory • People – delegation of responsibilities (configurations, responding to outages, etc.) • Multi-departmental occupancy of buildings • New technologies • Radiation Safety Project • VoIP – exacerbated by all of the above IAC Update
Issues and Difficulties (Cont’d) • 900+ switches • 120+ MDFs/IDFs • 40,000+ systems • Video, Environmental Controls, Power Meters, Voice, Police/Fire, Security Panels, Servers, Wireless, Private College LANs, etc… • Main campus, West Campus, South Campus, Remote sites comprising • 213 Subnets • 60 Subnet managers
Response Issues w/ Current Model • Typically, 90% of time (2-4 hours, average) is spent figuring out what changed, then a quick fix (put it back the way it was) • 10% of the time, problem is evident but very difficult to find the source and/or remedy • “Flying blind” is very difficult • We need to develop and maintain accurate cabling records • E911, CALEA Act Compliance, etc. IAC Update
Small sample and examples of concerns • Environmental controls offline due to multicast storm, After hours response by Facilities, April 2011 • Looped network takes down environmental controls requiring reconfiguration of switches and further monitoring after hours by Facilities, NOC, April 2011 • Firewall not configured to be on network and thus not firewalling, March, 2011 • Several building networks down along with environmental controls due to dual-homed server misbehaving. Discovered hubs and other devices as part of debug process. March, 2011 • VOIP and Wireless down due to cables being moved to wrong switch ports. March, 2011 • Compromised machines swamp entire LAN affecting police/fire, March 2011 • Departmental servers down, nobody knew location of switches affecting servers nor configuration of switches thus requiring NOC to respond, find, fix, March, 2011 • Unit carrying police/fire notifications moved to wrong port on switch. After hours response by NOC, Facilities, Feb. 2011 • Lost communication on large campus network due to switches being plugged into each other incorrectly. Feb. 2011 • Network device with duplicate IP takes down a College’s central server, Feb 2011 • Emergency fiber run had to be made to bypass departmental network device incapable of handling vlans for cardkey access, Feb, 2011
Directive, from VPIT • Must ‘harden’ our network for future applications, including VoIP IAC Update
Discussion • Operational management of switches is becoming more complex, and more critical • QoS, 802.1X, VoIP, documentation, life & safety, etc. • Need to “harden” the network, both physically and operationally • How can we do this, improving integrity of the network without impeding day-to-day functionality our users require? • Virtual vs. Physical Demarc IAC Update