Outsourcing ResNet

1. ResNet Ithaca College RESIDENCE HALL COMPUTER NETWORK Outsourcing ResNet One Institution’s Experience

2. Welcome! • A copy of this presentation is available at: www.ithaca.edu/staff/dweil David WeilDirector of Web, Systems and Departmental ServicesInformation Technology ServicesIthaca Collegedweil@ithaca.edu

3. Ithaca College • Private, residential undergraduate liberal arts college in upstate New York • Comprehensive, masters one institution • 5 schools & aninterdisciplinary division • Over 100 degree programs

4. Director of ITS Information Systems and Services Network and Communication Services Technology and Instructional Support Services Web, Systems & Departmental Services • HR, ERP, SIS • 19 people • Networks • Phones • 9 people • Field Service • User Support • 14 people • Web Dev. • Sys Admin • ITS Business • 18 people Information Technology Services

5. Residence Hall Network • Phased in over 3 years in early to mid 90’s • “One port per pillow” • Today - 53 residential halls with 4,400 ports

6. ITS provided ResNet • Flat network, 10/100 Mbps to the desktop • “Free” – no additional charge for the students • Shared main college Internet connection • ResNet bandwidth was allocated as a percentage of our main connection • Eventually allocated bandwidth by time of day (shifted more bandwidth to ResNet at night, less during day) • Shaped Internet traffic (allocated very little bandwidth to identified P2P traffic)

7. 38 Mb 3 Mb Bandwidth Allocated to ResNet

8. ResNet Pressures Build & Change

9. Fall 2003 – It boils over • Viruses, Worms, and other self-propagating nasties • And their significant impact on network performance • RIAA, MPAA, ESA… • Insatiable Bandwidth Desires • P2P “whack-a-mole” • Internal debate regarding the purpose of ResNet and use of our resources, and what is appropriate use of ResNet (games, P2P, etc) • Staff needed for other projects

10. Fall 2003 – It boils over • August 2003 Blaster and Nachi strike and ResNet could not handle the traffic from infected machines • On the Friday before opening weekend we turned off the entire ResNet to prevent further infections • It ended up being off for 7 days while we developed procedures to deal with the situation and worked with students to get their machines protected and cleaned

12. Why was it so bad? • Flat network • The entire ResNet was one subnet, no way to contain an infected machine • Full 10/100Mbps Ethernet speeds • Allowed viruses and worms to spread like wildfire • An unprotected machine would be infected within 30 seconds of connecting to ResNet • No firewall protection for ResNet machines • Individuals were not used to having to worry about computer security or patching

13. And once past opening weekend… • We continued to take incremental steps to address many of the performance, support and network architecture issues • Had to remain vigilant and constantly turn off ports of infected machines • One or two infected machines would have significant impact on network performance • Saw a significant increase in P2P usage • RIAA, MPAA complaints increased significantly • Continued to experience a huge drain on support, network and other ITS staff – many projects were put on hold • Bottom-line: throughout Fall 2003 ResNet performance was sporadic and ITS staff stretched thin

14. Students (and Parents) React • Student created “ResNet Sucks” Website with on-line petition and call-to-action • Editorials and letters in student newspaper • Students deliver a petition to ITS with 1,175 signatures demanding a better ResNet • And of course, angry emails and phone calls to the helpdesk, ITS management, and the college administration

15. www.evanm.com/resnet/indexold.htm

16. www.ithaca.edu/ithacan/articles/0311/06/news/LSresnet_revol.htmwww.ithaca.edu/ithacan/articles/0311/06/news/LSresnet_revol.htm

17. Something had to change • If we were to continue to support ResNet ourselves, we would need to: • Significantly change and upgrade the network • Install better traffic monitoring and compliance systems • Provide additional network and support staff • Find additional space for student computer repairs • However, we were concerned about the ability to find funding and staff resources to support this for the long term

18. Thoughts turned to outsourcing • Initial requirements: • Tiered service plan including a base no-cost option for the students • On-site support • Increased bandwidth • Robust network architecture that could better withstand viruses, Trojans and worms • A true partner interested in a collaborative relationship and who understands higher education

19. We solicited proposals • A major nation-wide telecom • The local cable-TV / broadband franchise • A small company specializing in supporting residence hall networks

20. The nation-wide telecom • Limited experience with this type of network and service • Limited experience with the academic environment

21. Local cable-TV / broadband provider • Would provide a significant discount for their broadband service • Based upon a guarantee # of subscribers from us • The college would need to make one payment to them, and have students pay the college for the service • Would not provide tiered or free service options • No direct on-site support

22. Local cable-TV / broadband provider • Would need to build an entirely new network infrastructure in the residence halls • Current cable-TV infrastructure could not support broadband service • College would be unable to support this cable-TV based network should the partnership end

23. Small Residence Hall Provider • Agreed to meet all of our requirements • Tiered service options (including “free” option) • On-site support • Increased bandwidth / network redesign • Understood residence hall networks and the academic culture • CEO very effective at talking with the students about their concerns and issues • Interested in a partnership-type of relationship

24. Small Residence Hall Provider • Concerns: • Located in Austin Texas with only two current customers • We would be their largest deployment (and farthest away) • Small company, operating “lean”

25. Selected Apogee Telecom • We felt that these concerns were manageable • Selected the small residence hall provider, Apogee Telecom • Felt they were the best overall fit for us

26. What we outsourced… • Redesign of the Residence Hall network • Network monitoring, troubleshooting, repair and replacement • Custom service levels and prices that met our particular needs and budgets • Management of the account sign up process • Student billing • Marketing and PR

27. What we outsourced… • An on-site technician for troubleshooting student network issues • A call center open 7 days-a-week • The ISP of record for DMCA and other complaints • Work collaboratively with the college for responding to DMCA complaints • College gets copies of all notices and still may impose judicial sanctions

28. Apogee Telecom – The Network • Redesigned our residence hall network and “over built it” • Provided 3.5 times the bandwidth (OC-3, 155 Mbps total) • Used our switches, but replaced most other components • Installed two Allot Communications NetEnforcers to control network traffic and bandwidth allocation • Significantly limited the number of computers on any one subnet • Students allocated bandwidth based upon their service level • Could use that bandwidth for whatever they wanted (any “legal use”)

29. Fall 04 Apogee Service Plans Internet Access *High Speed = 10/100 Mbps to all ithaca.edu resources

30. Apogee Service Costs • The College pays Apogee a base fee per month to cover: • 25 Mbps administrative internet service • “Free” 64 Kbps basic-level Internet service for all students • High-speed access to on-campus resources for all students • Apogee help desk support for ResNet students • This base fee is roughly equal to the amount we used to pay to our old campus network provider for the T3 service • so in the end, there was no increase in costs to the college

31. The Implementation March – August 2004

32. Timeline Contract signed Communicated plans to campus Technical site visits and design work Cable trenching Implementation sub-teams created Support procedures created Student antivirus strategy determined Network design finalized Onsite support staff interviewed and hired Legal / policy issues agreed upon Site coordinator starts Network infrastructure upgraded ResNet by Apogee Live March April May June July August

33. Campus Communication • Apogee CEO came to campus and met with administrators, student government, and conducted an open town hall meeting • Announcement letters sent to students and parents detailing changes • Ads placed in the student paper • On-going dialogs with student leaders and parents to respond to concerns…

34. www.evanm.com/resnet/index2.htm

35. A petition to ITS with 1,175 signatures responding to a February 26, 2004 student newspaper article. They request that we work with a vendor such as Apogee to fix ResNet and not impose any additional fees for what they consider a basic and necessary service.

36. Account Sign-ups • No matter what service level, people need to sign up for an Apogee ResNet account • Advance sign-up mailings sent over the summer to all students living in residence halls • Freshman sign-ups at orientation (June/July) • Could sign up via Web, phone or paper form • 2,892 people signed up in advance (67%)

37. Fall 04 Apogee Subscriptions As of December 28, 2004

38. Activation Process • A student plugs their computer into a ResNet port • Automatically directed to Apogee login page • Includes a link to the account sign-up page if needed • Student activates computer using Apogee supplied username and password • Only need to do this once per semester

39. Cutover and Move In – Here’s Murphy • Last of the new equipment cutovers occurred shortly before opening weekend • Major fiber-cut 60 miles away took down service the week before opening weekend • A few days before opening weekend a network problem developed where students were not getting their subscribed-to level of service • Would result in a MAJOR PR problem if it didn’t work! • Apogee was able to put a workaround in place late at night on the Friday of opening weekend

40. Opening Weekend • Our helpdesk and offices were quiet (it was strange!) • Apogee’s call center was busy, on-site staff were very busy and some students had to wait a few days for on-site help • Overall things went smoothly • Few if any complaints ever came to us regarding problems • ITS staff were unsettled by not having the ability to directly monitor ResNet performance or call volume to the Apogee support center

41. Early Fall • A few scattered equipment problems • One particular residence hall experienced a few interruptions due to someone repeatedly plugging an improperly configured wireless router into the network • Overall, things went well

42. Other (minor) issues that came up along the way… • Different expectations for who would be on-site during opening weekend • ITS staff and management had some trouble “letting go”….

43. How’s it working? • ITS has had time to focus on other things • “This has been my most productive fall in a very long time” • Students seem to like the service • Apogee conducted a customer survey in late fall. Out of 4,193 survey’s sent out, received 1,067 responses (25.4%)

44. “Connection is stable and reliable” Total responses: 1,067

45. “I am satisfied with the speed I signed up for” Total responses: 1,067

46. “The Apogee call-center support staff was knowledgeable in troubleshooting my issue” Total responses: 425

47. And even our friend Evan, seems to like the service….

49. Where do we go from here….

50. Spring 2005 • Continuation “as-is” • Continue to optimize network settings and support procedures