Network Operations Centers Training Programs

1. Network Operations CentersTraining Programs Quilt NOC Workshop Panel Discussion: CENIC Indiana University/Global NOC WiscNetPacific Northwest GigaPopOctober 3, 2007 Network Operations CentersTraining Programs

2. NOC Training Programs WiscNet Kika Barr October 3, 2007 Network Operations CentersTraining Programs

3. Who We Support • 53 Higher Education Institutions • Over 300 k12s • 12 CESAS – Cooperative Education Service Agency • 18 Technical Colleges • 13Library Systems • 36 Government Institutions • 1 museum • I hospital • BOREAS-Net WiscNet Network Operations CentersTraining Programs

4. Staff Workload • 100-200 tickets per month WiscNet Network Operations CentersTraining Programs

5. Our Staff • Size • 2 full time tech support specialists • 1 ¾ time tech support specialist • 1 Network Architect • Mix • The Tech Support Specialist are strong generalists • Yet, the staff tend to divide itself on the level of the expertise by service (Jack of all trades, master of some) WiscNet Network Operations CentersTraining Programs

6. Recent Training Content • Training offered to staff in the last 6 months • Seminars about Security • Fiber training • Fiber Optics training • Firewall training • Cisco related training • Juniper training WiscNet Network Operations CentersTraining Programs

7. Training Resources • Recent training was designed and provided by; • Vendors • Cisco • Packeteer • Engineers on staff WiscNet Network Operations CentersTraining Programs

8. Training Budget • We allocate training budget from the organization budget • Try to stay local to save on expenses • Take advantage of vendor training opportunities WiscNet Network Operations CentersTraining Programs

9. Training Logistics • Methodology for scheduling training • When training is out of the office, we normally just send 1 tech support member • When training is in house, our NOC is made aware that staff is on training. Tech Support specialist have their laptops on with IM so NOC can reach us if there is an emergency • Voice mail to Tech Support goes to support@wiscnet.net mail list • Where do you train? • In house using conference rooms. No training rooms available • travel WiscNet Network Operations CentersTraining Programs

10. Lessons Learned • If you don’t use it, you lose it • Consistency and relevance • Delicate balance between overdoing it and not doing it enough WiscNet Network Operations CentersTraining Programs

11. Training Program Example • WiscNet Support Staff if Tier 2 • Training as needed • Have been discussing with engineers about internal training and a more structured planned • Competencies: • Troubleshooting • Ticketing • Tools • Customer Services • Service Expertise • Optical networks around the corner WiscNet Network Operations CentersTraining Programs

12. Demonstrated Competencies • Excellent phone skills • Excellent understanding of WiscNet services • Technical proficiency in several areas of technology – router configuration, DNS, troubleshooting, firewall configuration, WAN infrastructure, and Optical networks WiscNet Network Operations CentersTraining Programs

13. Keeping the Program Current • Challenge for us because of time constraints • Hoping to work with WiscNet engineers to design a more predictable training train WiscNet Network Operations CentersTraining Programs

14. NOC Training Programs WiscNet Kevin Schmidtke October 3, 2007 Network Operations CentersTraining Programs

15. Who We Support • Number and type of sectors your organization supports • UW Madison Campus – network backbone, wireless • WiscNet – statewide educational network • BOREAS Net – regional optical network, research focused WiscNet Network Operations CentersTraining Programs

16. Staff Workload • # of incidents opened per month • 300 Clarify cases opened on average • WiscNet cases – 68% • UW Madison cases – 29% • BOREAS Net cases – 2% WiscNet Network Operations CentersTraining Programs

17. Our Staff • Size of Staff • 10 staff • 7 Professional Senior • 3 Professional Junior • 1 Process & Procedure coordinator • Staff Mix • All Junior staff scheduled with a Senior staff member. Seniors perform in a lead capacity. WiscNet Network Operations CentersTraining Programs

18. Content • Examples of training offered to your staff in the last 6 months • Infinera DNA application training • Infinera hardware configuration training • Modified WiscNet process and procedures WiscNet Network Operations CentersTraining Programs

19. Training Resources • Who designed the content and provided the training for your staff in the last 6 months • Most training was developed in-house • Most training was delivered by subject matter experts on staff WiscNet Network Operations CentersTraining Programs

20. Funding Sources • Two Sources • New network support contracts are expected to deliver training to all staff as part of the service rollout and acceptance • Training budget allocation for each staff. Traditionally this is used for specific professional development WiscNet Network Operations CentersTraining Programs

21. Training Logistics • Scheduling training • Training is typically presented in a minimum of two sessions to ensure all staff can attend. Rarely do we have an all-staff training session • Training facility • Several training rooms are available in data center complex WiscNet Network Operations CentersTraining Programs

22. Training Program Example • Knowledge Database Repository • All formal training results in an article or a series of articles • Many departments have instances of the knowledge database and articles can be shared from one instance to another • Only the originating instance administrator can modify the content of the article WiscNet Network Operations CentersTraining Programs

23. Training Program Example • Knowledge Database instance administrator • Creates all new knowledge database articles or recruits a subject matter expert for content • Reviews all articles originating from another department • Is responsible for making all NOC staff aware of new process or procedure • Ensures all staff review and accept responsibility for content of articles WiscNet Network Operations CentersTraining Programs

24. Training Program Example • Knowledge Database Lifecycle • All articles start out in draft form • All articles are set to team review • All staff members ‘sign off’ on content of the article • All articles are reviewed at a minimum of 6 months for accuracy and relevance • Articles are reviewed by Instance Administrator and/or subject matter expert WiscNet Network Operations CentersTraining Programs

25. Lessons Learned • Transition from hardcopy to online knowledge database required trust in materials. • All articles are keyword searchable. Review failed searches to determine better ways to define articles • Keep a copy of knowledge database on stand alone server on separate UPS WiscNet Network Operations CentersTraining Programs

26. NOC Training Programs Pacific Northwest GigaPop Linda Hornung October 3, 2007 Network Operations CentersTraining Programs

27. Who We Support • PNWGP • University of Washington Campus • University of Washington Medical Centers • Washington State K-20 network • TransitRail (4/2007) PNWGP Network Operations CentersTraining Programs

28. Staff Workload • Our NOC opens 900-1000 tickets per month across the various customer bases we handle PNWGP Network Operations CentersTraining Programs

29. Our Staff • Full time staff in the NOC and Computer Operations, who provide 2nd and 3rd shift coverage. • Four tier 3 • Seven tier 2 • Five tier 2 for off-shifts. • New NOC students – 6 students work 4pm-8pm M-F, and 9am-6pm Sat/Sun. Considered tier 1. PNWGP Network Operations CentersTraining Programs

30. Recent Training Content • Several of our staff have been to Cisco router classes and Cisco ONS classes. • Several others attended Juniper ONS classes provided on the UW campus. • Professional development courses such as leadership, communication, project management, team building. • Many presentations by Network Engineering and Network Architecture teams on new services and network design that have recently been implemented. PNWGP Network Operations CentersTraining Programs

31. Training Budget • No defined budget, but this past year has seen a tremendous increase in training opportunities! • We have been able to have training from outside companies, our vendors, the UW Professional Development program, and many in-house experts. PNWGP Network Operations CentersTraining Programs

32. Training Logistics • Because of the different shifts and the need to provide continuous coverage, we will always offer at least two different training sessions for NOC and other interested staff. • Usually train in local conference rooms. • Vendor classes have been held in larger facilities on campus. Always in well-connected facilities.  PNWGP Network Operations CentersTraining Programs

33. Lessons Learned • The staff that usually does our NOC training must have taken a Train the Trainer course. • Have defined training goals. • Use mixed media – don’t just talk. • Engage the audience through occasional questions or scenarios. • Provide take-away reference material. PNWGP Network Operations CentersTraining Programs

34. Training Program Example • Tier 1 staff: the focus is on escalation within the NOC and being able to gather appropriate initial information. • Competencies: • Troubleshooting • Ticketing • Tools PNWGP Network Operations CentersTraining Programs

35. Training Topics for Tier 1 Staff • Architecture overview • Services overview • Customer information • Monitoring • Opening trouble tickets • Handling and issuing outages • Escalation PNWGP Network Operations CentersTraining Programs

36. Demonstrated Competencies • Basic phone skills (answer, hold, transfer) • Searching tickets and email for previous information • Using tools to find out information on equipment location, customer contacts, circuit ID, etc. PNWGP Network Operations CentersTraining Programs

37. Keeping the Program Current • Continuous project to keep the content updated and accurate • Reliance on an internal wiki for procedure and information PNWGP Network Operations CentersTraining Programs

38. NOC Training Programs CENIC Cindy Abercrombie October 3, 2007 Network Operations CentersTraining Programs

39. Who We Support • CalREN Community (-approx site count) • California Community Colleges - 112 • University of California - 11 • California State University - 25 • California K-12 System - 74 • California Private Institutions - 9 • Associates and sites outside California - 11 • NLR • Layer 1 Services • Support equipment / deploy services across 250 sites • Pwave, CIIX, TransitRail, AARNet • Support equipment and services CENIC Network Operations CentersTraining Programs

40. Staff Workload • CalREN Monthly Ticket Stats • Avg 670 tickets opened • Community requests for support and outages • Deployments, upgrades, maintenance • Avg 375 support tickets resolved • Community requests for support and outages • NLR Monthly Ticket Stats • Avg 105 tickets resolved • Deployments, outages, maintenance CENIC Network Operations CentersTraining Programs

41. Our Staff • 15 NOC Engineers • 1/3 of our team have > 5yrs. Network support experience • 2/3 of our team have 2-3 years combination LAN/WAN and end-user technical support, with 1-2 years large-scale network support • Other Network Operations Support Resources • Our senior engineering team works with us daily on all types of request • Our Sys Admin team responds to NOC needs quickly and provides support to our community too CENIC Network Operations CentersTraining Programs

42. Content • Examples of Training Provided • Cisco 15454 MSTP Test and Turn-up • Configuring BGP on Cisco routers • Cisco CRS-1 Essentials • DNS in Practice CENIC Network Operations CentersTraining Programs

43. Training Resources • Cisco • Global Knowledge • CENIC Senior Engineers • Outside Specialty Organization CENIC Network Operations CentersTraining Programs

44. Training Funding & Logistics • Methodology for scheduling training • 2 training sessions on same topic, sometimes weeks apart or months apart • Reduce upgrade, maintenance and deployment activities • Reduce NOC coverage and include senior engineering resources • Purchase bulk training hours and use Cisco Learning Credits CENIC Network Operations CentersTraining Programs

45. Lessons Learned • Discussing training opportunities during recruiting process • Coordinate training with at least two people at a time, the more the better • Purchasing bulk training hours is more cost effective than a couple of courses at a time • If course feedback for a particular trainer is good, request the same trainer for future courses • Make training and professional development a regular part of communications and schedules CENIC Network Operations CentersTraining Programs

46. Day1 Training Program • Start of a Shift • Tools • Process • Escalations • Schedule Maintenance • Outages • Announcements • Contact Information • Where • Updates and requests for info CENIC Network Operations CentersTraining Programs

47. Non-Technical Training Topics • High visibility sites • Education acronyms • Our role in other network support organizations • Preparing turn-over work for the next shift • Standard signatures and greetings • How we do lunch • Program your cell phone • How to report you are delayed or absent CENIC Network Operations CentersTraining Programs

48. Demonstrated Competencies • Using tools to find out information on equipment location, customer contacts, circuit ID • Searching tickets and email for previous information • Document and articulate the known facts of the current issue • Know how to use our vendors’ processes CENIC Network Operations CentersTraining Programs

49. Keeping Network Operations • Everyone owns network operations • Sharing is caring = keep wiki current, send email • Ask questions to understand b/c there’s too much to memorize CENIC Network Operations CentersTraining Programs

50. NOC Training Programs IU/Global NOC Steve Peck October 3, 2007 Network Operations CentersTraining Programs