High Performance Network Analysis

1. High Performance Network Analysis High Performance Network Analysis Enterprise Operate Practice Cisco Services Andrew Wojtkowiak – Network Consulting Engineer

2. Getting Started • Background • Cisco Services performed an assessment of the wired infrastructure to serve as a holistic health check of the University Corporation of Atmospheric Research network • Goal of the assessment • To identify immediate remediation needs • Provide Opportunities for network improvement

3. Discussion Flow High Level Findings Strengths and Concerns Background And Key Areas Assessed Executive Level Findings Encompassing Projects Remediation Steps Looking Forward

4. Background • The High Performance Network Analysis (HPNA) was performed to assure the stability of the core routing and switching infrastructure • Performed as a holistic network health check • Emphasis placed on Availability and Resiliency with the Campus environments • On-site interviews and data collection • Analyzed ~80 devices as part of the HPNA • Collected detailed network data such as topology diagrams, software, network standards, protocols, etc…

5. Key Areas Addressed • Network Topology • Protocol Resiliency • Network Service Resiliency • Hardware and Software

6. Analysis Findings

7. Strengths • Dedicated and professional network staff • Everyone we worked with was very open, professional and accommodating • Excellent Hardware and Software replacement strategies • Hardware and Software is kept up to date and staff is knowledgeable of bugs and vulnerabilities • Change Management Process • Well documented and followed change management process • Individualized tools for Network Management • Tools for deployments, configurations, backups, and management

8. Concerns • Single Points of Failure • Increased risk of a pervasive network incident; scalability and availability concerns • Process Documentation • Lack of formal process to follow. No repeatable steps that all team members can use. • Global Configuration Templates • Templates will help reduce configuration inconsistencies and ensure services are configured according to policy • Configuration Inconsistencies • Increased time to repair due to troubleshooting overhead; decreased network security; compliance risk

9. Single Points of Failure • A few single points of failure • TCOM switch for internet connectivity • Foothills Lab secondary switch • NWSC second switch • Major risk with TCOM • Higher latency backup • Foothills under construction, second switch in move • NWSC secondary switch is being considered • Foothills and NWSC would limit connectivity from those locations to the rest of the network. • Current State • Network Risks • Financial Risks

10. Example Single Point of Failure

11. Process Documentation • Processes are well defined by the individuals who perform the tasks • Software and Hardware replacement • Standards for implementing new devices • No actual defined documentation • Only certain people are well versed in processes • Not easily reproducible • No defined steps for changes • Allocate time to turn processes into documentation • Allocate someone to review the documents • Keep them up to date as they change. • Current State • Network Risks • Recommendations

12. Software Resiliency Findings All CatOS has reached End of SW Maintenance, and will no longer receive attention with regards to defect or security vulnerability patching

13. Global Configuration Templates • Configuration standards are adhoc; without formal documentation • No way to perform configuration compliance to a template* • Number of configuration inconsistencies and errors (Protocol, Service, Security) • Network unpredictability • Potential increased troubleshooting overhead and operational difficulty • Prolonged loss of connectivity and service interruption to critical applications • Increased exposure to security vulnerabilities • Increased cost associated with operating the network • Current State • Network Risks • Financial Risks

14. Configuration Inconsistencies • HSRP inconsistencies • Partially configured advanced spanning tree features • Optimize/Standardize Spanning-tree priorities • OSPF passive interface • Some routers do not have a peer • Possible loops or rouge switches influencing the network • Routing updates are not limited • Implement changes to the network to remediate the smaller configuration inconsistencies • The standard templates will assist in ensuring fewer deviations from standard. • Current State • Network Risks • Recommendations

15. Other Considerations

16. Implement a Standalone Network Core • Three buildings connected in a partial mesh topology • Collapsed connections to each other • Port density growth at N*(N-1) rate for every new building • Lack of modularity and scalability • Large fault domains across all buildings • Network disruption and outages • Increased troubleshooting overhead • Quantifiable cost increase in both capital and operational expenditure • Current State • Network Risks • Financial Risks Cost to Add 4th Building Nx(N-1) = 12 Ports (6 Links) Additional Capital Expenditure associated with running fiber Additional Operational Expenditure associated with design complexity

17. Need for Core Current Topology - No Core • Fully-meshed distribution layers • Physical cabling requirement • Routing complexity

18. Recommended Design Center Green Mesa Lab Foothills • This leading practice hierarchical design has been proven to: • Promote easy growth and ease of troubleshooting • Reduce capital and operational expenditure • Create small fault domains • Promote deterministic traffic flows • Enable logical and physical topology mapping New Location Dedicated Core Dedicated WAN / Internet Switch Block Research Networks TCOM/FRGP Firewalls Internet

19. Implement Network Security • Monitoring facing the Internet • Intrusion Prevention • SPAN Sessions to security team • Extensive ACLs on core switches • No Control Plane Policing to protect devices • Limited methods to log and account for network incidents • Increased CPU usage on switches • Create method to evaluate internal ACLs routinely • Consider Control Plane Policing for basic router/switch services • Routing • Switching • Current State • Network Risks • Recommendations

20. 4 2 1 3 9months 0-6 months > year High priority Must Do – Reduce Risk Quick Wins – High Business Impact Correlating business impact (risk reduction) to ease of execution and exemplar implementation time Project List: 1) Remediate single points of failure Create, utilize and maintain global configuration standard templates Create, utilize and maintain process documentation Remediate configuration inconsistencies within the network Very Hard Easy But Low Return Low priority GAP Prioritization – Where To Focus More complex to implement Easy to implement

21. Q & A