GTA Network Management Systems

1. GTA Network Management Systems On Behalf Of BellSouth

2. Navigation • Moving through these lessons is accomplished with the left mouse button and the keyboard cursor arrows. • Take all the time you like to read or study images, and advance to the next page whenever you are ready. • Whenever the blue mouse is visible in the bottom right corner, you may click your mouse button to advance. • You may also use the keyboard left cursor arrow Q to back up to review any previous information or to restart a page. Z QYR Click your mouse to go to the next page

3. Introduction • The Customer Network Management Portal (CNMP or Portal) is a web application that gives the agency detailed information on the status of the network for IP services customers. • The Portal controls access to: • Netcool – Reports on network outage events • InfoVista – Tracks network performance • CTTS – Customer Trouble Ticketing System • This training will provide instruction on how to use each application: • CNM Portal • Netcool • InfoVista • CTTS

4. CNM Portal Management • Portal management will be administered by the GTA Network Management Team, including: • Establishing new users • Resolving user issues • Changing user entitlements (these define permissions) • For a new portal ID, email gtasolutionsmrktg@gta.ga.gov or JJCHAMBE@gta.ga.gov • To report trouble,call the Command Center at (404) 656-7378.

5. CNM Portal View • Everything within the Portal will be determined by your user ID. • When you log in to the portal, your user ID will define… • Which GTA Region or Agency levels you should be able to see • What devices you may poll to determine status • Which devices and routes you may query to measure response times • What tickets you may open to read updates

6. CNM Portal User Entitlements • Levels of views are dependent on permissions: • TCC / NOSC users will be able to monitor devices down through all levels • GTA Regional Operations will be able to see all levels within their geographic area • Agency users will be able to see within their agency and sub-agencies statewide • Sub-Agency users will be able to see only within their sub-agency Agencies will be able to see within their agency regardless of GTA Region, and retain visibility into their sub-agencies regardless of GTA Region. TCC / NOSC GTA Regional Ops Agency Sub-Agency

7. CNM Portal Interface To access the CNM Portal, open an internet browser and go to https://cnmp.bellsouth.net/portal/ Enter a User ID and Password, then click Login The portal will open and display a tabbed interface… dfcsuser       and a list of functions that are currently accessible At any time, click home to get back to this page Click logout when finished – before closing the browser

8. CNM Portal Interface • The CNM Portal consists of a single sign-on point, and controls access to other applications and functions: • User Management –Modify Personal Information • Netcool – Alert Management System • InfoVista – Performance Monitoring Tool • CTTS – Customer Trouble Ticketing System Other tabs and functions are either assigned to the Administrator, or not yet available.

9. Modify Personal Information User ID Properties – cannot be modified To modify your personal information, select User Management, and then select Modify Personal Information Personal Information – change if needed Change Password – enter old & new password Select Modify to save the changes

10. Naming Convention GTA GDOC 1 003 Each device needs a unique name to identify it on the network The device name will start with GTA as the first three letters So it will work like this: State Level (GTA) GTA We will examine GTAGDOC1003 and see how the name is derived The next four letters are derived from the Agency name Agency (GDOC) GDOC GTA Region (1) 1 The next single number indicates the GTA Region Site (003) 003 The last three numbers indicate the specific Device (003) The number after the site location 003, indicates the number of routes or devices at that location. That means that 003 (2) has 2 managed routes or devices at that location that we will be able to monitor using the CNMP applications.

11. CNM Portal Summary • Each of the Portal selections covered here will launch an application in the same window, below the Portal interface tabs. • At any time, you may make another selection from the Portal to change applications, or logout of the Portal when you are finished.

12. Netcool Introduction • Netcool is a network outage event application that operates within the CNM Portal. • Netcool gives the agency real-time information on the status of the network for IP services customers with color-coding for quick at-a-glance network diagnostics.

13. Netcool Access Levels • Level of view within Netcool is dependent on the Portal username: •  GTA Management and NOSC personnel will see the highest • levels of the MPLS network, and can drill all the way down •  GTA Regional operations will see all alerts within their GTA Region •  Agencies will see all alerts within their agency •  Sub-Agencies will see all alerts within their sub-agency

14. Netcool • The Netcool interface is displayed. • To access Netcool from the CNM Portal, click on the Network Management tab and select View Fault Events.

15. Netcool Alarm Indicators All Netcool alarm events are color coded by severity, and categorized by status. The severity color indicates the importance or urgency of the event, and the status indicates whether the event has been acknowledged. Green – Clear event. No active alarms. Purple – Indeterminate – An event that has been resolved. This usually occurs when a problem has been resolved or cleared and the node or interface is still returning to normal. • Red – Critical alarm - Service impacting alarm. • Examples: • Node down (NMS lost network connectivity to device) • Node unreachable (NMS lost route to device) • Network unreachable (NMS lost route to network) Light Blue – Warning Examples: CPU back to normal (CPU < 60% busy) Interface error returned to normal (< 3 error rate) Memory returned to normal (> 20% available memory) SNMP agent up (NMS can query device) Node up (connectivity restored to device) Buffer is back to normal (alarm automatically clears after 4 hour waiting period) Orange – Major alarm - Service impacting alarm. Examples: CPU alarm (CPU >= 85% busy) Interface error alarm (> 5% interface error rate) Memory alarm (10 % available memory) Link down alarm (interface lost carrier or LMI) Before we get too far into Netcool, let’s look at the alarm colors and meanings that will be displayed in the Netcool console. Yellow – Minor alarm - Non-service impacting alarms Examples: CPU alarm (CPU >= 60% <= 84% busy) Interface error alarm (3% - 5% interface error rate) Memory alarm (>= 10% <= 20% available memory) SNMP agent down alarm (String; nms can not query device) Buffer failure alarm (buffer failures have occurred on device)

16. Netcool Interface It is important to understand that the main Netcool screen is presenting the same event information in fourdifferent ways: 1. Network Status 2. Alarm Summary 3. GTA Region Status 4. Agency Status By looking closer at each of these four sections, we can determine just what each section represents.

17. 1. Network Status Network Status displays the total number of all alarm events, along with the color of the most serious current alarm event. This means that there are 52 total events of different severities, and the most severe is a Critical (red) event.

18. Event Details List Click the All Events icon to open a detailed description of all 52 events, sorted from most serious to least serious. Scroll down for details on the lesser alarms not visible.

19. Alarm Event Details 1 2 3 4 5 6 7 8 9 • There are 9 fields in the information displayed once an object has been selected: • Event ID- a system generated unique identifier number • Remedy Ticket - The Remedy ticket number, once it is assigned (BellSouth ticketing system) • Node - The device name for the affected route or device • City – The city in which the device is located • ACK – Indicates whether the alarm has been acknowledged. • Summary – A brief summary of the alarm • First Occurrence – A Date/Time stamp of the alarm, indicating the time/date when the device first went into alarm • Count – The number of times a device has alarmed since the initial alarm • Last Occurrence - Date/Time stamp of the last time the system indicated an alarm

20. 2. Alarm Summary Of the 52 alarms listed in All Events, the Alarm Summary section displays only the events that fit into these parameters: Click any of the alarm icons listed in Alarm Summary to open a list with the details for those alarm events. • Critical An outage affecting more than one site • Major Outage at an individual site • Last 30 Mins Events in the last 30 minutes • Count > 5 Error rate greater than 5% on a given interface • Minor Any error that does not cause an outage

21. 3. GTA Region Status Compare the orange Athens icon to the alarm color codes, and we’ll see that the most serious alarm in that GTA Region is a Major alarm. If we compare the red Rome icon to the alarm color codes, we’ll see that the most serious alarm in that GTA Region is a Critical alarm. Each Region icon color corresponds to the most serious alarm within that GTA Region. Remember the alarm color codes? Click any of the GTA Region alarm icons to open a list with the details for those alarm events in that GTA Region. This is the list generated by clicking the Rome icon.

22. 4. Agency Status Similar to the GTA Region status, the Agency icons represent the highest status of alarm events within each Agency, regardless of GTA Region. Click any of the Agency alarm icons to open a list with the details for those alarm events in that Agency. This is the resulting list generated for DLAW, with most serious events listed first. Since they have red icons, DHR and DLAW both have Critical events as the highest alarm. With yellow icons, DOC & DTAE both have Minor events as the highest alarm. DMVS has Indeterminate as the highest level alarm. DJJ, DNR, and CSB are clear, with green icons indicating no current alarms.

23. Netcool GTA Region or Agency View • Level of view within Netcool is dependent on the login username: •  GTA Management and NOSC personnel will see the highest • levels of the MPLS network, and can drill all the way down •  GTA Region operations will only see alerts within their GTA Region •  Agencies will only see alerts within their agency •  Sub-Agencies will only see alerts within their sub-agency The 4 GTA Region purple icons are regions where DOC has offices with active alarms, so the agency has visibility into those GTA Regions, but only as it pertains to DOC. The Netcool main screen we have been working with is from a GTA perspective- all agencies and all GTA Regions are visible. An Agency or GTA Region view will look similar but will show fewer results belonging only to that Agency or GTA Region. Here is a GDOC Netcool screen. All Events has only 5 total alarms, and the most serious is indeterminate. We know that because the All Events icon is purple. The Alb, Athens, Atl_Cen, Atl_N, Aug, and Mil icons are all green, which tells us there are no alarm events for the GDOC offices within those GTA Regions. The DOC agency icon is purple, and all others are green. This doesn’t mean that all other agencies have no alarms – it means they are masked, and will not show any results with a GDOC user ID.

24. Netcool Summary • Netcool is a fault management tool. • It is important to remember that Netcool doesn’t measure performance; it only indicates the presence or absence of traffic from network devices and routes. • Similar to a ping command, as long as traffic is returned, it can be demonstrated that the network devices and routes are still intact and functioning. • When the traffic is not returned for a device or route, Netcool displays an alert to indicate that lack of a returned message. So if you receive a call reporting that the network is down, Netcool is your first verification tool in diagnosing the issues at hand. • To measure metrics indicating network performance or slow-downs, and to compare performance for select dates, select the InfoVista application within the CNM Portal.

25. InfoVista and Internet Metrics • Latency – The time delay for a packet of data to get from one point to anotherpoint • PacketLoss– The percentage of packets sent that will be lost (out of 100%) in transmission • Jitter–the variation in the time between packets arriving – critical in VoIP applications, but less important for email, data, etc. Latency is typically expressed in milliseconds, or ms, and measures the transmission time to travel to the destination and return. Contributors to latency include the transmission medium (wire, fiber, etc.), packet size, hardware involved, and other variables. Packet loss may be an indication of network congestion or other factors. Jitter is frequently caused by packet transmission delays. Latency shown here is 72 ms P I N G time=72 ms

26. InfoVista Interface • To access InfoVista from the CNM Portal, click on the network management tab and select View Performance Reports. • InfoVista will launch just below the Portal interface tabs.

27. InfoVista Tabs The InfoVista tabs allow access to device metrics and reports CPE Top & Bottom tab – Top & bottom 10 CPE devices rated by CPU usage and load Access tab – Metrics displaying latency, jitter, and packet loss for access lines and routes Access Top & Bottom tab – Top & bottom 10 access routes rated by latency Main page tab – Starting point for InfoVista CPE tab – Metrics for CPE devices displaying CPU & memory usage on the CPE, WAN Interface metrics, and Frame Relay metrics

28. InfoVista Time Navigator • Both the CPE tab and the Access tab contain the Time Navigator. • The final reports and graphs also contain Time Navigator functionality. • In addition to the 5 tabs available, you’ll need to know about the Time Navigator and the Device Navigator to locate the correct information. • The Time Navigator designates the date for any generated reports. It also specifies the time period length for the report.

29. InfoVista Time Navigator The purpose of the Time Navigator is to specify the date and time period prior to generating a performance report. The report date and time are specified using these controls:        • Navigation arrows • Time period (specified for report) • Calendar icon • Time period menu • GO button The navigation arrows will increase or decrease the time period by the time period displayed in the time period menu. If the time period menu is set to Daily, each arrow click will change the date by one day. If the time period menu is set to Monthly, each arrow click will change that date by one month. Each arrow click will ALSO refresh the display if a graph has already been generated, slowing the application considerably while waiting to reload it. The GO button is used to generate the report after entering a date directly into the field via keyboard. If the time period is altered using the navigation arrows or the calendar function, InfoVista will begin to create the report when the new date is selected. The time period may be directly entered into this field using the keyboard, or it may be changed with the navigation arrows to move it forward or back in one day intervals, or a new date may be selected using the calendar function. Once set, any generated report will contain data for this time period until the time navigator is reset or modified again. The time period menu selects the time period for a report – whether it will cover a day, a month, or a year. As the time period menu is selected, the time period on the left will change to reflect the new time increment. The calendar function is convenient for adjusting the reporting period by larger amounts. Click the calendar icon, use the left & right arrows at the top to display the desired month, and then click on the desired date to generate the report.

30. InfoVista Time Navigator As the time period is selected on the right side, the time period field on the left side will change to match that time increment. By selecting Monthly, InfoVista will change the left field again to allow for monthly choices. The same with Quarterly and Yearly selections, or all the way back down to Hourly if needed. Within the Time Navigator, some fields will change as you make your selections. Let’s look through the time period options, and watch the time/date field on the left side as it changes. Whether we want to look at performance metrics for yesterday, last week, or last month, the Time Navigator is the first stop to select the date for reporting metrics. The default time period is Daily. The date field to the left displays the date, and allows the date to be entered or altered. If we click the time period menu and change it to Weekly, the date field on the left will change to allow selection by weeks.

31. InfoVista Device Navigator • Both the CPE tab and the Access tab contain the Device Navigator. • Just as the Time Navigator selects the time period to examine metrics or create a report, the Device Navigator selects the target device or access route for any generated performance reports.

32. InfoVista Device Navigator Remember the naming convention example in the portal section? That example was taken right out of InfoVista. If you’re familiar with Windows Explorer, you’ll notice that the Device Navigator is similar. Just like in Windows Explorer, use the tree on the left pane to navigate or drill down to the desired location or device. Select the  symbol to view devices below. Select the next  symbol to view additional devices. …and select the desired target device.

33. InfoVista Device Navigator • Right-click on a listing below and a small menu will pop up. • Click on any selection to generate performance reports: • Shows a list of frequently used daily and weekly reports • Shows WAN Interface metrics specific to this device • Shows Frame Relay metrics specific to this device • Shows Router performance metrics specific to this device • Updates all visible panes to display metrics for the selected device Let’s look closely at GTAGDOC1001. The target site location 001 shows a (4) after it. This indicates that 4 devices or interfaces are at that location. Click on 001 (4) and you can see a list showing those device or interface names. The first device is always the CPE router. It is followed by an ATM interface, and 2 virtual access lines. If you right-click on the CPE device as we did here, you will not get any results for WAN Interface or Frame Relay reports. That’s because this is not a WAN Interface or Frame Relay device. To get meaningful results, you must first select a device that matches the report type for the metrics you want.

34. InfoVista CPE tab The CPE tab displays Router Performance metrics, WAN Interface metrics, and Frame Relay metrics. The CPE tab is the starting place for diagnosing any performance-related issues with CPE devices.

35. InfoVista CPE – Router Performance Report The CPE tab page contains several types of information. By breaking it into sections, we can examine each part of it separately. We’ve already looked at the tabs, the time navigator, the device navigator and each of their functions. Let’s remove all of those so we can get a better look at each of the CPE panes containing devices and metrics. • We’ll explore each pane individually- • Router Performance Report • WAN Interface Report • Frame Relay Report Click on any device name for more detailed information on that CPE. We’ll select GTAGDOC2005. The report is generated, and it shows hourly CPU and memory usage over a 1 day period. Other quick reports are available by clicking the arrow next to the CPU usage numbers. We’ll look more into understanding reports later, but this covers the very basics of generating a performance report on a CPE device. The default Router Performance Report pane displays the top 10 CPE devices measured by CPU usage. Memory usage is also displayed. These two metrics provide a snapshot of the daily CPE traffic load.

36. InfoVista CPE tab – WAN Interface Report • Router Performance Report • WAN Interface Report • Frame Relay Report The WAN Interface Report lists metrics on each interface, including: Availability: The percentage of time that the WAN interface is available In Avg: The average amount of incoming traffic in Megabytes per second In Peak: The highest amount of incoming traffic in Megabytes per second Out Avg: The average amount of outgoing traffic in Megabytes per second Out Peak: The highest amount of outgoing traffic in Megabytes per second In Avg Load: Average amount of incoming traffic on the selected circuit Out Avg Load: Average amount of outgoing traffic on the selected circuit Out Peak Load: Peak amount of outgoing traffic on the selected circuit In Peak Load: Peak amount of incoming traffic on the selected circuit IN PEAK is the highest amount of traffic on this interface. This is important if you’re experiencing congestion or bogging down issues!

37. InfoVista CPE tab – WAN Interface Report Click on the arrow near the Availability percentage to generate a report Each graph displays time across the bottom and load percentage on the left side. Or select any device to view a report displaying time and load percentage.

38. Frame Relay pane • Router Performance Report • WAN Interface Report • Frame Relay Report 47% 34% The Frame Relay pane indicates a daily average for inbound and outbound traffic, measured in percent according to capacity. 21% 20% 12% Click on any inbound or outbound percentage metric to generate a report. GTAGDOC8011 shows in inbound average of 22.132 percent for the day. The graph report indicates actual traffic from 12% to 47% for that day.

39. CPE Top & Bottom tab CPE The CPE Top & Bottom tab displays a quick reference listing of the top and bottom 10 CPE devices rated by CPU and memory usage. The CPU devices are rated on a daily average of CPU usage measured in percent. Daily CPU usage in percentage Daily Memory usage in percentage Top 10 devices by CPU usage Bottom 10 devices by CPU usage

40. CPE Top & Bottom tab The reporting period may be modified as desired using the time navigator along the top. CPU usage in percent This is an hourly graph, but it can be modified to display daily or longer time periods. Metrics are displayed for that device showing CPU and memory utilization. t i m e i n h o u r s Similar to the CPE tab, select any device to see more details on usage metrics for that device.

41. CPE Top & Bottom Access tab The Access tab contains metrics displaying latency, jitter, and packet loss for access lines and routes. • CoS categories include: • BE – Best Effort • PB – Priority Business • IA – Interactive • RT – Real Time The BE annotation after the site name indicates the Class of Service (CoS) level. BE is Best Effort.

42. Access tab Navigating the Access Reports page is very similar to the CPE Reports page, with the Time Navigator and the Device Navigator. • The right pane, Availability Report, displays the same routes that are shown below the Device Navigator, along with additional information: • Latency • Packet Loss • Jitter

43. Access tab • Detailed report data is also accessible by right-clicking any route within the Device Navigator. • This will open a menu with the following selections: • Show Instant Reports • Performance Metrics • Synchronize All will open this window, and clicking any selection will generate an instant report displaying jitter metrics for the selected time period as listed. populates the right pane with metrics for the selected device will update the data to the newest available for the time period chosen.

44. Access tab – Show Instant Reports Double-click on any report line to generate a graphic display The performance graph will be generated for the selected device, similar to the instant reports in the CPE tab.

45. Access Access Top & Bottom The Access Top & Bottom tab contains a quick reference listing of the top 10 and bottom 10 access routes rated by latency.

46. Access Top & Bottom tab Similar to the ACCESS tab, you may select either the device name or the metric value to generate a graphic report. This list displays the bottom 10 Access lines for usage (average) in the time shown. This lists Packet Delivery (average) for the same time period shown above. This lists the Latency (average) for the same time period shown above. This list displays Availability during the same time period. This list displays the top 10 Access lines for usage during the time period shown. This lists the Latency (average) for the same time period shown above. This list displays Packet Delivery during the same time period. This list displays Availability during the same time period.

47. InfoVista Metrics & Reports InfoVista reports may have differing appearances based upon several factors. Regardless of how they appear, reports will all have a few things in common. • At the top of the report, there is a modified version of the Time Navigator. • Once a report has been generated, you can modify the time period or report date forward or back to view adjacent dates and view performance metrics without closing the window and then starting over. • We have the same navigation arrows, calendar, and GO button as the Time Navigator. There is also an additional field that defines the time span of the report- whether it covers a day, week, or longer. • The timeline on this report is set to begin at noon and display 1 day of data. • You can set the start date and time, set the duration, and then view a report on any time period for which data is available. If we examine the router performance report to the right, CPU usage is being measured. The highest value on the scale displayed along the left side is 36. That indicates 36%, although the highest CPU spike actually reached only 32%. InfoVista resizes the scale to fit the data. If the CPU were to suddenly spike again and rise to 65%, the next time this data is polled, InfoVista would resize the graph so the new spike would fit. The left side would be renumbered so it would reach 70% at the top, and all the data would still fit within the graph. The graph will auto-adjust to fit the lowest and highest values. InfoVista reports span a period of time that is indicated across the bottom of the graph. Whether it measures hours, days, or weeks, older time is on the left side, with more recent events on the right side. 32% 65%

48. Interpreting Results • As the time period and the duration are modified, any new reports that are generated will display performance metrics for that new timeline. • Here we can see reports with the timelines set to • Hourly, covering a 1 day period • Daily, covering a 1 week period • Weekly, covering a 2 month period • This allows report time periods to be customized, showing hour-by-hour performance peaks, or weeks and even months at a time to see larger trends that would not be obvious while viewing hourly or daily metrics.

49. Use This Report 10 • The controls across the top of the graphic report include report date… • in addition to: • Formats • Explain • Latest Allows the data to be displayed (and then saved) in several formats. Generates a small window to explain the data and values displayed in each graph. Refreshes the data to display the newest report data available.

50. Save This Report • Reports are a visual medium, and must be versatile to in order to be useful, whether to be emailed, printed, archived, or posted online. • Click the Formats button at the top left corner of a report and select the desired file type. The report will be generated in a new window. • The Formats button allows the report to be saved in several formats including: • PDF file • Plain text file • Excel spreadsheet • XML code for a web page • Save the new file as you would any other file, and then email, print, post to a webpage, or archive on a network drive.