SPC Management Center (CG-SPC)

1. SPC Management Center (CG-SPC) Alessandro De Palma – PM and CG-SPC Manager Giovanni Abbadessa – IT Architect and CG-SIC Manager Lorenzo Cuzzupè – System Analyst SW Manager and Responsible for Measurements and Reporting University La Sapienza Rome, 07 April 2009

2. Agenda • What is CG-SPC ? • Centro di Gestione del Sistema Pubblico di Connetività (CG-SPC): its Mission • SPC “system” • CG-SPC Organization • CG-SPC Dedicated Infrastructure • Main Functions of CG-SPC: • Centralized Security Services • SPC Quality Monitoring • Indirect Measures • Direct Measures • CG-SPC third party functions

3. CG-SPC: its Mission – from CNIPA RFP CG-SPC supports CNIPA with the deployment of the management and control operations on the Sistema Pubblico di Connettività (SPC), as a third party towards the providers of SPC connectivity and security services. CG-SPC has a crucial role for the functioning of the SPC system. On behalf of CNIPA it guarantees the right monitoring process and SLAs validation to end users and the security level of the whole SPC. Its third partyrole towards Q-ISPs or “SPC Monitored Subjects” is an essential guarantee of the correct management and ongoing activities of all the SPC services.

4. CG-SPC: its Mission - from DPCM 1 aprile 2008 DPCM 1 aprile 2008 “Regole tecniche e di sicurezza per il funzionamento del Sistema Pubblico di Connettività“: states at art.1 lettera o) that CG-SPC is: “the component of the shared infrastructures with the aim of the management of QXN … the provision of the security services as of art. 11, comma 3, and the monitoring of qualified SPC connectivity providers …” Art. 11, comma 3 … the whole SPC is configured as a trusted domain, constitued bya federation of trusted domains, with mutual organizational and technological relationships of trust. The security domain of SPC intersects with the connectivity, interoperability and application cooperation domains; including organizational procedures, services and infrastructures both at the Central level (interconnection domain) and at the single PA level (internal domain) …

5. PA SPC System CG-SPC Security Services Measurements, Data Collection and Distribution CNIPA Technical Assistance and Education QISPs = Monitored Subjects NOC/SOC QISP QISP NOC/SOC Probe Probe CG-SICA Probe NIV- SPC QXN Probe Probe QISP QISP NOC/SOC NOC/SOC Other SPC actors as defined by CNIPA Probe QISP: Fastweb/EDS, BT, Wind, Telecom QXN: Società Consortile

6. CG-SPC Organization Management Project Manager e CG – SPC Manager Qualità dei Servizi e Processi di Erogazione Livello Decisionale Comitato di Gestione RTI - CNIPA Responsabile Operativo Sicurezza (CG-SIC) Responsabile Misura, Raccolta e Distribuzione dei Dati Servizi di Raccolta Dati (Repository) Servizi di Distribuzione Dati Security Unit (UdS) PKI Servizi di Misura SLA Manager Esperto Reportistica Esperto Repository Operatore Repository Esperto Interfacce Dati Consulenti Sicurezza Esperti Sicurezza Operatori UdS Esperto PKI Operatore PKI Misure Indirette Esperti Networking Esperti Misure Rete Livello Operativo Delivery Misure Dirette IBM Global Threat Assessment e IBM Virus Response Team Technical Support and IBM Labs Education Consultants Technical Leader Community Human Resources ...... Support Services

7. CG-SPC Dedicated Infrastructure: Service view

8. CG-SPC Dedicated Infrastructure: Main Characteristics Scalabilty High Availability Data Integrity Logical and Physical Security Technical support from IBM labs

9. CG-SPC: The Management Center The Management Center helps CNIPA in the SPC governance:

10. Main Functions of CG-SPC Centralized Security Services SPC Quality Monitoring

11. Centralized Security Services Incident Response Team Abuse Desk PKI Management CNIPA and CERT-SPC-C Support

12. The Security Community of SPC • The SPC is a system where the connectivity is provided by various suppliers • The SPC is a federation of "trusted“ domains that allows the use of all protocols • RUPA (the SPC anchestor) was a system with only one connectivity provider • RUPA was a federation of "untrusted" domains which allowed only the use of certain protocols • The CG-SIC is working with CNIPA and CERT-GOV-C, to create "The Community Security of SPC", involving Q-ISPs and the ULS of each administration. • SPC’s Security Community aims to improve the security level of the SPC • A "system" should use common vocabulary, metrics and procedures (eg 'The process of incident management SPC')

13. The organization of the security for SPC Security in SPC is made of organizations and security services: • SCP-CERT (CERT-SPC C-Central; CERT R-SPC-Regional) • Center for Security Management (CG-SIC at the CG-SPC) • Q_ISP Suppliers with SOC • ULS (Local Security Unit) set up at each Administration using SPC • Set of 10 security services • Three security services required Firewall Management (Network) Firewall Management (Personal) Antivirus & Content Filtering Management Network Intrusion Detection System Management Event & Log Monitoring Management VPN Management Hardening dei sistemi Network Address Translation Management Host Intrusion Detection System Management Vulnerability Assessment

14. CG-SIC organization and interactions with other SPC subjects

15. Activities and tools at CG-SIC

16. Objectives of the process of incident management Handle security incidents in order to: • Contain and minimize the impact • Reduce inefficiencies and damages (direct and indirect) • Restore quickly normal operations Collect and analyze data to form a base of knowledge useful for the detection of: • Events precursors attack • Corrective actions • And for statistical purposes

17. Definition of security incident at SPC Computer Security Incident at SPC “Any event that requires a containment or reaction on the part of those who suffer, in order to avoid or limit the compromise of the correct functioning or the integrity of SPC’s systems and/or of networks and/or confidentiality of information stored in them or in transit or violates the security policies defined by SPC or the laws in force” Attack attemps “an event or series of security events of security occuring in the SPC, which would not cause harm and does not require containment actions or reactions” Source: The management process of security incidents in the Public Connectivity System (issued by CNIPA)

18. Handling of a Security Incident on SPC – Example 1 The incident Response Team (IRT) of CG-SPC coordinates the management response to security incidents that affect the SPC involving the resources to resolve the incident and minimize the damage e.g. coordinated attack on three PA The CG-SPC has a “global” view" of attacs taking place and according to that it may decide to issue warnings

19. Handling of a Security Incident on SPC – Example 2 • The ULS of PA7 detects an attack • It contacts the SOC of its Q-ISP • The notification of an incident, made by SOC Q-ISP4 to CG-SIC, gives the IP address of attacker • The CG-SCI notifies to SOC Q-ISP3 & l'ULS PA3 that an attack comes from a server, probably compromise • ULS PA3 then solve the problem on its server

20. The tools of the CG-SIC for security management: The collection of data relating to attack attempts • Data collected by FW and IDS at PA are sent to the analysis and correlation systems of Q-ISP. • These data are aggregated and sent to the CG-SPC by Q-ISP • CG-SPC stores them in its repository, and uses these data to have an indication on the overall state of the attempts of attack throughout the SPC • These anomalies can be further investigated to find precursors of attack events

21. The classification of the SPC attempted attack The types of attack attempts: • DoS • Intrusion • Acces & data compromission • Malware • Footprinting • Scanning • Enumeration • Probing • Non classified event

22. The tools of the CG-SIC for the management of security incidents: The portal for the notification of security incident CG-SIC has developed a system for reporting security incidents via web that is used by the ULS of Q-ISP and the ULS of Government Agencies Through the portal ULS staff notify that there is a security incident and request to IRT of the CG-SPC support in the management of a this incident This tool is available on the portal of the SPC (https:/www.spc.gov.it) after for authenticated ULS.

23. Reference documents for the management of SPC security incident The following documents, available to the ULS, are essential in order to properly manage an ICT security incident that affects the SPC: • The management process of security incidents in the Public Connectivity System (issued by CNIPA) • Classification of security incident in the Public Connectivity System (SPC) (issued by CNIPA) • Operational Manual for the communication of security incidents of SPC by the ULS (issued by the CG-SPC)

24. SPC Quality Monitoring: Measurement, Data Collection and Distribution Services • Measurement Services: include the collection trough defined interfaces of the data flows regarding performances and quality of SPC services sent by the Monitored Subjects (IndirectMeasures) and the direct and independent collection of sample of measures from Q-ISPs devices (DirectMeasures),so to verify performances on services they deploy. • Data Collection and Elaboration Services (Repository): include collection, analysis, elaboration and storage of all the information on SPC services and operations. • Data Distribution Services: include the creation and distribution of reports, providing CNIPA and PA with the information needed to manage and control services levels (SLA) provided by Q-ISP to PA, and to elaborate guidelines and for validate sizing estimations.

25. Dati Dati Dati Dati Dati Dati Dati Dati Dati Dati Security Security di Inventario di Inventario Amm. Amm. Config Config . . Prest Prest . . Affidab Affidab . . . . Logs Logs Repository Repository CG - SPC Interfaccia con i Soggetti Monitorati SPC Interfaccia con i Soggetti Monitorati SPC Dati Dati Prestazionali Prestazionali Dati di Affidabilit Dati di Affidabilit à à Trouble Trouble Tickets Tickets Dati di Dati di Inventario dei servizi Inventario dei servizi Dati di Configurazione Dati di Configurazione Log di sicurezza Log di sicurezza Dati Amministrativi NOC / SOC NOC / SOC Measurement Services: Indirect Measures – interface with QISP’s NOC/SOC Dati Dati Dati Dati Dati Security Configuration Performance Admin Availability Security di Inventario Inventory Amm. Config . . Prest . . Affidab . . . . Logs dei servizi Repository Repository CG CG - - SPC SPC Interface with SPC QISP Interfaccia con i Soggetti Monitorati SPC SPCQISP Provisioning Dati di provisioning Performance Dati Prestazionali Dati di Affidabilit Availabilty à Trouble Trouble Tickets Tickets Dati di Inventario dei servizi Inventory Dati di Configurazione Configuration Log di sicurezza Security Administrative Dati Amministrativi presently: • Fastweb/EDS • BT • Wind • Telecom NOC / SOC NOC / SOC

26. Repository

27. Indirect Measures: ETL process for a QISP

28. Distribution Services: CG-SPC Portal

29. Indirect Measures: Main Characteristics and Objectives • Definition, setup and management of the interface to collect data flows from all SPC suppliers and for every service they provide to PA. • Collection and organization into a single Repository of all the Administrative and Inventory data for all QISP • Independent SLA calculation with measurement data received from QISPs • Quality control of QISPs Processes: continuos check of the actual process of data gathering and reporting, against the validated static “model” certified by CNIPA test commission • Actual verification of SLAs and penalties reported by QISPs with the independent CG-SPC elaboration • Provision of aggregate information to CNIPA and PA regarding SPC progress, sizing and ongoing quality

30. Measurement Service: Direct Measurements – Main Goals Monitor the continuous and correct operation of the Monitored Subjects’ network performance measurement tools and methods; Highlight any “drift” from formally accepted behaviour caused by changes, configurations, technical problems in the MS network performance systems during the operations; Contribute to the improvement and evolution of measurement tools and methods adopted by the MS; Provide a tool for CNIPA to assess SPC service quality levels; Carry out ad hoc surveys in cases where the perceived and measured quality of service differ, possibly using alternative methods than those adopted routinely by the MS. 30 RTI: IBM SIRTI

31. Measurement Service: Direct Measurements – Quality parameters “Always on” service Round Trip Delay (RTD) One Way Delay (OWD) Jitter Packet Loss Satellite service Round Trip Delay (RTD) Packet Loss PLMN access service Round Trip Delay (RTD) Packet Loss Quality parameters subject to direct measurement for each SPC service: VoIP service Call setup time SCS CPU Load Electronic Mail Percentage of delivered e-mail messages Delivery time QXN Node QXN Node crossing delay QXN Node packet loss rate 31 RTI: IBM SIRTI

32. Measurement Service: Direct Measurements – tools and technologies Tools and Technologies SNMP Poller (InfoVista) Active Probes (technology developed for CG-SPC) • SNMP Poller • Permits all measures which may be based on the reading of standard or vendor Management Information Base (MIB) on the equipments providing the service • E.g. bandwidth load, CPU load, dropped packets on physical/logic interfaces, … 32 RTI: IBM SIRTI

33. Measurement Service: Direct Measurements – Active Probes • Active Probes ↔ M-Box (in SPC terminology) • The CG-SPC specifications allow the use of specific measurement systems (M-Box) to be inserted beside the monitored PAS and the QXN nodes • M-Boxes inject test traffic into the monitored CdT • Measurement equipments located near measurement points or network equipment software mark test packets with departure and arrival “timestamps” 33 RTI: IBM SIRTI

34. Measurement Services: Direct measurements – network architecture overview

35. Measurement Services: Direct measurements – network architecture details • IBM x3650 Servers • Cisco 3750 Multi-Layer SW • Linux RHEL 5 • Bonding configuration • Heartbeat cluster • Measurement VPNs defined as BGP-MPLS extranet • Trunk configuration

36. Measurement Service: Direct Measurements – Active probes • M-Boxes main requirements • What Operators use for their own network performance monitoring: Cisco IP SLA, HW Ping • Indications from IETF IP Performance Metrics (IPPM) working group have been taken into consideration: RFC 2330, RFC 2679, RFC 2680, RFC 3393, RFC 3432 • Overcoming some limitations: the measurement data are stored on the M-Box for long time; SNMP is not used for measurement data collection; no “missing-in-action” packets • Measurements with a couple of M-Box at measurement end points: • UDP packets with customized payload structure • Asymmetric or Symmetric measurement mode • “Mirror Ping” measurements: • Based on ICMP protocol and Echo Request - Echo Reply messages • ICMP packets with customized payload structure • Deployment of centralized measurement architecture 36 RTI: IBM SIRTI

37. Measurement Service: Direct Measurements – Customized payload 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ICMP Header or UDP Header | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Id Campaign | Id CdT | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Base period | Packet _ | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | _ Order Number | Origin Departure Timestamp | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Origin Departure Timestamp | DAT | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Arrival Timestamp | DDT | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Departure Timestamp | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 37 RTI: IBM SIRTI

38. Measurement Service: Direct Measurements – Customized payload 38 RTI: IBM SIRTI

39. Measurement Service: Direct Measurements – Timestamps • How to obtain timestamps: • Difference between “host time” and “wire time”: the timestamp extracted using Gettimeofday() on Linux systems may be very different from the actual packet sending or receiving instant (tens of milliseconds) • This behaviour has been frequently detected on highly loaded and low performance Linux systems, and sometimes on high performance and poorly loaded systems as well • As a consequence, available “Ping like” utilities can not accomplish the mission of accurate network delay measurements • The solution: E1000 NIC driver has been modified (used by Intel PRO 10/100/1000 NICS): • Timestamp fields of inbound and outbound UDP packets with ports ranging in 49152-65535 (IANA declared “dynamic and/or private ports”) are filled by the driver itself • Inbound and outbound ICMP packets containing a special label in customized payload are timestamped by the driver as well 39 RTI: IBM SIRTI

40. Measurement Service: Direct Measurements – Comparison with Indirect measures • Monitoring measurement systems and processes of the SPC Monitored Subjects using Direct Measurements. • Agreement between CG-SPC and QISP on comparability criteria: defined in the document “Relazione sulle misure dirette”. • Comparison process between direct and indirect measures: • Execution of single-shot direct measures and raw data collection; • Calculation of performance parameters based on direct measures. The time aggregation level is the same as the one adopted by the MS: hourly (Fastweb, QXN) or daily (BT, Wind, Telecom Italia); • Direct measures error estimation (The MS does not provide any measurement error) • Definition of a set of values based on direct measurement values and uncertainties: if indirect measure provided by MS belongs to such set then direct and indirect measures are said to be “matching”. 40 RTI: IBM SIRTI

41. Measurement Service: Direct Measurements – Maximizing Comparability with Indirect measures • To perform the verification and validation of SPC Monitored Subjects measurement systems and processes by means of direct measurements, the SPC Management Center adopts alternative but comparable methods than those of MS: • Direct measure test packets and MS test packets use the same path (apart from occasional re-routing); • The test packets are treated according to the QoS rules for the CdT Class of Service: TOS field is filled with the same value as the user traffic one; • Configuration of measurement flows consistent with QISP: test packet size, number, frequency and phase (if possible); • Measuring points are consistent with those of QISPs; • Calculation of performance parameters using the same aggregation algorithms as QISPs; • NTP synchronization with Galileo Ferraris time servers. 41 RTI: IBM SIRTI

42. Measurement Service: Direct Measurements – Measurement errors Analysis of RTD, OWD and Jitter measurement errors for a network performance measurement system. Error components: Offset between host A e B clocks in a particular moment • Offset variation respect to UTC of host x occurred in a particular time period: • Difference between nominal and effective oscillator frequency; • Drift of frequency oscillator due to aging; • Temperature, noise, NTP adjustments on system clock. Resolution error of host x clock 42 RTI: IBM SIRTI

43. Measurement Service: Direct Measurements – OWD Errors Error estimation expression for One Way Delay measures: • Estimated errors; • Upper bounds. New OWD estimator with offset error correction: Error expression for OWD estimation by means of offset correction: 43 RTI: IBM SIRTI

44. Measurement Service: Direct Measurements – RTD and Jitter Errors Error estimation expression for Round Trip Delay measures: • Estimated errors; • Upper bounds; • Very low dependancy on offset error Since Jitter estimator is derived from OWD measures: Error expression for Jitter can be obtained from OWD error expression: • Low dependancy on clock offset; 44 RTI: IBM SIRTI

45. Third Party Functions of the CG-SPC Monitoring of SPC Suppliers Support and e Consulting to CNIPA on SPC Security Overall Quality Monitoring of the SPC “system” Q-ISPs’ Performance and Availabilty comparison Support to CNIPA and “Organismi di attuazione e controllo” on SPC evolution

46. CG-SPC third party functions Monitoring of SPC Suppliers:CG-SPC independently elaborates and reports Performance and Availability measures of the SPC Suppliers and Shared Infrastructures. SPC “system” Quality Monitoring:measuring performances of the various SPC actors over time, CG-SPC can provide information and data to the “Organismi di attuazione e controllo” on the overall quality of the SPC “system”. This function will grow of importance and value with the inclusion of local PA in SPC; Support and e Consulting to CNIPA on SPC Security:CG-SPC, analysing the collection of data and trends, provides CNIPA and CERT-SPC-C with reports and supports and counsels them on the elaboration of Security politics and procedures; Q-ISPs’ Performance and Availabilty comparison:collecting and reporting mesures from all the SPC suppliers, CG-SPC can provide index and trends to compare among providers and support the improvement of their quality processes. Support to CNIPA and “Organismi di attuazione e controllo” on SPC evolution: by all the functions mentioned above, CG-SPC can provide an important support on the SPC system evolution.

47. Summary View – QISPs SLA Analysis

48. Summary View – QISPs’ Trouble Ticket Analysis

49. Quality Monitoring – fault escalation management CG-SPC mangages escalations on faults impacting more than one SPC Q-ISP and coordinates and verifies their resolution.

50. Back-up slides