Identifying and Managing Essential Cyber Assets: Closing the Loop on the BCS

1. Identifying and Managing Essential Cyber Assets: Closing the Loop on the BCS Dr. Joseph B. Baugh Senior Compliance Auditor – Cyber Security WECC Reliability and Security Workshop San Diego CA – October 23, 2018 Western Electricity Coordinating Council

2. The Parable of the Essential Cyber Asset Western Electricity Coordinating Council

3. External Users Western Electricity Coordinating Council

4. Help compliance personnel become more familiar with the processes to identify all essential Cyber Assets associated with each of the BES Cyber Systems [BCS] identified under CIP-002-5.1a to ensure appropriate access controls and ensure data flows into and out of the BCS are adequately managed and protected Reliability & Security Objective Western Electricity Coordinating Council

5. During recent audits, the CIP team has found several instances of Potential Noncompliance relative to legacy configuration files on EACMS, poor management of Intermediate Systems, failure to properly identify and protect PACS elements, and other issues associated with managing essential Cyber Assets CIP Team Observations Western Electricity Coordinating Council

6. Develop an ability to discuss essential Cyber Assets with entity Subject Matter Experts [SMEs] and develop a stronger CIP compliance program that contributes to better reliability and security in the Western Interconnection Key Takeaway Western Electricity Coordinating Council

7. If half the attendees in this room worked closely with their SMEs to correctly identify and manage all essential Cyber Assets, the reliability and security of the BES would be improved This may also result in an ancillary impact of better compliance performance at audit and fewer Self-Reports What May Change Western Electricity Coordinating Council

8. Some may be thinking to themselves, “If these essential Cyber Assets are not specifically spelled out in the CIP Standards, why should I care about them?” • Most essential Cyber Assets are included in the CIP Standards, either directly or indirectly • All help “close the loop on BES Cyber Systems [BCS]” • The CIP Team has noted issues in past audits • Don’t be this entity, “Oops, I should have closed the loop on my BCS!” Why Should You Care? Western Electricity Coordinating Council

9. Examine essential Cyber Asset types in detail • What is it? • Where are they generally located? • Associated Standards • How are they configured? • What are common examples? • What are common problems and pitfalls that may lead to noncompliance? • Group Exercise • Review • Questions • Summary Agenda Western Electricity Coordinating Council

10. An essential Cyber Asset is NOT a defined term, but relates to several common Cyber Assets used to protect and/or access a High or Medium BCS, some of which are defined terms What is an Essential Cyber Asset? EACMS include EAP, IS, IDS/IPS. We can also add Gateways, Modems, TCA, and RM to our list of essential Cyber Assets Western Electricity Coordinating Council

11. Physical Access Control System [PACS] Electronic Access Control and Monitoring System [EACMS] Intermediate Systems [IS] Intrusion Detection Systems / Intrusion Protection Systems [IDS/IPS] Serial Gateways Dial-up Gateways & Modems Transient Cyber Assets [TCA] Removable Media [RM] Types of Essential Cyber Assets Western Electricity Coordinating Council

12. Some graphics used in conjunction with the following essential cyber asset type slides are intended for illustrative purposes only and represent common cyber asset types observed by the CIP team during site visits to remote locations However, the use of openly accessible graphics or other materials under the Fair Use Act should not be construed as an endorsement, support, or promotion by the WECC CIP Team for any specific vendor or specific essential Cyber Asset product Disclaimer Western Electricity Coordinating Council

13. Physical Access Control System [PACS] Cyber Assets that control, alert, or log access to the Physical Security Perimeter(s), exclusive of locally mounted hardware or devices at the Physical Security Perimeter such as motion sensors, electronic lock control mechanisms, and badge readers. (NERC, Glossary, p. 22) PACS – What Is It? Local Devices (Readers, Cameras, Door Sensors) PSP-Like Security Perimeter PACS Server Direct Connection PACS Panel PACS Workstation Typical PACS configuration

14. Where are PACS generally located? • Physically within the confines of a defined Physical Security Perimeter (PSP) • Physically outside the confines of a defined PSP, but secured • PACS must be provided certain physical security protections • CIP-006-6 Parts R1.1, R1.6, and R1.7 • Guidance & Technical Basis provides some relief for these for PACS devices residing within a defined PSP PACS – Where is It? Western Electricity Coordinating Council

15. Which Standards are associated with PACS? • Primarily CIP-006-6 • All Requirements where PACS are associated with applicable BCS, including: • CIP-004-6 R2, R3, R4, R5 (Parts 5.1, 5.2, 5.3) • CIP-006-6 R1 (Parts 1.1, 1.6, 1.7), R3 • CIP-007-6 R1 Part 1.1, R2, R3, R4 (Parts 4.1, 4.2, 4.3), R5 • CIP-009-6 R1, R2 (Parts 2.1, 2.2), R3 • CIP-010-2 R1 (Parts 1.1 – 1.4), R3 (Parts 3.1, 3.4) • CIP-011-2 R1, R2 PACS – Associated Standards Western Electricity Coordinating Council

16. PACS should be configured to meet specific physical security issues for each entity or at each PSP, but in general: • Typical system components include application host server(s), intelligent access control panel(s) and personal computers (aka: workstations) used to perform systems and access administration, alarm monitoring and/or badge provisioning • Workstations may utilize a client or web based interface • Workstation communication paths may include an intermediate jump host PACS – Typical Configuration Western Electricity Coordinating Council

17. The CIP-006 team has noted common failures to address these issues: • Identify PACS client workstations as PACS assets • Ensure all PACS Cyber Assets are identified, classified, and protected • Ensure PACS located outside PSPs are secured and monitored with alerts within 15 minutes • Properly annotate PSP diagrams • Document PACS alarm assessment and response procedures • Document PACS responses for each incident of potential unauthorized physical access • Provide a hard key management program PACS – Common Problems & Pitfalls Western Electricity Coordinating Council

18. Electronic Access Control and Monitoring System Cyber Assets that perform electronic access control or electronic access monitoring of the Electronic Security Perimeter(s) or BES Cyber Systems. This includes Intermediate Systems. (NERC, Glossary, p. 12) Essential Cyber Assets include EACMS, Electronic Access Points [EAP], Intrusion Detection/Prevention Systems [IDS/IPS] and Intermediate Systems [IS] EACMS – What Is It? Western Electricity Coordinating Council

19. EACMS – Where is It? • EACMS generally establish the Electronic Security Perimeter [ESP] for BCS • EAP are located on the ESP as required, but the EACMS may have additional interfaces • IDS are typically located on the ESP network, but • Passive IDS may also be located outside the ESP, or • IPS may be inline with traffic • IS are typically located on a DMZ segment External Users Western Electricity Coordinating Council

20. Which Standards are associated with EACMS (includes EAP, IDS/IPS, and IS)? • CIP-004-6 R2, R3, R4, R5 • CIP-005-5 R1, R2 [Specifically EAP, Intermediate System, and Dial-up Connectivity] • CIP-006-6 R1 (Parts 1.2, 1.3, 1.4, 1.5, 1.8, 1.9), R2 • CIP-007-6 R1 Part 1.1, R2, R3, R4, R5 • CIP-009-6 R1, R2 (Parts 2.1, 2.2), R3 • CIP-010-2 R1 (Parts 1.1 – 1.4), R2 (high impact only), R3 (Parts 3.1, 3.3, 3.4) • CIP-011-2 R1, R2 • CIP-008-6 (Proposed Standard in Comment Phase) EACMS – Associated Standards Western Electricity Coordinating Council

21. EAP are typically configured with explicit access control lists [ACL] Passive IDS typical configuration seen on audit IPS are not as common IS are typically Terminal Servers or similar Cyber Assets In general, the CIP team will examine specific configurations, as required by the Standards EACMS – Typical Configuration Western Electricity Coordinating Council

22. EAP Firewalls AAA Servers Token Servers IDS/IPS devices Domain Controllers SIEM Devices IS Terminal Servers EACMS – Common Examples Western Electricity Coordinating Council

23. Entities may fail to: • Ensure IRA protocols are sourced only from an IS • Ensure system-to-system traffic is approved • Ensure spanning is configured in HA architectures • Disable access control rules for system maintenance • Refine access permissions to EACMS devices to specific IP addresses • Document reasons for granting access on EACMS • Provide system hardening beyond required controls • Address detection of malicious communications within encrypted communications through EAPs EACMS – Common Problems & Pitfalls Western Electricity Coordinating Council

24. A Cyber Asset or collection of Cyber Assets performing access control to restrict Interactive Remote Access to only authorized users. The Intermediate System must not be located inside the Electronic Security Perimeter. (NERC, Glossary, p. 17) • To fully understand IS, we also need to understand Interactive Remote Access [IRA] IS – What Is It? Western Electricity Coordinating Council

25. User-initiated access by a person employing a remote access client or other remote access technology using a routable protocol. Remote access originates from a Cyber Asset that is not an Intermediate System and not located within any of the Responsible Entity’s Electronic Security Perimeter(s) or at a defined Electronic Access Point (EAP). Remote access may be initiated from: 1) Cyber Assets used or owned by the Responsible Entity, 2) Cyber Assets used or owned by employees, and 3) Cyber Assets used or owned by vendors, contractors, or consultants. Interactive remote access does not include system-to-system process communications. (NERC, Glossary, p. 16) IRA – What Is It? Western Electricity Coordinating Council

26. Per the Glossary definition, the IS Cyber Asset(s) must be located outside the ESP Encrypted IRA data sessions may only be established by an authorized user IS – Where is It? Untrusted External Network Typically Unencrypted May Be Encrypted Must be Encrypted Western Electricity Coordinating Council

27. Graphic (TDI Technologies, n.d., EACMS Whitepaper, used for educational purposes only) retrieved from https://www.tditechnologies.com/wp-content/uploads/2017/03/TDi-CIP-005-CIP-007-CIP-010_2-1.pdf, p. 6 IS – Complex Configuration Western Electricity Coordinating Council

28. Installing applications with capability to directly operate one or more elements of the BCS on the IRA client and/or on the IS Failing to ensure all IS Cyber Assets are afforded applicable EACMS protections Allowing back channel communications into the ESP that bypass the IS and/or EACMS IS – Common Problems & Pitfalls Western Electricity Coordinating Council

29. An intrusion detection system (IDS) is a device or software application that monitors a network or systems for malicious activity or policy violations. Any malicious activity or violation is typically reported either to an administrator or collected centrally using a security information and event management (SIEM) system. A SIEM system combines outputs from multiple sources, and uses alarm filtering techniques to distinguish malicious activity from false alarms. Some IDS products have the ability to respond to detected intrusions. Systems with response capabilities are typically referred to as an Intrusion Prevention System [IPS]. (Wikipedia) IDS/IPS – What Is It? Western Electricity Coordinating Council

30. Where are they generally located? • Network intrusion detection systems (NIDS) • May be located on either or both sides of the EAP/EACMS to monitor inbound and outbound traffic • Host intrusion detection systems (HIDS) • Located on mission critical servers to monitor traffic to/from that specific server • Intrusion Prevention System [IPS] • IPS often sits directly behind the firewall and provides a complementary layer of analysis that negatively selects for dangerous content • Unlike IDS, IPS is placed inline (in the direct communication path between source and destination), actively analyzing and taking automated actions on all traffic flows that enter the network IDS/IPS – Where Is It? Western Electricity Coordinating Council

31. IDS/IPS – Typical Configuration (NIDS/HIDS) Untrusted Networks Layer 3 Switch Western Electricity Coordinating Council

32. Require experienced technicians to administer IDS/IPS, recognize and act on threats Entities may fail to regularly review IDS/IPS logs to analyze cyber intrusions High false positive rates can contribute to noise that inures system operators to actual intrusions Attack signatures must be updated regularly IDS/IPS – Common Problems & Pitfalls Western Electricity Coordinating Council

33. Serial Gateway • In the context of the Grid, typically an Ethernet-serial communication processor that collects and formats data from field processing units, such as relays, over serial connections and communicates with remote EMS/SCADA systems over a routable protocol for transmission between sites. Serial Gateway - What Is It? Western Electricity Coordinating Council

34. Where are they generally located? • Typically in substations to aggregate data from serially connected relays and other IEDs for transmission to EMS/SCADA systems • May also be located in generation stations as part of Distributed Control Systems [DCS] to collect data from multiple points Serial Gateway – Where is It? Western Electricity Coordinating Council

35. Generally has capability to allow personnel to configure relays and other serial devices remotely May also have dial-up capability, depending on configuration and location Modem may or may not be in ESP, depending on connection to gateway Serial Gateway - Typical Configuration Event Recorder Serial Gateway Western Electricity Coordinating Council

36. Failure to classify the gateway as a BCA when accessed through an EAP/EACMS Failure to classify routable gateways and serial relays as a BCS Serial Gateways Common Problems & Pitfalls Western Electricity Coordinating Council

37. Many remote substations are accessible only through a standard analog telephone connection. For those situations, the Dial-up Gateway enables sharing a single substation telephone line between devices, such as modems connected to fault locating relays, meters, etc. Dial-up Gateways & Modems What Is a Dial-up Gateway? Western Electricity Coordinating Council

38. A modem (modulator–demodulator) is a network hardware device that modulates one or more carrier wave signals to encode digital information for transmission and demodulates signals to decode the transmitted information. Modems can be used with any means of transmitting analog signals, from light-emitting diodes to radio. A common type of modem is one that turns the digital data of a computer into modulated electrical signal for transmission over telephone lines and demodulated by another modem at the receiver side to recover the digital data. (Wikipedia) Dial-up Gateways & Modems What Is a Modem? Western Electricity Coordinating Council

39. Typically used with POTS service • Generally seen at remote substations (Low impact BES Assets) that do not have higher speed access methods • Dial-up gateways can serve as a required authentication point as prescribed by • CIP-005-5 (Part R1.4, p. 7) • CIP-003-7 (Attachment 1, Section 3.2, p. 22) Dial-up Gateways & Modems Where are They? Western Electricity Coordinating Council

40. A BCS with Dial-Up Connectivity is reachable via an auto-answer modem with an unchanged default password A BCS has a wireless card on a public carrier with a public IP address A BCA has dual-homed interface cards, one of which may be an accessible modem, and IP forwarding is enabled by default Dial-up connectivity is directly to a modem or modem bank with no authentication gateway Dial-up Gateways & Modems Common Problems & Pitfalls Western Electricity Coordinating Council

41. Transient Cyber Asset • A Cyber Asset that (i) is capable of transmitting or transferring executable code, (ii) is not included in a BES Cyber System, (iii) is not a Protected Cyber Asset (PCA), and (iv) is directly connected (e.g., using Ethernet, serial, Universal Serial Bus, or wireless, including near field or Bluetooth communication) for 30 consecutive calendar days or less to a BES Cyber Asset, a network within an ESP, or a PCA. (NERC, Glossary, p. 32) TCA – What Is It? Western Electricity Coordinating Council

42. Examples of Transient Cyber Assets include, but are not limited to, Cyber Assets used for data transfer, vulnerability assessment, maintenance, or troubleshooting purposes • Backup drives • Relay technician’s laptop • Network packet sniffer • Vulnerability scanning tools and software TCA – Common Examples Western Electricity Coordinating Council

43. Removable Media [RM] • Storage media that (i) are not Cyber Assets, (ii) are capable of transferring executable code, (iii) can be used to store, copy, move, or access data, and (iv) are directly connected for 30 consecutive calendar days or less to a BES Cyber Asset, a network within an ESP, or a Protected Cyber Asset. (NERC, Glossary, p. 27) RM – What Is It? Western Electricity Coordinating Council

44. Where are TCA/RM generally located? • Locations can vary widely across the entity’s system • TCAs can be assigned to individuals or groups, for use at multiple BES Assets • Relay technician laptops • Communication department laptop • TCA/RM may reside at applicable BES assets and connected as needed to specific BCA • Substation maintenance workstation/laptop • Substation A’s thumb drive • RM devices are ubiquitous and travel easily in pockets, briefcases, laptop bags, purses, etc. TCA/RM – Where are They? Western Electricity Coordinating Council

45. CIP-010-2 R4, Attachment 1 • High and Medium impact BCS • Section 1: TCA managed by the entity • Section 2: TCA managed by third-parties • Section 3: RM • CIP-003-7 R2, Attachment 1 • Low impact BCS • Section 5: Malicious Code Mitigation TCA/RM – Associated Standards Western Electricity Coordinating Council

46. Examples include, but are not limited to, • USB flash drives • Floppy disks • Compact disks • External hard drives • Other flash memory cards/drives that contain nonvolatile memory RM – Common Examples Western Electricity Coordinating Council

47. Ensure TCAs are managed in defined methods, e.g. on-demand versus on-going • Ensure devices are authorized per plan • Recommend developing tracking methodology to ensure devices are not connected to applicable Cyber Assets for more than 30 consecutive calendar days • Retain evidence the entity is following plans • e.g., if RM is scanned prior to being connected to BCA/PCA, capture artifact of scan(s) TCA/RM – Common Problems & Pitfalls Western Electricity Coordinating Council

48. Automate operations where feasible • Implement an asset life-cycle inventory system, e.g. Configuration Management Database [CMDB] • Implement consistent naming and labeling conventions for all Cyber Assets • Ensure devices are evaluated for applicable Technical Feasibility Exceptions [TFE] • Consider utilizing layer 3 to segregate BES Cyber Systems from other non-ESP networks General Best Practices Western Electricity Coordinating Council

49. Essential Cyber Assets • Are important components of the BCS • Represent risk to the reliability of the BES • Proper security measures will help ensure entities close the loop on the BCS Essential Cyber Asset Review Western Electricity Coordinating Council

50. In your table groups, please address these points (10 minutes): • How do essential Cyber Assets impact your entity? • Identify essential Cyber Assets in your environment? • How do you protect essential Cyber Assets? • What are common problems and/or pitfalls? • Prepare a debriefing statement • Select a spokesperson to share your statement Table Group Discussion Western Electricity Coordinating Council