Orphaned Servers and Broken Processes

1. Orphaned Servers and Broken Processes 2007 Security Professionals Conference April 12, 2007

2. Introductions Background Do You Have Orphaned Servers? Vigilance and Awareness Lessons Applied @ CSUN Reflections @ CSUN Lessons Relearned @ CSUN Summary Questions / Discussion Agenda

3. Moran Technology Consulting (MTC) conducted an Incident Audit for a large research university Audit focused on two (2) systems that were thought to have been compromised: A server housed in central IT data center containing alumni donor information, including > 100,000 social security numbers A server outside of central IT data center containing campus health center data, including > 50,000 patient records Background

4. Health Center Server Incident Detection Vigilant scanning of system following prior incident Systems Management No monitoring or support from central IT(in spite of HIPPA related risks) Comprises detected: A Rootkit (Hacker Defender Trojan) W32/pate.b virus Server had participated in attack onanother university server Alumni Donor Server Incident Detection External complaints of brute force attack emanating from server Systems Management System was “decommissioned” and powered-off and removed from the network System was removed and returned to the network several times (as an orphaned server) Following compromises detected: A Rootkit 4 GB of unauthorized music files The Hideout virus Background The Compromises occurred 4 months and possibly 12 months prior to detection, respectively.

5. Why did these compromises occur? Insufficient Operational practices and procedures within central IT Lack of formalized standards for secureserver configuration No clear roles and lines of responsibility resulted in lack of accountability for systems Failure of change management procedures to ensure proper system tracking and decommissioning While 75% of campus servers reside outside of central IT data center little or no effort had gone into securely supporting or monitoring these systems A “silo” culture existed within central IT Staff reductions left central IT with too few resources Previous attempts to establish campus-wide polices failed due to lack of executive support Background

6. Do you know who owns and is managing EVERY server: In the central data center? Across the campus? How much do you know about the servers on your campus? What types of data do they house or connect to? Who is accountable for these systems and data? Who owns and maintains these systems Do you have servers on your network that are believed to have been powered OFF but are really powered ON? What are your current decommissioning policies and procedures Have these policies and procedures been formalized? Do they apply and are they in practice campus-wide? Do You Have Orphaned Servers?

7. What formal IT Policies and procedures guide security management roles and responsibilities? Do the they cover EVERY server on the campus, including both Central Data Center servers AND department servers? Executive Ownership and Participation is REQUIRED for Security Policies to be effective Vigilance and Awareness

8. Decided to use the incident at another institution to repose the salient questions and to review our security stance! Have we identified EVERY server in the central data center? Have we identified EVERY server exposed to the Internet? Do we have sufficient information to protect these assets? What additional policies, procedures, and processes are needed? Developed an approach to review our environment. Reviewed firewall rules for all openings Reviewed DNS for named servers Reviewed ARP tables for IP assignments Performed network scans to identified live servers Established primary and secondary system owners for each server Formulated a plan to address deficiencies found. Lessons Applied @ CSUN

9. We thought we were in a very secure position … but we were wrong! We had spent a lot of time protecting the systems and networks … everything you would expect of a professionally run IT organization to do. We were very surprised by the number of orphans we found on our network. What introduced our orphaned servers? Well meaning staff (“I’ll get to it.”) Shifts of responsibilities and new staff. (“I thought they/he owned it”) Overloaded staff and lack of resources. (refer back to #1) This is a President’s worst nightmare! The overall cost of an incident includes the damage done to the reputation and credibility of the institution. The question is not whether but when… Hackers want two things: Personal information they can sell and Access to powerful computing resources Reflections @ CSUN

10. Always presume that you are less secure than you are! Use external and internal security events to Increase security awareness at both the staff and executive levels Perform ad-hoc reviews of your environment Establish periodic reviews of your environment, which includes: Scans of the network (don’t assume your change management process are fail-proof) Review the ownership of servers and identify the administrators Review the associated Firewall and DNS rules for all servers Structure your IT organization/environment with Clearly defined role and responsibilities for every unit and individual Predefined separation of duties, with effective working teams. Established strong change-management procedures that include removing of services from production. (The cleanup process is the killer.) The “one throat to choke” principal does not apply! Everyone must participate to ensure success. Lessons Relearned @ CSUN

11. Security Incidents happen to everyone. Incidents that occur at other institutions are excellent opportunities to improve your own security awareness and practices Preparedness and Vigilance are key to minimizing cost Presume that you are still vulnerable (You are!) Perform on-going reviews of your environment Develop a Incident Response Plan Institute policies and procedures to identify, manage and properly decommission every server on campus Orphan servers provide a window for hackers to exploit your entire IT environment Summary

12. Questions / Discussion