Network Vulnerability Assessments Lessons Learned

1. Network Vulnerability AssessmentsLessons Learned Chris Goggans chris@patchadvisor.com

2. What are Vulnerability Assessments? • Internal and external attacks • Validation of existing security mechanisms • Detailed analysis of all networked devices and services • Audits for policy compliance and deficiencies in existing policies • Prioritized recommendations for improving security posture

3. Vulnerability Assessments: WHY? • Only realistic way to determine vulnerabilities • Get a baseline of vulnerability state • Prioritize remedial actions • Correct serious problems quickly • Assure that policies address real vulnerabilities • Industry best practice

4. Vulnerability Assessments: HOW? • External Assessment • Internet • Modems • Wireless • Partner Connectivity • In-briefing • Internal Assessment • Out-briefing • Report preparation and delivery • Executive briefing

5. Government Contractor • Unpassworded TELNET access into print server • SNMP Read/Write community string exposed in printer configuration menu • Community string also used on devices such as routers, switches, etc. • “Level 7” hashes in Cisco config files exposed the password “mbhafnitsoscar” • This password also used by Domain Administrator on Windows PDC • Windows Domain also tied to NetWare eDirectory, sharing account names and passwords • In total, compromise of nearly 15,000 accounts and 99.99% of all systems and networkdevices…all from one insecure printer

6. Government ContractorLessons Learned: • Even the most insignificant network device can provide information that uncovers a major attack path • Always examine everything connected to your networks! • Always utilize the greatest password protection possible • MD5 vs Level7 • In situations where accounts and passwords are shared across platforms, a single compromise of the weakest platform can lead to massive compromise • Rainbow Tables & NTLMv1

7. Civilian Government Agency • Development WWW server running with default cold fusion scripts that allow remote viewing of web source • Attack Path 1 • ODBC setup in web page source code exposed z/OS RACF account and password used for DB2 queries • Account found to have “system programmer” access on IBM Mainframe • Attack Path 2 • MS-SQL “sa” account and password in web source • SQL “XP_CMDSHELL” stored procedure gave remote “SYSTEM” access to Windows OS • Local SAM file exposed Domain Administrator account • SAM file on PDC had roughly 50,000 accounts • Certain users used same password on Windows as they did on very large High Performance Computing cluster

8. Civilian Government AgencyLessons Learned: • Passwords embedded in web applications are never a good thing • Web Application Vulnerability Assessments have become as important as Network Vulnerability Assessments • Database security is also critical and often left unchecked

9. Another Government Agency • Large network of Solaris and Windows systems • All machines and applications patched • Many important UNIX services are TCP-wrapped • NFS, NIS, etc. • Customer had recently deployed new KVM switches in their racks • In addition to 3rd party software, KVM Switch also had HTTPS based management interface with a default “Admin” account with no password • HTTPS-based access also provided JAVA-based remote console program • Open consoles found on Windows system (as Administrator) and Solaris system (as root) • NIS passwd-byname tables and Windows SAM and locally cached account hashes pulled • All systems compromised

10. Another Government AgencyLessons Learned: • Every host on a network must undergo some level of security hardening before being allowed to connect to the production network • Every console should be forced to either screen-lock or auto-logout when not in use

11. Managed Service Provider • Customer had limited Internet exposure, primarily HTTP traffic allowed to “Hosting” LAN (primarily only TCP 80 & 443) • Web server compromised with Apache “Chunked-code vulnerability” • Server had 3 interfaces (2 for normal access, 3rd interface leading to NOC management-LAN for “out of band” SNMP) • System on NOC network compromised with common Windows vulnerability • NOC network had visibility into entire corporate network, as well as other hosted customers

12. Managed Service ProviderLessons Learned: • It only takes one vulnerable service to give attackers a strong foothold into your network • Management networks or VLANs are often excellent points to bridge between networks without direct connectivity • Systems hosted by 3rd parties are often compromised by attacks originating from less secure customers at the same hosting facility

13. Telecommunications • Large US telephone company • Dial-ups found with unpassworded pcAnywhere • pcAnywhere system used primarily for access into security camera monitoring of unmanned facilities • Full access to internal network, including switching systems, billing, etc.

14. TelecommunicationsLessons Learned: • Modems can still be a major external threat! • Critical systems should be firewalled against general network access

15. Emergency Response • Organization responsible for state-wide emergency medical services • Internet connectivity shared with major university • Organization tied to other state-run networks through dedicated lines • Firewall rules allowed certain hosts on University network & State Government networks into EMS network using insecure protocols (MS-SQL, SMB) • Common exploits led to massive compromise

16. Emergency ResponseLessons Learned: • All inbound partner connections should be examined as part of a vulnerability assessment • Firewall rules should likewise be examined to uncover any potential weak points for permitted communications • Your network is ultimately only as secure as the weakest host connected to you

17. Law Enforcement • Compromised Windows Workstation through un-patched IIS Web server – obtained local SAM file with domain accounts • Compromised Windows PDC by escalating privileges with access gained from previous machine • Compromised Investigator’s workstation with Domain Admin rights • Workstation had VNC remote control software – password retrieved from Windows registry • Logged in using remote control GUI • Icon on desktop for MILES/NCIC • Just a keystroke capture program away from access to the FBI

18. Law EnforcementLessons Learned: • Users should not be allowed to install random applications on their workstations • Especially those that facilitate remote control! • Applications that utilize proprietary authentication are usually easily broken • Any system with multiple Network Connections can be used as a gateway to bridge secure networks to insecure networks • The same holds true for VPNs, PPP connections, Wireless LAN access, etc. • Even one of the most “secure” systems in the US can be compromised if accessed in an insecure fashion

19. Uncovering Attacks During Assessment • Many assessments will reveal existing evidence of prior attack activity • Some attacks may be more serious than others • Most attack information found on Internet-based hosts are from random hacker groups running scripts • Attacks found on internal machines are usually much more serious

20. Real Incident – Local Government • Internet assessment found that IIS server was vulnerable to attack • Several strange files found in the “SCRIPTS” directory • Files were backdoor CMD shells and host scanning scripts • Web log analysis showed that the host had been compromised by at least two separate groups: one in USA, one in Korea • Host was patched and files were removed

21. Real Incident – Service Provider • Customer had servers hosted at Tier-1 internet provider • Poor password on MSSQL server led to compromise of machine • Customer noticed that the server was losing disk space • Hidden directories had over 30GB of movies and music • Netstat output showed that server was connecting to IRC server • Ethernet Sniffing revealed IRC channel and channel key being used • IRC Channel was being used by German software pirates • We joined the channel and surprised the pirate group. They apologized and told us we could keep copies of the movies.

22. Real Incident - Telephone Company • Systems Administrator making threats about taking down Telephone switches • Multiple root-shell files found on critical UNIX servers throughout the enterprise • Backdoor access to switching systems found through X.29 PADs • Administrator’s contract was terminated

23. Real Incident – Web Services • Systems administrator fired for sexual harassment • Windows machines began experiencing problems • False accounts discovered on Internet accessible machine • Trojan Horse discovered on internal workstations • Real motive was intellectual property theft, and administrator was arrested

24. Real Incident - Banking • Vulnerability assessment conducted against bank network • Trojan Horse discovered on workstation • Workstation used primarily as database for all customer credit-card data • No data available to identify how Trojan Horse was delivered • All credit cards on server had to be re-issued

25. Common Assessment Problems • Customer Perception • Misconfigured Hosts • Bad Programming

26. Problem – Customer Perception • Customer knew that we had gained access to all UNIX systems • Administrator complained that TACACS server no longer worked, and thought our assessment caused the issue • Review of TACACS config file showed that it had been recently modified by the Administrator • We discovered that the Administrator had put in an additional # in file that caused the problem • Administrator was very embarrassed

27. Problem – Misconfigured Hosts • Solaris file system became full and caused kernel panic • Problem occurred when the server was port scanned, starting somewhere above 30000 • /var/log/messages file showed that “Inetd” process was failing and writing hundreds of errors per minute to file, causing the disk usage • Analysis of /etc/inetd.conf file showed that a process (kcms_server) was allowed to spawn by inetd on port 32774 • Examination of files showed that the kcms_server program (and many other unused programs) had been erased by system integrator during original install • Addition of a single # in inetd before the program name corrected the problem

28. Problem – Bad Programming • Port scans of a host indicated multiple unknown applications running on database server • When connecting to these services with netcat or telnet to obtain banner and protocol information, the service crashed • Analysis of the source code indicated that application programmers did not put in any error handling routines for TCP connections • Programmers were able to fix the issue very quickly

29. Final Thoughts • Insecure Web Applications have become one of the biggest targets for attackers. • Modems(authorized and unauthorized) are still not receiving the attention they deserve as potential threats. • Patch & Configuration management are consistently neglected, or only applied to Core OS (not applications). • The concepts of Internal and External access have begun to blur: • Partner connections • Inbound “Phishing” emails and Web Pages downloading malicious code that open up outbound “shell” access • Poor passwords are the number one mechanism to gain host-level access.