1 / 20

Proactive Security Response

Proactive Security Response. About Us. Wayland Morgan – IT Security Analyst Chuck Geigner – Lead IT Security Engineer. About the Office. 2 Analysts 5 Engineers 1 Outreach 1 Policy 1 ISO. In the “news”. Sunday, March 16 th Voice of Russia/ Referendum2014.ru UIUC: Secret Gov HQ?

ohio
Download Presentation

Proactive Security Response

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Proactive Security Response

  2. About Us • Wayland Morgan – IT Security Analyst • Chuck Geigner – Lead IT Security Engineer

  3. About the Office • 2 Analysts • 5 Engineers • 1 Outreach • 1 Policy • 1 ISO

  4. In the “news” • Sunday, March 16th • Voice of Russia/ Referendum2014.ru • UIUC: Secret Gov HQ? • No abuse contact • This… could be a problem • What do the logs say?

  5. Interesting Flow Data…

  6. Monlist • Remote command for querying last 600 servers • Deprecated in newer versions of ntpd • Small queries command large responses • 1 machine on an unfiltered 1 gbps link can create a 450+ gbps attack

  7. In practice Credit: Sans Internet Storm Center

  8. Credit: Sans Internet Storm Center

  9. For remediation • Disable ntpd if you don’t need it • Update ntpd, monlist is deprecated • Disable monitor in /etc/ntp.conf • Restrict ntp server traffic at campus firewall (done) • Being proactive saved the University a lot of trouble. • Teamwork within the community is crucial for this… Working with you, we reduced the total number of vulnerable hosts down to a few.

  10. Heartbleed • Tuesday, April 8 • So… OpenSSL had a problem. • Memory read overrun • Heartbeat function • Enabled attackers to get chunks of memory • Passwords • Private keys • Problem: • POC was circulating before patch • By, some accounts, MONTHS

  11. Containment, Part I: Patching • On a “bad” scale of 1-10, this was a solid 9 • First order of business: CONTAINMENT • “Stop the bleeding” • Emergency Patch order • Find everything affected on campus • Contact system/service owners

  12. The Day We Almost Turned Off Shib • Wednesday, April 9 • All patched, now what? • First order of business: Evaluate authentication services. • AD: OK • Siteminder: OK • MIT Kerberos: • Shibboleth: • After a meeting and a few tense moments, found out why PFS kicks all sorts of butt. • Shib stays on

  13. Containment, Part I cont’d • Thursday, April 10 • Word’s getting out. • But a massmail was still needed • Coordinated with CIR and CIO’s office • Things are getting patched. • Sort of. Still no word from VMWare and a few other vendors.

  14. Containment Part II, SSL Certs • Friday, April 11 • Certs assumed compromised • A few “lucky” backleveled folks… • Get Certmanager ready. • Do high-vis targets 1st: • illinois.edu, www.illinois.edu • Shibboleth • Tell everyone else affected to regen and revoke their old

  15. SSL Certs, Comodo Gets Smoked • Monday, April 14 • We do what everybody on the Internet is doing. • Comodo’s interface folds • Again • After adding more capacity? • Again • Comodo’s datacenter can’t handle the upgrades • Amy just wakes up at 1:00am when nobody’s around

  16. Security Considers Step III • Recap: Heartbleed may leak credentials to attackers (amongst other info). • The POC was in use longer than we have logs • There were affected systems that we didn’t initially consider: • Affected web apps that did direct AD authentication • Apps with local accounts that were “NetID-like”

  17. Containment Part III • Realization: with insufficient logs to audit whose credentials were exposed… • …the “safe” route would be to organize a password reset for each and every NetID. • 80K Kerberos principles • 106 AD accounts • For real.

  18. Logistics, Logistics • Time allotment • 21 days • Finals. Terrible timing • Acceptance rate • Help Desk resource needs after X patrons expire out their AD accounts • Expiring AD accounts • Number of changes to be done per batch • Password Manager is touchy • “everyone,” all at once would be a bad thing • Notifications • One per customer, on its own schedule • Verboten message, private notification to ITPros

  19. Containment Part III • Still happening • Good outcomes:

  20. Questions? • geigner@illinois.edu • waylandm@illinois.edu

More Related