1 / 12

Security components of the CERN farm nodes

Security components of the CERN farm nodes. Vladim í r Bahyl CERN - IT/FIO Vladimir.Bahyl@cern.ch Presented by Thorsten Kleinwort. Outline. Current state Our typical system Possible risks Protection methods Security related Against denial of service Conclusion. A typical farm node.

jaron
Download Presentation

Security components of the CERN farm nodes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Security componentsof the CERN farm nodes Vladimír Bahyl CERN - IT/FIO Vladimir.Bahyl@cern.ch Presented by Thorsten Kleinwort

  2. Outline • Current state • Our typical system • Possible risks • Protection methods • Security related • Against denial of service • Conclusion Vladimir.Bahyl@cern.ch, Autumn HEPiX 2003, Triumf, Vancouver, Canada

  3. A typical farm node • 2 CPUs / 1 GB RAM / 20 GB disk • 100 Mbit network • CERN RedHat Linux 7.3.3 • 50-70 users • 70 primary interactive nodes Vladimir.Bahyl@cern.ch, Autumn HEPiX 2003, Triumf, Vancouver, Canada

  4. Risks • Security related: • Exploits to the system to get root • Services started on unprivileged ports • System can be used to • scan other nodes • originate spam • Denial of service: • “Heavy” processes started • Disk filled by “runaway” jobs Vladimir.Bahyl@cern.ch, Autumn HEPiX 2003, Triumf, Vancouver, Canada

  5. Our protection methods • Keep the system secure and up-to-date (with CDB & SPMA) • Log more verbosely than default • Collect the logs centrally • Scan for certain patterns in the logs • Keep the system accounting • Provide secure access methods only • Transfer sensitive information securely • E.g. password files – but in general anything; use GPG for encryption • Monitor the current state • Disk – quota is enabled • CPU usage – beniced daemon • Incident reaction – as quick as possible • No later than the next working day – compromised account is blocked Vladimir.Bahyl@cern.ch, Autumn HEPiX 2003, Triumf, Vancouver, Canada

  6. Log more • It is always good to have more information to go back to in case of a need • Daemons are configured to log as much data as is convenient • portmap -v • netlog – the ultimate kernel module • logs TCP activity • outputs a line whenever a listening socket or an incoming or outgoing connection is established • it logs the program concerned, the session id, process id and user id • it also logs connection details (protocol, local/remote addresses and ports • provides extremely useful data in forensic investigation Vladimir.Bahyl@cern.ch, Autumn HEPiX 2003, Triumf, Vancouver, Canada

  7. Netlog example • Incoming connections: Oct 14 21:49:43 node kernel: netlog: info: connect start TCP 172.17.35.98:44073 <- 192.168.161.90:52453 31440 31476 240 pmg_agent Oct 14 21:49:43 node kernel: netlog: info: connect stop TCP 172.17.35.98:44073 <- 192.168.161.90:52453 31440 31476 240 pmg_agent • Outgoing connections: Oct 14 18:18:49 node kernel: netlog: info: connect start TCP 172.17.35.98:34434 -> 192.168.11.10:80 22163 23183 1648 wget Oct 14 18:18:52 node kernel: netlog: info: connect stop TCP 172.17.35.98:34434 -> 192.168.11.10:80 22163 23183 1648 wget Vladimir.Bahyl@cern.ch, Autumn HEPiX 2003, Triumf, Vancouver, Canada

  8. Collect the logs • Make sure that all daemons log via the syslog facility • Combine the logs in a single file • Option in /etc/syslog.conf • Process log files locally on each node • Combine the connection data • Remove uninteresting information • E.g. node boot messages • Compress • Forward to a central place = Oracle database • Do it in regular intervals to prevent loss of data (~ every hour) Vladimir.Bahyl@cern.ch, Autumn HEPiX 2003, Triumf, Vancouver, Canada

  9. Scan for certain patterns • For example: • IRC activity • IRC servers violate CERN’s computing rules • SUID ptrace exploit attempts 2003/07/23-13:10:38 Uu ? node.cern.ch[172.17.35.98] kernel request_module[net-pf-14]: waitpid(28284,...) failed, errno 512 2002/01/31-01:46:23 Uu ? node.cern.ch[172.17.35.98] kernel ptrcchk: uid=19201 tried ptrace on suid/sgid file /usr/bin/passwd • Generated by a proprietary kernel module • network sniffer 2003/10/03-17:44:54 Uu ? node.cern.ch[172.17.35.98] kernel device eth0 entered promiscuous mode • repeated login failures • etc. • This data can use used together with network IDS results Vladimir.Bahyl@cern.ch, Autumn HEPiX 2003, Triumf, Vancouver, Canada

  10. Keep system accounting • Also very important element in forensics analysis • 3 months of raw data on each node • Will soon be centralized • Compressed with bzip2 -9 • Parsed, summarized and stored centrally for statistical purposes Vladimir.Bahyl@cern.ch, Autumn HEPiX 2003, Triumf, Vancouver, Canada

  11. Conclusion • When does it work well ? • Repeated intruder activity • When there is a new intrusion pattern we quickly add a new scan pattern • Intruder doesn’t know our infrastructure • What are the limits ? • First time when there is a new way to break in we do not know about • Intruder discovers our infrastructure (clusters) Vladimir.Bahyl@cern.ch, Autumn HEPiX 2003, Triumf, Vancouver, Canada

  12. Questions ? • Contact: Vladimir.Bahyl@cern.ch • Thank you !

More Related