340 likes | 349 Views
Learn about the evolution of Facebook's Vulnerability Management program, implementation design principles, industry standard technology, architecture mindset, and lessons learned. Discover the journey towards building a highly scalable system to manage vulnerabilities at scale.
E N D
Vulnerability Management at Scale • Alexandre Fiori • Production Engineer
Overview The evolution of the Vulnerability Management program at Facebook, the birth of the Production Engineering team to support the program, and the pragmatic approach to build a highly scalable system for an ever-growing environment. What’s this about
“Facebook was built on an open source stack. We support and encourage the use and development of open source software and hardware”
Vulnerability Management Timeline • …2015: PCI DSS • 2016: Internal scanning tools • 2017: Broader network scanning • 2018: Reboot, build up • 2019: VMaaS
New challenge • Scan all infrastructure • Merge scanning technologies • Improve vulnerability matching • Start and track remediation • Manage remediation lifecycle
Mindset • Big data • Scalability and reliability • On-line vs off-line pipelines • Concept validation and XFN • Fast prototype and launch
Collectors • Scan asset inventories and print to standard output
Processors • Pre-process, scan vulnerabilities, post-process
Reporters • Reporters manage escalation lifecycle
Inventory Classes • Hosted software • Installed software • Running software • Network scanners • Hardware
Vulnerability Database • Public vs Proprietary • Multi-vendor system • General purpose datasets • Specialized for ecosystem • Standard format for product->vulnerability matching
Design Principles • Command line tools do one thing • Communicate over a text interface • Core functionality shared as libraries • Composable code, tools, and pipelines • Rely on well established UNIX conventions
Industry Standard Technology • MITRE / NIST / NVD • Common Platform Enumeration • Common Vulnerabilities and Exposures • Common Weakness Enumeration • Common Vulnerability Scoring System
Infrastructure • Tools • Services • Data warehouse • Dashboards • Notifications
Internal Pipelines • Tupperware • Container images and packages • First-party vs third-party codebase • Bad Binary Hunter and Buck • Attribution from package to service
Vulnerability Database Tools • nvdsync and $vendor2nvd • vulndb command line tool • Uses NVD CVE JSON 1.0 format • Manages versioned datasets backed by MySQL • Supports vendor snapshots, custom CVEs, and snoozes • vulndb thrift service • CVE lookup and CPE matching
Decoration • CWE and CVSS • Domain and sub-domain • First seen • Backlog vs influx • Owner (on-call ID or UNIX username) • Threat Intelligence
Remediation • Starts from CVE inventory • Depends on decoration data • Supports feedback loop (e.g. snoozes) • Understands release cycles per inventory class • Manages escalation and lifecycle
Lessons Learned • Normalization, aggregation, and blackholes • Per-customer decision trees are burdensome • Handling delays, XFN work, and fine tuning • Tasks and notification updates are annoying • False positives can compromise credibility
Goals • Self-service system for vulnerability scan • Custom aggregation defined by collectors • Configurable providers and thresholds • Tier-based service, starting from bronze • Default dashboards and reports per tier • Common CVE inventory for all customers
Progress • XFN partnership with early adopters • Migrated “hosted software” inventory class • Total of 10+ collectors in operation • Customers fixing bogus CVEs in the database • Snoozes effectively helping fine-tune reports
Next up • Migrate other inventory classes • Improve data quality and detection speed • Tackle backlog via remediation campaigns • Tackle influx via push-blocking scans • Influence company culture outside security org