160 likes | 173 Views
Cyber Metrics in the DoD or How Do We Know What We Don ’ t Know?. John S. Bay, Ph.D. Executive Director. Things People Have Asked Me. How much money should I spend this year on cyber defense technologies? How many attacks has your firewall repelled this month?
E N D
Cyber Metrics in the DoDorHow Do We Know What We Don’t Know? John S. Bay, Ph.D.Executive Director
Things People Have Asked Me • How much money should I spend this year on cyber defense technologies? • How many attacks has your firewall repelled this month? • If I only had a dollar to spend on cyber, where should I spend it? • Why is cyber research such a slog?
Answers(which did not go over well) • How much money have you got? • We repelled all of them … except that one you read about in the paper • Spend your dollar on upgrades • Cyber research is a slog because there is no physics theory underlying it all, liker Maxwells’ Equations or Newton’s Laws
But really … it DEPENDS • The “threat” factor is common in cybersecurity, but mostly not elsewhere • … and it IS true that there is no useful PHYSICS for the problem
DoD Taxonomy of Threats From: Defense Science Board, Resilient Military Systems and the Advanced Cyber Threat, January 2013
So Then, What to Measure? • Qualitative • Capabilities • Missions lost • Quantitative • Performance • Cost • To achieve • Not achieving
“Stoplight Chart” Assessments See: SPIDERS JCTD
Costs to Us • All vulnerabilities are bugs • All code has bugs • Bugs are expensive • Exploits are cheap the “asymmetry” problem
Mission-Assurance Approach • Helps focus attention • Requires a “map” o the mission • Implies a prioritization on missions (something loses) • Requires reconfigurable systems and networks • Is not cheap From: DUSD(I&E) Office, HANDBOOK For SELF-ASSESSING SECURITY VULNERABILITIES & RISKS of INDUSTRIAL CONTROL SYSTEMS On DOD INSTALLATIONS, December 2012
Just Good Enough (Incremental)Approach • How long would our red team take to penetrate the system? • An empirical measure, at best. • Implies a canonical red team Bad code prob(first vulnerability is discovered) Gamma distribution? Better code time
The Accountability Approach • NIST 800-53 guidelines • The “did we do everything we know how to do” approach From: NIST Special Publication 800-53, rev. 4, Security and Privacy Controls for Federal Information Systems and Organizations, April 2013
Conclusions: Which is Best? • None of them. They service somewhat orthogonal purposes. • But they can provide apples-to-apples comparisons • Can they answer the Generals’ questions? • No • … except maybe the one about the firewall • There is CERTAINLY no satisfactory “physics” to guide anybody • Cyber Metrics is still an extremely important and high-priority problem for OSD!