Governance Without Metrics Is Just Dogma

Entertaining RSA Conference panel titled Why U No Haz Metrics discusses the importance of measuring security controls against exposure to loss

Without a solid security metrics program, organizations will struggle to institute risk management in meaningful ways and could be basing their security on false assumptions, an expert panel warned at the RSA conference last week.
You know what you call governance without metrics? Dogma, said Alex Hutton, director of operations risk and governance at Zions National Bank. You know what you call governance guided by metrics? Risk management.
As Hutton sees it, theres very little separating governance and risk management, but without metrics to feed into risk models, measure security performance, and relate security controls to exposure to loss, enterprises will have a difficult go at managing risk. According to fellow panelists, metrics bring the rigor of discipline and measured decision-making to the security industry.
The metrics that we try to use and leverage and develop are intended to inform and turn assumptions into understanding, more than anything else, said Jack Jones, principal of CXOWARE Inc. We have as an industry a bad habit of being a little bit superficial in our treatment of the problems we face. If we want to evolve, we have to [have] a little more critical thinking in our approach.
As they stand by themselves, risk models are essentially hypotheses. To ensure that an enterprise is truly operating under a valid hypothesis, some sort of feedback loop needs to be instituting to test against it, Hutton said.
If you talk about probabilistic pursuits, so someone who is an actuary or quantum physicist or whatever they are, theyll talk to you about concepts like model fit, model updating, yadda yadda, yadda, he said. Youve got to ask, wheres the feedback loop?
Panelists addressed the resource limitations that many security and risk professionals offer as an excuse to forgo developing a metrics program. But metrics dont have to take a lot of extra resources, said David Mortman, chief security architect at enStratus. They can often be found from existing data used in creative ways.
You have more data than you think you have, and you dont need as much as you think you need, he said. Test. See what happens.
According to Hutton, security metrics programs cant be bought, anyway.
You cant buy the metrics program or the risk management program off the shelf. Its not like you can just hire a CISSP and say, Youre the metrics guy, he said. In fact, one might argue that thats the exact opposite thing you should do. You should find some kid from the local university with a biostats or an econometrics program who is also a very creative individual and bring them in to help them out.
Panel moderator John Johnson, global security program manager for John Deere, agreed that creative staffing can make a big difference in the industry and in bringing metrics to the forefront of risk management.
You need someone with analytical skills, creative skills. My background is nuclear physics, he said. You just dont know where talent is going to come from.
With or without help from a motley crew of number crunchers, metrics dont have to be an all-or-nothing factor in risk management, said Jones, who wanted to dispel the notion that metrics only work if a department is making quantitative decisions across the entire practice.
Thats ridiculous, he said. There are marvelous opportunities for quantifying things that will make a tremendous difference in your ability to be smart about how you do your work, but most of the decisions that cross your desk on a daily basis youre going to find [are made using] expertise youve acquired over years.
Caroline Wong, director of IT governance and risk products for Symantec and former security guru for eBay, agrees.
The strength of your security program does not come from a product -- it doesnt come from a big four consulting company, and it does not come from a framework, she said. Actually, it comes from your brain.
Wong related a story of one of her first forays into developing a metrics program at eBay years ago when the firm brought in a Big Four accounting firm to help.
So they said to us, Tell us all the technologies you have and what kind of data you have, and what they gave to us for somewhat of an absurd amount of money was an Excel spreadsheet with all of these data sources, she said, explaining that this unsatisfactory result had her and her team creating something from scratch.
Where they found their success was through conversations with security and operations staff on the ground. That started by asking product developers, from a security perspective, what worried them most on the website. Vulnerabilities came up as a big answer, so the team started developing a baseline and then, rather than pushing a goal down the product development teams throat, another conversation had security asking what a reasonable goal would be.
So I think metrics is not about sifting through a mountain of data and trying to derive something meaningful, she said. I think it starts with a conversation. I think it starts with a goal, engagement of stakeholders, and proper reporting.
Proper framing of the metrics themselves is also critical, Jones said.
When were focusing on metrics, the question I ask is, What are we measuring? Whats the value proposition of those numbers in terms of loss exposure? he said. Is it telling you something about your ability to manage loss exposure over time? Im trying to characterize or frame metrics in these terms, because otherwise theyre just numbers, and theyre a waste of resources.
Have a comment on this story? Please click Add Your Comment below. If youd like to contact
Dark Readings
editors directly,
send us a message
.

Last News
▸ ArcSight prepares for future at user conference post HP acquisition. ◂ Discovered: 07/01/2025 Category: security	▸ Samsung Epic 4G: First To Use Media Hub ◂ Discovered: 07/01/2025 Category: security	▸ Many third-party software fails security tests ◂ Discovered: 07/01/2025 Category: security

**Cyber Security Categories**
Google Dorks Database	Exploits Vulnerability	Exploit Shellcodes

CVE List

Tools/Apps

News/Aarticles

Phishing Database

Deepfake Detection

Trends/Statistics & Live Infos

Tags:
Governance Without Metrics Is Just Dogma