The Trouble With Security Metrics

A Q&A with the author of The Security Risk Assessment Handbook

Security practitioner Doug Landoll is passionate about risk assessments and security measurements. Author of
The Security Risk Assessment Handbook
and CEO of Assero Security, a risk consultancy for SMBs, Landoll believes the industry engages in far too many theatrical risk assessments for the sake of audits. These assessments never return solid measurements of risk because the collection methods are faulty, he says. As organizations seek to meet risks head on, they need better visibility into which security initiatives work, which dont, and which need improvement. Done right, security metrics can help provide the estimates to plan out effective strategies.
Dark Reading
recently caught up with Landoll to talk about his thoughts on how organizations can improve their collection methods to create security metrics that mean something.
Dark Reading
:
I know youre a big believer that if you depend on a faulty method of gathering data, security metrics wont really matter for much. Can you explain a little about your philosophy when it comes to measuring risk?
Landoll:
You want to measure whats important to the organization, instead of thinking what you measure is whats important. Theres a difference.
It is what I call the SEIM dog wagging the governance tail. When youve got some great technology thats pumping out tons of data, the bottom-up approach is to say, Wow, look at all this data. Lets build a dashboard. So you get this dashboard with all these really cool metrics and trends and everything, and now that becomes what you measure and that becomes whats important to you -- which is way backward. Whats important to you should be based on your company mission. Based on your company strategy, your security strategy should have some goals youre trying to get to -- those are the things to measure.
Just because that metric is not being produced by a piece of technology, its a mistake if we dont grab it, we dont report it, we dont trend it, we dont correct our strategy. Top down is the only way to do metrics.
Dark Reading
:
Can you give an example of a risk measurement that people wouldnt think of because maybe they dont have an automated dashboard to spit it out at them?
Landoll:
Lets say youre going to spend money to develop a computer incident response. I think you have some expected improvements from that. Before the project even starts, you can collect metrics on it. I think youre expecting a decreased time from incident discovery to recovery. I think youre expecting to be able to demonstrate compliance with breach notification requirements, and I think you want to minimize damage from incidents. If I can prove those things, that has been a successful project.
Im not sure that people are trying to take the time from incident discovery to recovery. I dont think it would be that hard, but I think thats not pumped out by a tool, so, therefore, youre not usually going to measure it. So how would you do that?
One example would be to grab three to five common incidents that happen and measure the various phases. I think you should be able to have a date and time for the detection phase, the analysis phase, containment, eradication, and recovery. Heres how we get the call, how long it takes them to figure out what else it is, etc., etc. And then you check to see if that is getting any better six months down the line.
That would be interesting. But as you know, there is no tool that does that. But you have a ticketing system and maybe its in the incident report. But its there somewhere.
[How are CISOs preparing for 2013? See
7 Risk Management Priorities For 2013
.]
Dark Reading
:
What about metrics around security awareness? Its seems it is easy to game that system by measuring how many people have been trained versus actual results.
Landoll:
Some other things you can do are testing for susceptibility to phishing attacks. Use a company like PhishMe. Its a pretty affordable service. Do it on a regular basis and get reports on the results. I would hope that in six months, when the culture gets used to these phishing emails going around and people get embarrassed by clicking on them, nobodys going to fall for that stuff anymore. Or there will be a considerable improvement.
Dark Reading
:
When it comes to thinking up new metrics or effective ways to measure security, what kind of mindset do you need to take into that process?
Landoll:
I find its not cookie-cutter, but it tends to be easier than youd think. The methodology I use is, I start at the high level and say, What would I like to know? Then I think about whether there is any information out there. And if its not there, I figure out how to generate it. So lets say, for example, I want to know how many laptops might be stolen this year from my traveling consultants. I think that datas there, or at least last years data is there, so who would have that at a company? I think, the guy that ordered those new laptops.
Hell just tell you, it was six last year and there were eight the year before. So you have a pretty good guess its between five and 10 this year -- unless youve done some kind of change control.
I think if you start at the high level and think, heres what Id like to know, where would that information be, or how would I create it. Very few times youll have to create it, but then when you create it, its not as hard as you think. It could be a survey, it could be, Lets talk to the guy in charge and see what value hes put on that. People shy away from that because they feel inaccurate. But measurement is reduction of uncertainty. It is estimation. We dont have to say well lose exactly 7.1 laptops per year. Its PK to say its between five and 10.
I would also recommend Doug Hubbards book. Its called
How To Measure Anything: Finding the Value of Intangibles in Business
. Theres a chapter on security. He certainly understands our industry. The mistake everyone makes is that they think their industry is unique and theyve never had this problem before, and he says nonsense to that. People had all of these problems before and they thought of clever ways to grab metrics.
Dark Reading
:
In your opinion, what would you say are some of the biggest mistakes people make when measuring security?
Landoll:
I would say letting the available data drive the metrics program. Thats a huge mistake. Another one is not collecting enough data points and relying on single data points, or assuming that the data you want isnt available and settling for something else. I think thats a poor assumption.
Dark Reading
:
Youve certainly crusaded against relying on single sources of data for risk assessments and measurable, advocating for what you call the RIIOT (Review, Interview, Inspect, Observe and Test) method of risk assessment. Why is it so important to pull metrics and assessment data from different collection methods?
Landoll:
There are a lot of approaches to risk management that are not accurate. When you boil down your security program to a 50-question questionnaire, and you divide it among the people in the organization, send it out, and compile it, you know its not accurate and they know its not accurate. Were just checking a box and when the auditor comes, we say, Here, we did a risk assessment. Good luck.
This is your plan. This is your strategy. This is where you determine which security activities youre going to be doing in the next few years. And yet youre not collecting the data, and youre making wrong conclusions, and youre getting budget, making a plan based on faulty data. Thats alarming.
Lets say you do a traditional risk assessment where youre reviewing documents and youre interviewing people about security awareness training. The first interview is going to come out really good -- people tend to want to make themselves look good. If thats your only input, you have to conclude that security awareness training doesnt need to be improved. However, if you were to observe that post-it notes were on those screens, and you just tried one social engineering trick and it worked, now youve got to conclude that its not effective. Thats a complete 180. And just by doing a few more tests.
Another good example is on system hardening. Its not about finding the error; its about finding the root cause. You can do a scan, find a vulnerability, and say you patched it. But what if I reviewed the documents and then saw that the hardening documents werent detailed enough? And I interviewed people and they said, Theres change management procedures, but we dont use those, and we didnt write that hardening document and dont use it. So the recommendation from that shouldnt just be to patch the system. Its really a little training, some governance, some hardening documents.
Im just really concerned that the one security activity that we do as professionals to help plan out our strategy suffers from way too many shortcuts.
Have a comment on this story? Please click Add Your Comment below. If youd like to contact
Dark Readings
editors directly,
send us a message
.

Last News
▸ ArcSight prepares for future at user conference post HP acquisition. ◂ Discovered: 07/01/2025 Category: security	▸ Samsung Epic 4G: First To Use Media Hub ◂ Discovered: 07/01/2025 Category: security	▸ Many third-party software fails security tests ◂ Discovered: 07/01/2025 Category: security

**Cyber Security Categories**
Google Dorks Database	Exploits Vulnerability	Exploit Shellcodes

CVE List

Tools/Apps

News/Aarticles

Phishing Database

Deepfake Detection

Trends/Statistics & Live Infos

Tags:
The Trouble With Security Metrics