NSS Labs Revises Endpoint Security Test Model

New product ratings system comes amid growing shift in the testing market toward more open and transparent evaluation of security tools.

[3/5/2020 This story was updated with a correction that NSS Labs will continue to use its Recommended, Neutral, and Caution notices with endpoint products, in addition to the new ratings system, and with additional detail that its nonprofit will cover consumer security products, not just IoT.]
Cybersecurity testing company NSS Labs, which was quietly acquired by a private equity firm late last fall, has launched both a new ratings system for endpoint security product testing for that product category, and a new nonprofit testing organization for consumers for security and Internet of Things (IoT) products.
NSS Labs in October 2019 was purchased for an undisclosed figure by private equity firm Consecutive Inc., a move that was not publicly announced by the companies but which they later confirmed. Multiple sources close to NSS Labs described the merger as a fire sale of sorts to restructure the company amid financial woes, but NSS Labs CEO Jason Brvenik tells Dark Reading that the deal represents a reorganization by the company in order to better focus its resources.
According to Brvenik, the previous venture capital (VC) model was not a fit for NSS Labs or the testing market, mainly due to VC focus on growth and product. NSS Labs was under pressure from investors to sell a security-as-a-service threat intelligence offering for exploits, but the now-defunct Cyber Advanced Warning System (CAWS) service failed to gather steam among enterprises. CAWS, which was developed by NSS and had
integrated
with various threat intel vendors in that space, alerted customers on active exploits in the wild.

NSS Labs since has folded its CAWS technology into its testing as a bundled service offering, according to the company.
What we heard from the market was they didnt want more work from us [with the service]; they wanted answers and not data that makes them do more work, Brvenik says.
Were now back to focusing on what we are really good at and what were known for, he says. It allowed us to look more at what we deliver to market and to make pivots to the cloud and other areas, he noted.
NSS Labs announced the new initiatives of new test rankings and a new nonprofit testing arm for consumer security and IoT products during the RSA Conference in San Francisco last week. The new product ratings method, which the testing firm has first launched for endpoint protection products, rates vendor tools based on the criteria of management, false-positive rate, resistance to evasion, total cost of ownership, and their block rate of malware, exploits, and targeted attacks. NSS said it will also will continue to flag products as Recommended, Neutral, or Caution, as well as now rate the products on a grading scale of AAA as the highest to D as the lowest.
The Testing Conundrum
The new moves by NSS Labs come at a time when traditional security product testing is undergoing a slow but welcome transformation. Vendors and test labs long have had an uneasy and often contentious relationship over control of the testing parameters and process, and NSS Labs at times has been at the center of that battle: The company in May 2019
retracted and apologized
for a 2017 publicly released
endpoint protection test
report on CrowdStrikes Falcon, which CrowdStrike in turn challenged in a lawsuit alleging that the test was incomplete and used illegally obtained Falcon software.
CrowdStrike had hired NSS Labs the year before to conduct private testing of Falcon, but later
terminated the testing engagement
over concerns over the quality of the tests after it detected legitimate apps as malicious. NSS Labs continued to publicly test Falcon, using software it had acquired through a reseller.
In September 2018, NSS Labs
filed an antitrust lawsuit
against cybersecurity vendors CrowdStrike, ESET, and Symantec, as well as the nonprofit Anti-Malware Testing Standards Organization (AMTSO), over a vendor-backed testing protocol it deemed as unfair and vendor-centric. AMTSOs testing protocol aims for transparency between testers and vendors.
But NSS Labs dropped the lawsuit in December 2019, citing progress in how AMTSO and vendors were working with test labs. That doesnt mean that NSS Labs is now all-in for the AMTSO testing protocol, however: Brvenik says NSS Labs has no plans to adopt the AMTSO protocol for its testing programs. We have not seen sufficient evolution there, he says. It remains a vendor-driven environment.
Meanwhile, enterprises — at which testing is aimed — have been caught in the middle of such spats and faced with an often opaque testing model that critics have described as a vendor pay-to-play. Most dont have the resources to conduct their own in-house testing of security products, so they are left with recommendations from consulting firms, third-party testing organizations, or just claims of the vendors.
Brian Monkman — executive director of NetSecOPEN, an industry organization that coordinates network security performance testing based on its Internet Engineering Task Force standard-based process — says enterprises should be able to get open and transparent security testing from a neutral third-party testing organization.
When enterprises are looking at testing results to help them decide what security products get short listed, they need to look at how the testing was done and the level of detail, and what level of detail security product vendors are prepared to provide, Monkman says. An open and transparent nature is starting to emerge in the endpoint testing market.
Take Mitres commercial testing of endpoint security products, which it launched in late 2018. The nonprofit evaluates the products against its ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) model, using well-documented attack methods and techniques employed by nation-state and other advanced threat groups. Mitres tests are based on open standards and methods, and the vendors perform live defenses with their products. Its a more collaborative test environment, many security vendors are embracing it, and the results are made public.
Meanwhile, the goal of NSS Labs new product ratings scale is to ensure that organizations can choose the product or technology that best fits their needs, which may not be the most leading-edge product, notes Brvenik. Scoring endpoint products with a percentage grade is not necessarily representative of just how good a product is, he says.
If you have five products and four of them are at 99.9% and one is at 99.5%, its going to look like it [stinks] in a 2D [two-dimensional] axis, even though its a great product. That model didnt fit well in that space, he says.
Chester Wisniewski, a principal research scientist with security vendor Sophos, says customers are demanding more transparency from security vendors. But there are plenty of challenges with endpoint testing, including that vendors can block only the threats they know about, he notes. Theres no way to test nation-state stuff with todays tests, he says.
The underlying issue is that more attacks today are launched by humans behind a keyboard using stolen credentials. The human will just keep changing the malware until they get through, Wisniewski says, a scenario thats difficult to simulate in most test environments.
IoT
The details of NSS Labs new nonprofit are still being ironed out, but the organization will use NSS Labs test infrastructure to put consumer security and
IoT products under the security microscope
and publish the results for the public. One concern is the intersection between enterprise networks and their users when they go home to smart devices and their Wi-Fi networks.
NSS Labs isnt the first to take on consumer IoT security testing: Theres the Cyber Independent Testing Lab (CITL), a nonprofit led by Peiter Mudge Zatko and Sarah Zatko, which recently teamed up with Consumer Reports on a digital standard for consumer privacy, for instance. They are doing cool consumer stuff, and [looking to conduct] cybersecurity testing in a rigorous environment, says security expert Bruce Schneier.
Schneier also points to a security and privacy consumer labeling project underway at Carnegie Mellon Universitys CyLab, which is building
a prototype Privacy and Security Label
akin to a nutrition label that could be affixed to an IoT products box. The goal is to help inform consumers about an IoT products security and privacy features — or lack thereof — including how data it collects is used and whether or how it requires authentication, for example.
Related Content:
Latest Security News from RSAC 2020
MITRE Changes the Game in Security Product Testing
NSS Labs Files Antitrust Suit Against Symantec, CrowdStrike, ESET, AMTSO
Cybersecurity Certification in the Spotlight Again
Assessing Cybersecurity Risk in Todays Enterprise
Check out
The Edge
, Dark Readings new section for features, threat data, and in-depth perspectives. Todays featured story:
How to Prevent an AWS Cloud Bucket Data Leak
.

Last News
▸ Researchers create BlackForest to gather, link threat data. ◂ Discovered: 23/12/2024 Category: security	▸ Travel agency fined £150,000 for breaking Data Protection Act. ◂ Discovered: 23/12/2024 Category: security	▸ 7 arrested, 3 more charged in StubHub cyber fraud ring. ◂ Discovered: 23/12/2024 Category: security

**Cyber Security Categories**
Google Dorks Database	Exploits Vulnerability	Exploit Shellcodes

CVE List

Tools/Apps

News/Aarticles

Phishing Database

Deepfake Detection

Trends/Statistics & Live Infos

Tags:
NSS Labs Revises Endpoint Security Test Model