CrowdStrike Fails In Bid To Stop NSS Labs From Publishing Test Results At RSA

NSS results are based on incomplete and materially incorrect data, CrowdStrike CEO George Kurtz says.

A security vendor’s failed attempt to prevent a report containing details on its product’s capabilities from being released at the RSA conference in San Francisco this week has focused attention on the need for more widely accepted testing and validation services for security technologies.
The report is from NSS Labs, a company that bills itself as an independent security product-testing firm, which also sells its own threat protection platform. NSS recently conducted security tests on Advanced Endpoint Protection (AEP) products from 13 vendors and released the results of its testing at RSA this week.
Each of the products, according to NSS, was tested against multiple attack threat vectors for their resilience against evasion techniques, their tendency to generate false positives, and other metrics. The products were then graded for their overall security effectiveness and total cost of ownership per protected endpoint agent and assigned a rating of “Recommended,” “Security Recommended” or “Caution”.
Security vendor CrowdStrike, one of two vendors in the report to garner a “Caution” rating
filed a lawsuit
in federal court in Delaware last week seeking to prevent NSS from releasing the report at RSA. In a legal brief, CrowdStrike claimed that NSS’ analysis of its Falcon advanced endpoint protection technology was based on incomplete testing conducted via illegally obtained access to its software and against CrowdStrike’s specific instructions to stop.
The court dismissed CrowdStrike’s request for a temporary restraining order and preliminary injunction against NSS Labs on the grounds that the company had failed to show how it would suffer irreparable harm from the report becoming public. It will now decide whether NSS acted illegally when conducting its tests on CrowdStrike’s Falcon technology.
George Kurtz, president and CEO of CrowdStrike says his company’s dispute with NSS has to do with the incompetent manner in which the tests were initially conducted and the fact that the results are based on incorrect and materially incomplete information.
CrowdStrike first hired NSS last year to conduct private testing on Falcon but quickly put a halt to it over concerns about the competency of the testing and the methods used, Kurtz says. As one example he pointed to NSS flagging legitimate software such as Skype and Adobe as malicious, during testing.
Even after paying a total of $150,000 for two private tests on Falcon, CrowdStrike was left with no confidence about NSS’ abilities and decided not to participate in a subsequent public test of advance endpoint protection products by NSS.
NSS however, proceeded to conduct public tests on Falcon anyway, using access it obtained illegally via a reseller. When CrowdStrike discovered what was going on the company immediately pulled NSS’ access to Falcon. So any subsequent conclusion that NSS made about Falcon’s effectiveness were based on incomplete and materially incorrect information and without all product features being turned on, Kurtz says.
“This is not about trying to silence independent research,” Kurtz says. “We welcome open, fair, transparent and competent testing. We didn’t necessarily see it here. This isn’t the Consumer Reports of cybersecurity. It’s bad tests, bad data and bad results.”
The episode shows why it is important to have more standard processes and organizations for testing the effectiveness of security products, he says. While companies like NSS Labs purport to be independent and unbiased, he says they have a for-profit business model and make money selling test reports and their own security platforms. “Unfortunately, there is no Underwriters Laboratories or testing house for security products,” Kurtz says.
Vikram Phatak, CEO of NSS Labs expressed regret over CrowdStrike’s stance. “We obviously disagree and are disappointed with Crowdstrikes characterization of NSS,” he said in emailed comments.
Though Crowdstikes request for a temporary restraining Order and preliminary injunction were denied by the federal court, the lawsuit itself is still pending. “So we are limited in what we can say,” Phatak said. “Whether or not it is their intent, their suit has the effect of keeping us from debating the facts publicly.”
Such disputes over product testing and validation are not unusual says Pete Lindstrom, an analyst with IDC. Still, it is slightly unsettling when companies get as aggressive as CrowdStrike did in its dispute with NSS Labs, he says.
Technology buyers generally understand the limitations of product testing and know how difficult it can be for testers to replicate actual real world conditions when assessing the effectives of security products, Lindstrom says.
“In my mind security vendors should be looking for ways to get more information about the efficacy of their products out into the market,” he notes. “Anyone confident in their solution, should be willing to do that even within the limitations of testing.”
As long as all the circumstances of the testing are described fully and vendors have a chance to rebut claims, users are well served, Lindstrom said. An enterprise buyer that is already interested in CrowdStrike for instance is unlikely to view the company negatively based on one report, especially if all details are fully available.
“These reports are generally used for confirmation rather than negation,” he says.
Related Content:
What to Watch (& Avoid) At RSAC
Darkness & Hope On Display At RSA Conference Keynotes
National Security, Regulation, Identity Top Themes At Cloud Security Summit

Last News
▸ DHS-funded SWAMP scans code for bugs. ◂ Discovered: 23/12/2024 Category: security	▸ Debunking Machine Learning in Security. ◂ Discovered: 23/12/2024 Category: security	▸ Researchers create BlackForest to gather, link threat data. ◂ Discovered: 23/12/2024 Category: security

**Cyber Security Categories**
Google Dorks Database	Exploits Vulnerability	Exploit Shellcodes

CVE List

Tools/Apps

News/Aarticles

Phishing Database

Deepfake Detection

Trends/Statistics & Live Infos

Tags:
CrowdStrike Fails In Bid To Stop NSS Labs From Publishing Test Results At RSA