Product Ratings

Ratings in CyberRatings.org Tests

When CyberRatings.org tests products, we begin with a methodology that is published before the test. After extensive testing of a product, the test report will show a variety of important metrics on how a product defended against exploits, how many evasions could bypass protection, and if the device would remain stable under adverse conditions. An overall Security Effectiveness score for the tested product is the outcome of a number of key metrics including exploits, malware and evasions, TLS/SSL functionality, and product stability. Rated performance is based on connection rates, transaction rates, the number of simultaneous sessions, unencrypted raw data (UDP), HTTP, and HTTPS using the top cipher suites. When appropriate, we run other real-world applications.

At the end of the test, a Comparative Report is written with a Security Value Map™ (SVM) showing the test results for all products in the test and how they performed against the other products with the focus on Security Effectiveness, Performance and Total Cost of Ownership (TCO). This view for the enterprise has three ratings: Recommended, Neutral and Caution. Typically, those products receiving the highest Security Effectiveness scores would be Recommended.

Rating Scores for MEF Certification

A Product Rating is a forecast about a product’s capacity to meet its obligations to consumers over time. Product ratings inform consumers — enhancing transparency and enabling them to focus on considerations that are most critical to their organizations. Ratings range from AAA – D. A product rated ‘AAA’ has the highest rating assigned by CyberRatings.org. The product’s capacity to meet its commitments to consumers is extremely strong. A product rated ‘D’ is actively being breached by known threats and is unable to protect consumers. Ratings offer forward looking guidance on a product’s ability to meet future commitments. Test results included security effectiveness, performance, TLS/SSL functionality, management, and customer feedback.

wdt_ID	RATING	DEFINITION
1		A product rated ‘AAA’ has the highest rating assigned by CyberRatings.org. The product’s capacity to meet its commitments to consumers is extremely strong.
2		A product rated ‘AA’ differs from the highest-rated products only to a small degree. The product’s capacity to meet its commitments to consumers is very strong.
3		A product rated ‘A’ is somewhat less capable than higher-rated categories. However, the product’s capacity to meet its commitments to consumers is still strong.
4		A product rated ‘BBB’ exhibits adequate stability and reliability. However, previously unseen events and use cases are more likely to negatively impact the product’s capacity to meet its commitments to consumers.
5		A product rated ‘BB,’ ‘B,’ ‘CCC,’ ‘CC,’ and ‘C’ is regarded as having significant risk characteristics. ‘BB’ indicates the least degree of risk and ‘C’ the highest. While such products will likely have some specialized capability and features, these may be outweighed by large uncertainties or major exposure to adverse conditions.
6		A product rated ‘BB’ is more susceptible to failures than products that have received higher ratings. The product has the capacity to meet its commitments to consumers. However, it faces minor technical limitations that have a potential to be exposed to risks.
7		A product rated ‘B’ is more susceptible to failures than products rated ‘BB’; however, it has the minimum capacity. Adverse conditions will likely expose the product’s technical limitations that lead to an inability to meet its commitments to consumers.
8		A product rated ‘CCC’ is susceptible to failures and is dependent upon favorable conditions to perform expected functions. In the event of adverse conditions, the product is not likely to have the capacity to meet its commitments to consumers.
9		A product rated ‘CC’ is highly susceptible to failures. The ‘CC’ rating is used when a failure has not yet occurred, but CyberRatings considers it a virtual certainty.
10		A product rated ‘C’ is highly susceptible to failures. The product is expected to fail under any abnormal operating conditions and does not offer a useful management systems and logging information compared with products that are rated higher.
11		A product rated ‘D’ is actively underperforming and failing and does not meet the use-case. The ‘D’ rating is used when the product is not operational without a major technical overhaul. Unless CyberRatings believes that such technical fixes will be made within a stated grace period (typically 30-90 calendar days), the ‘D’ rating also is an indicator that existing customers using the product have already experienced a failure and should take immediate action.

Ratings in CyberRatings.org Tests

Rating Scores for MEF Certification

Sign Up for our Newsletter