Enterprise Firewall Comparative Test Results Show That Encryption and Evasions Matter

Six out of the eight products are Recommended with one in Neutral and the other in Caution. Firewalls will not see attacks delivered via HTTPS unless configured to do so.

AUSTIN, Texas – RSAC 2023 – April 25, 2023 – CyberRatings.org, the non-profit entity dedicated to providing transparency on cybersecurity product efficacy, has completed an independent test of eight market leading security vendors in its Enterprise Firewall comparative evaluation. Six products received Recommended ratings with high security effectiveness scores ranging from 94.05% to 99.94%.

Security Effectiveness tests measured how well the enterprise firewall controlled network access/applications and prevented exploits/evasions, all while remaining resistant to false positives. Products were subjected to thorough testing to determine their support for TLS/SSL 1.2 and 1.3 cipher suites, how they defended against 1,724 exploits, whether protection could be bypassed by any of 1,482 evasions, and if the devices would remain stable under adverse conditions.

Performance was measured using both clear text and encrypted traffic in order to provide more realistic ratings that are based on modern network traffic. Performance was measured with security enabled, and security effectiveness was measured while under moderate performance load. This was to ensure vendors did not take security shortcuts to improve performance nor enable overly aggressive security protections that would adversely impact performance. Connection rates and throughput of TLS 1.2 and TLS 1.3 encrypted traffic were significantly lower. Average connection rates of encrypted traffic were between 65% to 86.5% lower than unencrypted traffic.

Evasions were measured by taking several previously blocked attacks and then applying evasion techniques to those baseline samples. This ensured that any misses were due to the evasions, not the baseline samples. Several vendors missed evasions, with one vendor missing 72 evasions.

Key Findings:

• Encryption matters: Roughly 80% of web traffic is encrypted. The top four cipher suites account for over 95% of HTTPS traffic.
  • Decryption is not on by default: Firewalls will not see attacks delivered via HTTPS unless configured to do so.
  • There is a performance cost when TLS/SSL is turned on. Sometimes performance is significantly different.
• When a “known good” exploit is blocked by a firewall, applying an evasion technique to that exploit is often easier for an attacker than finding a new exploit that isn’t blocked by that firewall.
  • Many firewall evasion defenses are not on by default, potentially leaving customers at significant risk.
  • Most enterprises are not testing for evasions.
  • Some products have concerning gaps when it comes to evasions.

• At times, CyberRatings found multiple signatures/rules for the same CVE, with some more effective than others.
  • Attempts to provide rapid coverage for vulnerabilities that are not fully understood can result in multiple exploit-specific signatures that may be inaccurate, ineffective, or prone to false positives.
  • A single poorly written signature/rule can significantly impact performance.

“Firewalls are the keystone of most network security programs ,” said Vikram Phatak, CEO of CyberRatings.org. “It is concerning that some market share leaders are falling behind. CISOs should put pressure on those vendors to improve and look at alternatives in case they don’t.”

The following products were evaluated:

• Check Point Quantum QLS250 Lightspeed R81.20
• Cisco Firepower 2130 v7.3.1-19
• Forcepoint 2205 NGFW version 7.0.1.28052
• Fortinet FortiGate 600F v6.4.12 build5431 (GA)
• Juniper Networks SRX4600 22.3R1.12
• Palo Alto Networks PA-3220 v10.2.3
• Sangfor NGAF 5300 AF8.0.47.1004
• Versa Networks CSG5000 versa-flexvnf-22.1.1-B