While catching up on security news and blogs the other day, I came
across a blog post from ICSA Labs entitled “Why a Test Lab Needs to be Wary of Commercial Exploit Packet Captures” and thought that it would be a good conversation starter to inform our readers about how BreakingPoint approaches developing test cases for security device testing, our methodology behind why we develop our test cases the way we do, and the thought processes and conclusions behind those decisions.
First, it’s important to note that ICSA’s blog post is primarily talking about test tools that replay packet captures as their security tests. While the BreakingPoint devices do provide a packet capture replay component, this component is not what we use for security testing. The BreakingPoint devices provide a dedicated security component that
execute packaged attacks targeting individual vulnerabilities that we call “strikes”. Strikes are not packet captures, and we’ll discuss how strikes operate and the benefits derived from them a little later in this post.
Toward the beginning of their blog post, ICSA wrote the following:
“If ICSA Labs were to use one or more exploit packet captures created elsewhere, then we would be effectively vouching for the quality and accuracy of these packet captures. But that is the problem; we cannot vouch for their quality and accuracy.”
This is also one of the primary reasons that we do not use packet captures of attack traffic that we have come across in our research. However, we take it one step further and and don’t even use packet captures created in-house. We simply don’t use packet captures for security testing at all, which brings me to the first subject I’d like to discuss:
Attack Realism
Let’s look at what ICSA has to say on attack realism using third-party packet captures:
“ICSA Labs does not know whether the code for each would-be exploit actually works as expected. Even if it did work, we cannot confirm that the would-be exploit was run against a vulnerable system when the capture was made. And assuming it was a working exploit that was run against a vulnerable system, we do not know whether the attack succeeded when the packet capture was made. Also, information in the commercial tool typically indicates at which vulnerability each exploit packet capture is aimed. But again, a test lab has no reasonable way to confirm that. To use the tool in this way ICSA Labs would have to make many assumptions and essentially trust an entity outside of our control.”
The BreakingPoint Labs team builds each strike by hand after performing our own analysis of the vulnerability. We have a high degree of certainty that our attacks are correct because we do this analysis and
then we test the strikes afterward when possible against the actual vulnerable target. Then, we use these strikes (not packet captures of them) in testing performed using the BreakingPoint device. There are currently two ways to test using these strikes; passing attack traffic
through an intermediary Device Under Test (DUT), and sending attack traffic directly to an endpoint DUT, which I’ll cover next.
Attack Simulation
“But what happens if the vendor’s IPS proxies traffic or alters the content of traffic as some IPS products do? Keep in mind that this is a replayed packet capture, not a live exploit. If the commercial tool with its packet capture of an exploit is run against an IPS that does one of these things and the IPS fails to block the attack, did the IPS really fail? Remember, the IPS modified the traffic on the line.”
This is a valid concern when testing an intermediary DUT, and even more so when you’re using static data from packet captures. In this scenario, our strikes act as both the attacker and target, and send the attack traffic from one port on our device, through the DUT, and back to a second port. In this way, it’s really an attack “simulation” using real attack
traffic because we’re essentially sending traffic back to ourselves rather than a real target. Because we know what valid attack traffic looks like for each individual iteration of the strike, we know what data we’re sending, and we know what the data should look like when we (the
target) receive it, if the DUT modifies the attack traffic in transit we consider the attack blocked as it is no longer the attack traffic that we sent and is invalid.
“One-Arm” Strikes
“If the IPS vendor cannot reproduce the issue reported to them by the test lab, then the test lab should be able to confirm its findings in some way. But minus the real attack and actual vulnerable system, that is either a very tall order or impossible!”
Once again, we’re in total agreement here, which is why we use real attacks. To the extent possible, strikes that target servers can be run in “one-arm” mode where rather than passing attack traffic through a DUT and back to ourselves, the traffic is sent to the DUT as the attack’s target server. In this mode, strikes can be used to actually trigger vulnerabilities on actual vulnerable systems. This is what test houses that use BreakingPoint devices like NSS do to verify that the test cases they are using are indeed valid, even though they are provided by BreakingPoint, their vendor.
Custom Strikes
What if BreakingPoint doesn’t have a strike for the vulnerability you want to test? Or what if, like ICSA, you don’t trust third party content at all? Even though BreakingPoint provides you with real
attacks packaged as strikes, users can easily develop their own strikes. I won’t cover this topic in any detail here, as we’ve already had a three part series (1, 2, 3) on this subject posted to the blog.
Strike Development Goals
-
Trigger Just the Vulnerability & Use Unidentifiable Payloads
-
Randomness & Uniqueness on the Wire
-
Evasions
One of the most frequently raised concerns about our strikes is that
they contain no active payloads or executable shellcode. This is by
design. Sure, network security devices often have filters for
well-known shellcode and common payload encoders, and we have specific strike categories to test those specific cases, however if you are relying on the detection of such by your IPS to protect you from actual vulnerabilities then you
have already failed. Most network security devices are reactive in
nature, and in order to detect a particular shellcode or payload encoder, it must first be aware of it and/or have a filter for it. We know there are payload encoders and shellcode out there that devices are unaware of, so we simulate this by using completely random data as our payloads. This forces the DUT to identify attacks based on the properties of the vulnerability, not by relying on detecting known shellcode or a decoder stub from an encoded payload. We focus entirely on triggering the vulnerability, not actually exploiting it with an operational payload.
“ICSA Labs is unwilling to risk its reputation and the trust of end users through the use of packaged exploit packet captures in its testing. All of the exploit packet captures we use in network IPS testing were captured here in the lab by our experts. And in ALL cases, we are in a position to verify our coverage protection test results by running the real, live attack against the actual vulnerable system.”
The problem with ICSA’s approach here is that you’re initially still testing with static packet captures. Consider the scenario where you replay your packet capture of a malicious TIFF file traversing the wire. The IPS under test blocks it, and you mark that as a success. How do
you know that if some unrelated parts of the TIFF file are modified, that the IPS won’t miss it? How do you know that if you add a whole lot of padding or superfluous structure to the file and move the evil from the beginning of the file to beyond the padding, that the IPS won’t miss it? If you’re initially relying on packet captures of static attack traffic and then only breaking out the real exploits and targets when something seems amiss or a customer questions your tests, you’re not being thorough in your testing.
BreakingPoint’s approach to providing these various attack permutations is to identify all of the components of the attack that are absolutely essential for the attack to work and trigger the vulnerability. We identify these values and their upper and lower bound thresholds as well as identify behavioral protocol and process interactions and what combination and permutations of these are valid. We then develop our strikes to randomize these properties as much as possible while still
conforming to the identified valid parameters. Further, we randomize as much other data as possible that is not directly related to triggering the vulnerability while still remaining valid for whatever protocol, file format, or other data structure is being used in the attack. All of this context information and the flexibility provided by dynamic test cases such as strikes as opposed to packet captures is the benefit we get from performing the vulnerability analysis ourselves, understanding the operational bounds of the data involved, and developing strikes that launch attacks that actually utilize that knowledge. You can read more about this subject in one of my previous blog posts, File Format Vulnerabilities and Dynamic Exploit Generators.
To further the previous point, BreakingPoint can optionally also mutate attack traffic by employing various evasion techniques. When you combine evasion techniques such as IP fragmentation with fragment reordering, using various text encoding methods, and HTTP chunked encoding transmission, among others, with the randomization of the attack traffic that we are already performing as outlined in the previous section, nearly endless permutations of a single attack are dynamically generated which using static packet captures simply can’t compete with. Forgive me for quoting a deodorant commercial, but “anything less would be uncivilized.” For much more in-depth information on the subject of evasions, please see our recent webcast entitled Harden Security Devices Against Increasingly Sophisticated Evasions or this previous blog post on the subject.
Conclusion
I hope you enjoyed this look into the BreakingPoint strike development and security device testing mindset and found the information both useful and enlightening. Please do follow some of the links above as there is much more information available about the topics discussed.