How Mutant Exploits Defeat IDS 2006

How Mutant Exploits Defeat IDS

A report on the ACM paper:
“Testing Network-based Intrusion Detection Signatures Using Mutant Exploits”
Downloaded from:
http://portal.acm.org/browse_dl.cfm?linked=1&part=series&idx=SERIES320&coll=portal&dl=ACM via a search for “mutant”.

A well known network security concept, Intrusion detection systems function by listening on the network for suspicious activity. This paper explores some of their weaknesses and shows that they are not very effective when exploit mutations are applied.

According to the authors the tested IDS's are “network-based misuse detection systems”. An example of network misuse could be running an exploit against a server, scanning the entire network for hosts, or sending traffic which results in a Denial of Service attack.

The authors preform rigorous tests of 2 IDS systems, ISS RealSecure and Snort. These were chosen because they “represent the leading products in the open-source and commercial worlds” and “possess excellent attack detection capabilities and correctly manage many different types of evasion”. Both of these IDS systems are reliant on “signatures” of known attacks. At first glance this seems a very effective way of detecting network-misuses, however this paper shows that in reality, doing this is much more difficult.

It is difficult because “attacks that exploit a certain vulnerability may do so in completely different ways”. While it is easy to write IDS signatures for publicly known exploits, realistically not all exploits are going to be released, nor will all versions of a specific exploit be released. One would think “well how am I supposed to detect misuses that I don't even know exist?”. A valid question, however the problem gets even worse.

IDS systems typically have signatures for thousands of exploits. These signatures are very static, effectively searching for a specific packet or set of packets sent across the network. Problems arise however, when an exploit is modified or “mutated” in ways that allow it to still function in compromising a host, yet in a way which obfuscates the network traffic. The paper breaks down such “Mutation Mechanisms” into a number of groups. 

Network Layer Mutations include using Ipv6 to evade Ipv4 only IDSs, and IP Packet Splitting to evade IDS systems that are not smart enough to reassemble packets before checking them against attack signatures.
Application Layer Mutations include Protocol Rounds, FTP Evasion Techniques, HTTP Evasion Techniques, and SSL NULL Record Evasion Techniques. All of these mutations change the data sent to the application by the exploit, in a way which the application can still understand the attackers message however the message does not match the IDS signature for the attempted exploit. Also, “examples of HTTP protocol evasion techniques include neglecting the use of carriage returns, random insertion of whitespace characters, and inserting junk characters into parsed numerical fields”. While RFCs strictly specify protocols, often times real products are more lax in regards to them, allowing for the strange behavior which makes such mutations possible.

Exploit Layer Mutations are the newest of the group. They include polymorphic shellcode, and Alternate Encodings. ADMmutate “generates self-decrypting exploit payloads that will defeat most popular NIDS shellcode detectors, including "XOR-encoded payloads, randomized NOP generation, and support for banned characters".

The shellcode part of the exploit is easily added to IDS signatures. Since many exploits use predefined shellcode. IDS check for shellcode because regardless of the exploit some shellcode usually has to be pushed onto the compromised host in order to run some command for the attacker. Encoding the shellcode in a different format works as does the “insertion of non-destructive junk instructions and the reordering/substitution of code”. In this way you have a functional shellcode which does not match any IDS signatures.

Alternate Encodings work by encoding the exploit into some format the server can decode and execute, but the IDS does not. Examples include TAR, ZIP, or url encoding via hex.

The paper tested 10 exploits mutated in various ways. The exploits were run against the server. Detection results were collected from the IDS systems. Success results were collected from an “oracle” application which they used to check whether the exploit was successful.

The exploits were chosen to represent the range of target operating systems, protocols, and categories of attacks. Linux, Windows, OpenBSD, FTP, HTTP, IMAP, RPC, and SSL were tested, along with buffer overflow and directory traversal attacks.

The following specific exploits were tested:
IIS Escaped Characters Double Decoding
WU-ftpd Remote Format String Stack Overwrite
Microsoft DCOM-RPC aka blaster
IIS Extended Unicode Directory Traversal
NSIISlog.DLL Remote Buffer Overflow
IIS 5.0 .printer ISAPI Extension Buffer Overflow
WS-FTP Server STAT Buffer Overflow Denial-Of-Service
Apache HTTP Chunked Encoding Overflow
OpenSSL SSLv2 Client Master Key Overflow

Each exploit was run through the authors' “mutation engine” to generate mutant exploit combinations. Once a particular exploit was found which evaded the IDS systems yet still functioned, the authors moved on to the next exploit. One point made by the authors is that for some exploits the number of functional mutations could be quite large, leading to “an unmanageable number of mutant exploits”. The mutation engine tried using a variety of the previously mentioned methods.

The results of all these tests were quite sobering. Snort only detected 4 out of 10 of the mutant exploits, and IIS RealSecure only detected one. So what does this tell us? That a determined attacker could probably easily evade most commercial signature based IDS systems. I would say that signature based IDS are good for detecting "canned" exploits run at the "script kiddie" level. However, for someone who understands how they work, they seem to be rather trivial to defeat with automated tools such as ADMmutate.

Security means defense in depth, and signature-based IDS systems are only one layer. Another type of IDS system is based on Cisco's netflow technology. While I am not aware of any commercial products which do this, it is possible to correlate the data from Cisco router's netflows, in order to detect suspicious patterns such as port scans. Professor Skwarek touched on this briefly in his network security class.

Another layer of IDS that should not be over looked is host based IDS systems. Often exploits create anomalies in the log files which can be watched for and reported, possibly even before the attacker explores enough of the compromised system to discover the alarm message has been sent. Attackers can modify logs, change configuration files, or replace system binaries with trojaned "rootkits". 

Such changes can be detected with programs like tripwire, chkrootkit and rkhunter. Tripwire keeps a database of the hashes of the system files it protects. Chkrootkit and rkhunter act similar to network based IDS systems except they scan the system files for suspicious names, rootkit strings and configuration changes. However, these also are reliant on signatures and thus vulnerable to mutations of the known rootkits.