How well can network behavior anomaly detection (NBAD) products detect rootkits and malware?

Ask the Expert

How well can network behavior anomaly detection (NBAD) products detect rootkits and malware?

I've read that worms can now be tracked down by analyzing their connection/packet rates. Are these non-signature-based techniques effective and are they any different than network behavior anomaly detection (NBAD) products?

Continue Reading This Article

Enjoy this article as well as all of our content, including E-Guides, news, tips and more.

By submitting your email address, you agree to receive emails regarding relevant topic offers from TechTarget and its partners. You can withdraw your consent at any time. Contact TechTarget at 275 Grove Street, Newton, MA.

You also agree that your personal information may be transferred and processed in the United States, and that you have read and agree to the Terms of Use and the Privacy Policy.

Safe Harbor

Connection and packet-rate analysis is a subset of the overall approach known as network behavior anomaly detection (NBAD). Rootkits and other forms of malware have become so good at stealthily burrowing deep into end-user systems that organizations have come to rely on the help of network-based detection resources.

When systems are infected with malware, their communication patterns usually change in a detectable fashion. Consider this example:

Client machines usually talk with servers. Servers very seldom initiate a connection back to clients, except for occasional services like File Transfer Protocol (FTP) not used in passive mode. Also, clients almost never communicate with other clients, and servers have only a little communication with other servers. Hence you have a nice pattern that automated tools can check for.

When a worm or bot infection occurs, there is often a huge uptick in client-to-client session initiation. As you point out in your question, there might be a major rise in the bandwidth consumption of one or more infected machines. There also may be a hike in the number of connection initiation attempts. Each of these measurements is helpful and can be detected by various NBAD products. Network-based intrusion prevention systems, security information management (SIM) products, some intrusion detection systems, as well as distributed denial of service (DDoS) monitoring products all offer such capabilities.

Beyond these products, there are large-scale, Internet-based projects that look for network anomalies. One of the most prominent is the DShield project, administered by the SANS Internet Storm Center. This project has over 45,000 volunteer-operated sensors distributed around the Internet. The sensors gather data, make it anonymous and send it to collectors. Software and people then analyze the resulting information, which includes communicating sessions and the ports they use. The top 10 worldwide rising ports, as well as various unusual session activity, are plotted and updated every day on the DShield Web site.

More information:

  • Learn which security information management tools can spot network anomalies.
  • Compare signature detection with anomaly detection.
  • This was first published in June 2007