blueblog - by Christian J. Dietrich - Botnets

Performance Profiling Analysis using perf

nospam@example.com (Christian J. Dietrich) — Mon, 08 Apr 2013 22:22:00 +0200

While most of the stuff we analyze is Windows malware, when it comes to implementing detection or analysis approaches, we surely turn to GNU/Linux. One of the best tools I stumbled upon when it comes to profiling, i.e. analyzing the execution performance of C code under Linux is perf. Since most of the time we have to develop code that has to run fast, especially when dealing with carrier-grade network links of 10 GbE, profiling is inevitable. perf is extremely useful as it barely adds any overhead, is easy to use and precise. For example, in order to measure while running a userspace application, perf is called as easy as:



perf record -g ./application [params and args for the application]

On a multicore system, the performance overhead induced by perf while monitoring a single-threaded application is hardly noticeable. But the real magic lies within the simple yet powerful result reporing. In order to get a histogram of time spent in each function, it is as simple as



perf report

The first column shows the relative amount of time spent in the respective function. At the top, the functions where most computing time is consumed are displayed. This provides the developer with a quick view on the functions which potentially profit most from code optimizations. Thus, as shown in the example screenshot above I would dig into the libmagic stuff first. The console user interface is interactive and allows to navigate and dig down tree-structured call chains. In addition, it can annotate the assembly using the surrounding C source code, as shown here (simply press 'a'):

Finally, perf can print an annotated call graph snippet using

perf report -g

Note that the code should be compiled with -fno-omit-frame-pointer and -ggdb. To me, one of the results clearly indicated to use tcmalloc instead of the standard glibc malloc. I can really recommend profiling with perf.

Tracking the Command and Control Activity of Botnets

nospam@example.com (Christian J. Dietrich) — Wed, 13 Feb 2013 22:22:00 +0100

As part of our research on botnets, we developed recognition techniques for botnet command and control flows, such as CoCoSpot. Obviously, we use these techniques to track C&C channels and their activities. Throughout our analysis period of more than three years, we have seen several botnets come and go. Some botnets have faced dedicated takedowns, such as Rustock, Mariposa, Mega-D, Kelihos and Pushdo, while others cease without further ado.

By help of our tracking means, we are able to classify the activity of C&C channels. The following figure graphs the activity of some 25+ prevalent botnets in terms of consecutive C&C activity by family, as seen and covered by our tracking means, as well as public attention. The x-axis reflects the time period since February, 1st, 2010 until February, 13th, 2013, while the y-axis lists botnet families. A star depicts a dedicated takedown action. Note that, in case of Mariposa and Mega-D, the takedown actions have taken place before the beginning of the time period in this graph. The Mariposa takedown has occurred on December 23rd, 2009, and Mega-D has been taken down in November 2009. In these cases, the stars are placed on the start of the time period in order to visualize the preceding takedown. A thin black line with black markers represents the time period where new binaries are distributed, but none of the binaries exhibit an active C&C channel. Inactive C&C channels are caused, for example, by outdated binaries, unreachable C&C servers or sinkholing (possibly after successful takedowns). The Rustock takedown operation b107 by Microsoft's DCU is an example of such a successful takedown. Similarly, the Harnig botnet, which is believed to have been the main distributor of Rustock (most likely via pay-per-install), exhibits inactive C&C since March, 2011. This correlates to the time of the Rustock takedown operation. Coincidence?

C&C activity of botnets over time

In contrast to inactive C&C, a thick red bar represents the time periods where active C&C communication has been observed, and this is where things get really interesting. Every time I look at this graph, I am astonished by the fact that some botnets manage to operate successfully since several years. Cutwail, Virut, Sality, Palevo and Lethic are mere examples for such long-living botnets. However, their strategies for continued operation differ. While Sality has had a builtin peer-to-peer (P2P) component for years, Virut has experienced the comfort of using just a handful of domains in order to contact the C&C server. In case the domains cannot be resolved, a domain generation algorithm (DGA) kicks in. However, only recently, a takedown operation addressed Virut. Palevo seems to keep it simple, neither P2P nor DGA nor significantly low TTLs on the DNS responses (which could indicate fast flux). Instead Palevo seems to rely on new and migrating domains. Similar to Palevo, Lethic manages to bootstrap its C&C by plain old DNS resolution. These are just some examples of long-lasting botnets with active C&C, most of which have a centralized primary C&C architecture. In contrast to botnets with centralized C&C, several peer-to-peer botnets have emerged. Storm, Miner, ZeroAccess, Kelihos and Zeus P2P exhibit a primary C&C which is based on a P2P network, probably motivated by increased resilience. While Storm and Miner ceased, ZeroAccess, Kelihos and Zeus P2P show significant C&C activity — an observation also covered in our yet-to-be-published research paper on P2P botnet resilience.

On the sustain of botnet takedowns
What do we learn from this? On the one hand, as can be seen in the figure, the takedowns of the Bredolab as well as the Rustock botnet have a long-lasting effect. Although for Bredolab and Mega-D we witness active C&C during up to several months after the takedowns, none of these botnets manage to achieve active C&C communication in the long run. On the other hand, we have seen quite a few botnet families resurrect from takedowns. For example, researchers initiated a takedown of the Pushdo botnet in August 2010. However, even two years after the takedown, we still observe active executions of Pushdo. The Mariposa botnet is believed to have been taken down in December 2009, but we have seen active Mariposa command and control traffic ever since. Similarly, although the Nitol botnets have been taken down in September 2012, many Nitol binaries manage to successfully bootstrap and reach a viable C&C server, because they do not rely on the suspended domains such as 3322.org. In addition, the Tedroo botnet has been addressed in a takedown action in July, 2012. However, again, we observed active C&C communication of the same botnet family just within days after the takedown, continuing for at least three months. Thus, from an observational point of view, the challenge of botnet family takedowns lies in their sustain.

Some botnets even cease without a dedicated (publicly known) takedown of the C&C infrastructure. For example, the Miner botnet has not been addressed in a dedicated takedown operation of its peer-to-peer-based C&C, but its activity diminished significantly after October 2011. Similarly, the Renos botnet has ceased its operation, although possibly after removal signatures of the Renos binaries have been distributed by Microsoft as part of its Removal Tool MSRT. It is an open question as to whether the criminals behind these botnets just moved on to the next botnet. Most notably, the authors of Kelihos/Hlux (some consider Kelihos the successor of Waledac) stick to their botnet and change aspects such as the C&C encryption when faced with a takedown.

While many of the botnets mentioned above have been subject to takedown or sinkholing, it becomes obvious that a successful takedown is by far not an easy process. Bot masters distribute their infrastructure so that a takedown operation requires many different parties to cooperate — a challenging task.

Remarks
Finally, just a couple of remarks: Note that the monitoring time period has been interrupted by maintenance periods, from mid-February to mid-March 2011 as well as end-May to mid-July 2012. Furthermore, the per-family perspective does not always reflect all activity of the correspondings botnets; however, it can be considered a lower estimate. Especially since some families exhibit (lots of) distinct botnets, we certainly do not cover every unique botnet of that particular family. For example, in case of toolkit-based families such as Zeus and SpyEye, several distinct botnets may be formed. In our C&C activity tracking, we restrict ourselves to whole families instead of individual botnets on purpose.

Exploiting Visual Appearance to Cluster and Detect Rogue Software

nospam@example.com (Christian J. Dietrich) — Fri, 07 Dec 2012 13:37:00 +0100

While malware comes in many different flavors, e.g., spam bots, banking trojans or denial-of-service bots, one important monetization technique of recent years is rogue software, such as fake antivirus software (Fake A/V). In this case, the user is tricked into spending money for a rogue software which, in fact, does not aim at fulfilling the promised task. Instead, the rogue software is malicious, might not even have any legitimate functionality at all, and entices the user to pay. However, all rogue software has in common to provide a user interface, e.g., be it to scare the user, or in order to ask for banking credentials, or to carry out the payment process. The following figures show example screenshots of two typical rogue software flavors.

The first displays a Fake A/V user interface, mimicking a benign antivirus application, while the second exhibits a ransom screen (in German), asking the user to pay before the computer is unlocked (in fact the computer appears just visually locked). Especially the later category, ransomware, is considered an increasing threat with more than 120,000 new ransomware binaries in the second quarter of 2012. In addition, referring to the C&C tracking of Fake A/V and ransomware botnets, we observe a significant increase in activity since June 2011.

As rogue software is required to provide a user interface, we aim at exploiting its visual appearance in order to cluster and classify rogue software. We motivate our efforts by the relatively low A/V detection rates of such rogue software, and we aim to complement existing techniques to strive for better detection rates. In particular, we observed that the structure of the user interfaces of rogue software remains constant and can be used to recognize a rogue software family or campaign. Using a perceptual hash function and a hierarchical clustering approach, Christian Rossow and I, we propose a scalable and effective approach to cluster associated screenshot images of malware samples.
In short, the main contributions of our work are threefold:

We provide a scalable method to cluster and classify rogue software based on its user interface, an inherent property of rogue visual malware.

We applied our method to a corpus of more than 187,560 malware samples of more than 2,000 distinct families (based on Microsoft A/V labels) and revealed 25 distinct types of rogue software user interfaces. Our method successfully reduces the amount of more than 187,560 malware samples and their associated screenshot images down to a set of human-manageable size, which assists a human analyst in understanding and combating Fake A/V and ransomware.

We provide insights into Fake A/V and ransomware campaigns as well as their payment means. More specifically, we show a clear distinction of payment methods between Fake A/V and ransomware campaigns.

As an example and in order to underline the usefulness of our approach, we used the our visual clustering to enumerate the campaigns that can be attributed to the Winwebsec family. Note the visual similarity among the different campaigns, sometimes it is even difficult to spot the difference. Typically, the campaigns differ in names as well as logos.

I am delighted to announce that our paper was accepted for the security track of ACM's 28th Symposium On Applied Computing 2013. A preprint of our manuscript for ACM SAC 2013 is available here "Exploiting Visual Appearance to Cluster and Detect Rogue Software".

CoCoSpot - Recognizing Botnet C&C Channels using Traffic Analysis

nospam@example.com (Christian J. Dietrich) — Wed, 24 Oct 2012 15:48:22 +0200

A defining characteristic of a bot is its ability to be remote-controlled by way of command and control (C&C). Typically, a bot receives commands from its master, performs tasks and reports back on the execution results. All communication between a C&C server and a bot is performed using a specific C&C protocol over a certain C&C channel. Consequently, in order to instruct and control their bots, bot masters ‐ knowingly or not ‐ have to define and use a certain command and control protocol.

Plaintext C&C

Historically, bots used cleartext C&C protocols, such as plaintext messages transmitted using IRC or HTTP, as shown in the Rbot C&C example above. However, a C&C channel relying on a plaintext protocol can easily be detected. Methods such as payload byte signatures or heuristics on common C&C message elements such as IRC nicknames are examples for such detection techniques. To evade payload-based detection, botnets have evolved and often employ C&C protocols with obfuscated or encrypted messages as is the case with Waledac, Zeus, Hlux, TDSS/Alureon, Palevo, Renos, Virut and Feederbot, to name but a few. In fact, pretty much all recent botnets employ some kind of encryption in their C&C protocol. The following two images show an encrypted Virut C&C message and its decrypted plaintext. This reveals that the underlying carrier protocol is still IRC, or IRC-like. The encryption is basically a four-byte XOR with a random bot-chosen key.

Encrypted Virut C&C

Decrypted Virut C&C message

The change towards encrypted or obfuscated C&C messages effectively prevents detection approaches that rely on plaintext C&C message contents. Lately, we take a different approach to recognize C&C channels of botnets and fingerprint botnet C&C channels based on traffic analysis properties. The rationale behind our methodology is that for a variety of botnets, characteristics of their C&C protocol manifest in the C&C communication behavior. For this reason, our recognition approach is solely based on traffic analysis.

As an example, consider a C&C protocol that defines a specific handshake – e.g., for mutual authentication – to be performed in the beginning of each C&C connection. Each request and response exchanged during this handshake procedure conforms to a predefined structure and length, which in turn leads to a characteristic sequence of message lengths. In fact, we found that in the context of botnet C&C, the sequence of message lengths is a well-working example for traffic analysis features. The following figure shows the sequence of the first 8 messages in four Virut C&C flows and two Palevo/Rimecud/Pilleuz C&C flows. Whereas Virut exhibits similar message lengths for the first message (in the range 60-69) and a typical sequence of message lengths at positions five to eight, for Palevo, the first three message lengths provide a characteristic fingerprint.

Examples of message length sequences for Virut and Palevo C&C flows

Leveraging statistical protocol analysis and hierarchical clustering analysis, we develop CoCoSpot, a method to group similar botnet C&C channels and derive fingerprints of C&C channels based on the message length sequence, the underlying carrier protocol and encoding properties. The huge benefit of our approach is to be independent from payload byte signatures which enables the detection of C&C protocols with obfuscated and encrypted message contents. In addition, our C&C flow fingerprints complement existing detection approaches while allowing for finer granularity compared to IP address or domain blacklists. As a side-effect, our C&C flow clustering can be used to discover relationships between malware families, based on the distance of their C&C protocols. Experiments with more than 87,000 C&C flows as well as over 1.2 million Non-C&C flows have shown that our classification method can reliably detect C&C flows for a variety of recent botnets with very few false positives.

CoCoSpot classification evaluation results

The figure above shows the results of the CoCoSpot classification as a cumulative distribution function for all families that were included in both, true positive and false positive analysis. Note that the true positive rate values are given on the left y-axis, the false positive rate values on the right y-axis. More than half of all families have a true positive rate of over 95.6%. A small fraction of seven C&C families had true positives rates lower than 50%, which was caused by too specific cluster centroids. In most of these cases, we had too little training data to learn representative message length variations of a particular active C&C protocol, which could be improved by adding more training data for this family. For 88% of the families, the false positive rate is below 0.1%, and 23 cluster families do not exhibit any false positive at all. For the few families that cause false positives, we observed that the corresponding cluster centroids have high variation coefficients on many message length positions, effectively rendering the centroids being too generic and possibly matching random flow patterns.

I am happy to announce that the manuscript of our journal article on CoCoSpot has been accepted by Elsevier's Computer Networks journal in the special issue on Botnets. The published version of our manuscript is available from Sciencedirect.

Feederbot - a bot using DNS as carrier for its C&C

nospam@example.com (Christian J. Dietrich) — Fri, 02 Sep 2011 18:05:00 +0200

DNS as carrier for botnet C&C seems to be getting popular. Concerning its usage as botnet C&C, DNS has not been seen so far. Additionally, in typical network environments, DNS (at least when destined for the preconfigured DNS resolvers) is usually one of the few protocols – if not the only one – that is allowed to pass without further ado. Thus, botnets using DNS as C&C benefit from the fact that currently there is no specifically tailored detection mechanism, which in turn, raises the probability for the botnet to remain undetected.

During our work on covert communication of botnet command and control channels, we analyzed Feederbot in some detail and monitored it over the last year. In this post, I will provide some insight on the C&C.
Not only Feederbot, but also Morto seems to be using DNS as carrier for its command and control channel.

But let us focus on Feederbot for now. Feederbot uses valid DNS syntax for its DNS messages. Messages from the C&C server to the bot are transmitted in the rdata field of a TXT resource record. The DNS requests have the several different schemes for the question domain name (qname), similar to the following where [CHUNK-ID] is an int >= 0, incremented by 1:



[50 bytes].[CHUNK-ID].[qdparam].0.f2.[TLD].   IN   TXT

The DNS responses typically carry one TXT RR in the answer section (sometimes repeated in the authority section) with a 220 byte string that is base64 encoded. Here is an example:



xMtwHYRyZu/z4QbhBKZIVWvPBfiuGn+jb1WQxtZN7PR9Wf0sfnAqxDOJD9LgmwfFaU

Go6fdtgZ0lIQyAx1VWJw+vzdHdxMpHu6xfMRq8sVSfqwPvI9TEIV8pkXw4P4TCSH05

BAO1LGPMQ+XD+TYLY2woxM1j06mCMhrNjWzI8WbmCBlj2/dpR73KBnDl/DRmheKWMJ

x2dUTp4iFMH4N9kXjeOYis

Once base64 decoded, the messages are still no real plaintext, because they are encrypted with RC4. Feederbot uses a variety of different RC4 encryption keys and even stacks RC4 encryption. A specific part of the DNS query domain name is used to transmit parameters for key derivation. As an example, one such parametrized key derivation function takes as input a substring of the query domain name, denoted as 'qdparam' in the example above. The value of the substring 'qdparam' is then RC4-encrypted with the (constant) string “feedme” (hence the name of the bot) and the result is used to initialize the RC4 decryption of the actual C&C message chunks. The stream cipher is used in a stateful manner, so that if a message chunk gets lost, decryption of all subsequent message chunks will fail. In addition, Feederbot’s C&C message chunks make use of cyclic redundancy checks to verify the decryption result. The CRC32 checksum preceeds message chunk payload and is not encrypted.

The fact that CRC32 checksums are used makes it comfortable to know whether decryption works or not. Interestingly, we have seen ANY as resource record type in some of the queries, too. In order to perform the DNS requests, the bot relies on Windows DNSAPI.dll::DnsQuery_W.

The following figure shows an important part of the disassembled RC4 initialization routine:

Well, the drawback of encryption is that you need a key and you better choose one that is easy to remember, such as:

So, what is the lesson we learn from Feederbot? Watch your DNS traffic!

DNS as carrier for botnet C&C

nospam@example.com (Christian J. Dietrich) — Mon, 22 Aug 2011 17:49:00 +0200

Botnets have become one of the biggest security issues on the Internet imposing a variety of threats to Internet users. Advances in malware research have challenged botnet operators to improve the resilience of their C&C traffic. Partly, this has been achieved by moving towards decentralized structures (like P2P) or by otherwise obfuscating and even encrypting communication.

Recently, Christian Rossow and me, we looked into what we term covert communication, that is command and control communication which is hidden in what seems to be regular Internet traffic. We discovered and reverse engineered Feederbot, a botnet that uses DNS as carrier for its command and control. Using k-Means clustering and a Euclidean Distance based classifier, we correctly classified more than 14 million DNS transactions of 42,143 malware samples concerning DNS C&C usage. Interestingly, this analysis revealed yet another bot family with DNS C&C. In addition, we correctly detected DNS C&C in mixed office workstation network traffic.

Our paper dealing with DNS as carrier for botnet command and control channels got accepted at this year's EC2ND conference. I will be presenting the results at EC2ND which is going to take place in Gothenburg, Sweden, September 6-7 at Chalmers University.

Secure Network and Server Administration

nospam@example.com (Christian J. Dietrich) — Wed, 26 May 2010 15:35:00 +0200

Being a system administrator myself I have to say that secure system administration has become much more challenging in the last few years. With SSH bruteforce attacks hammering down on your servers and frequent software updates that need to be applied, administrators have a hard time keeping a server farm secure.

But not only fixing known vulnerabilities is important these days. In my experience as a malware and botnet researcher I have learned another important lesson: Know your network. Whenever there is an intrusion, it is surprising that more often, system administrators realize it by help of characteristic statistics of their network and its servers - especially if (for whatever reason) intrusion detection systems are not in use. Let me give an example:

Say, you work as a sysadmin of a corporate computer network. Say people are allowed to send email only via your own smarthost and your internet gateway thus blocks outbound TCP port 25 (SMTP). Do you count the SMTP connection attempts to outside servers (that your firewall blocks)? What is the typical number of blocked SMTP connection attempts (on a weekday)? Why could this be an interesting metric, you may wonder.

Let me explain. We analyze the network behavior of quite a bit of malware. It is not surprising that nowadays, lots of malware comes as a bot, i.e. a remote-controllable flexible piece of software that awaits instructions from a botmaster. Furthermore, neither is it a surprise that many bots are used to send spam. Ah, you get the point? Let us assume one of the computers on your network is infected. Then there is quite a bit of a chance that it is trying to send spam to outside mail servers. "But", you might say, "that spam is blocked, so there is no harm." Indeed, yes. But who says that the same piece of malware does not steal credentials or is involved in a click fraud or DDoS campaign? Even if you cannot do anything against it, it might be good to just know that there is an infected host in your network.

The number of (failed/blocked) connections attempts is not the only useful metric in this context. Based on my experience, I can recommend:

number of (failed/blocked) outbound connection attempts for TCP port 445 (infection attempts)

number of (failed/blocked) outbound SSH connection attempts for TCP port 22 (bruteforce scan attempts)

number of (failed/blocked) outbound DNS connection attempts (UDP port 53)

Depending on your environment, you may want to use a relative metric instead of absolute numbers, i.e. number of failed/block connection attempts over number of total connections.

Will Conficker destroy the world on April 1st?

nospam@example.com (Christian J. Dietrich) — Wed, 01 Apr 2009 00:25:00 +0200

Rumors are that one of the most widespread malware, Conficker (or Downadup) might strike on April 1st, 2009. I gave a short interview to the German TV station ARD that was partly broadcast as part of the ARD Mittagsmagazin (also broadcast on ZDF at the same time).

To me, there is no reason to wait for a specific date such as April 1st, for a malware to become active. This is a bit different if the malware itself spreads via email, such as Storm worm, and is completely based on social engineering. Furthermore, in my eyes, system administrators are fully aware of the danger that Conficker might pose, once its activated and thus look for it with special attention on April 1st. Any other date would then - from the attacker's point of view - make much more sense.

Safer Internet Day 2009

nospam@example.com (Christian J. Dietrich) — Wed, 11 Feb 2009 21:00:00 +0100

On the occasion of today's Safer Internet Day, Feb 11th, the German radio station SWR broadcast an interview of a couple of IT security researchers, including me.

The McColo story from the spam and botnet perspective

nospam@example.com (Christian J. Dietrich) — Tue, 25 Nov 2008 22:22:00 +0100

On Thursday 11/11/2008, the US company McColo (AS26780) got cut off the Internet. McColo has been known for some doubtful activities - some say that McColo is responsible for as much as 75% of all spam sent on the Internet. These activities have stopped instantly as McColo got disconnected. I looked into this at our blacklist mirror. Since Thursday evening (2200 local time CET), the total number of requests on the blacklist are much lower than on the previous days. The traffic that is caused by the requests has - compared to the peaks - nearly halved.

After McColo (AS26780) "went" offline on 11/11/2008, the global spam volume has remained for about 10 days at about half as much as before. The low volume which even hit an annual minimum on 11/21/2008 was probably amplified by the fact that a lot of command and control channels of botnets were hosted at McColo.

On 11/15/2008, I realized that McColo suddenly reappeared for short periods of time through other Autonomous Systems, such as TeliaNet Global Network, AS 1299). At the bottom of this post, I have added some screenshots of bgplay that show the changes in the routing to McColo.

By the way, on 11/21/2008 one of the largest distributed denial of service attacks became public. The attackers targeted the German Hosting company InternetX with more than 40,000 bots and 800,000 packets/second causing a total bandwidth of 20 GBit/s during peaks. It is difficult to say whether the attack was done using McColo-controlled botnets. Interestingly, the number of requests to the blacklist once more decreased heavily on 11/21/2008 and the following day.

This might have been caused by the fact, the those bots while participating in the DDoS attack have stopped spamming.