Investigating Our Technology — Internet of Things or Internet of Threats?

One cold winter afternoon as I sat in my office, cursing the air several degrees warmer around me due to slow internet connectivity, I thought to take a look at exactly the issue was. I had recently installed a new system of wireless access points which should be blanketing the entire house with a strong enough signal to make the air glow well out into the yard.

I logged into the controller for the APs, which helpfully provided all manner of statistics regarding the different devices connected, who was connected to to which AP, how the bandwidth was being used, and a dizzying array of other statistics (one of the selling points of these devices to a data nerd). Being a relatively technology-prone household, I was reasonably sure I would find the internet connection being choked out by several devices streaming at once, some large download running, or something similar, and would castigate the offender and perhaps adjust the rate limiting rules a bit to keep things a bit more sane in the future.

On digging into the data, I found an unfamiliarly-named device sending and receiving unusually large amounts of data almost continuously. Curious, I copied the MAC address from the wireless controller and looked it up on the internet  — Ubiquiti Networks Inc. This was one of the wireless video cameras set up to watch the house. Uh oh.

The IoT Problem

Why uh oh? In the fall of 2016, compliments of the Mirai botnet, hundreds of thousands of compromised Internet of Things (IoT) devices were used in distributed denial of service attacks against a variety of targets. These attacks were sufficiently powered to take down large swaths of the internet and even cause internet access issues across entire countries.

The issue that Mirai, and surely others to follow, took advantage of is that many of these IoT devices ship with a default set of credentials that is common across all devices of a model, or even all devices of a manufacturer. Mirai searches for these devices and recruits them into its botnet, using them to attack other targets on the internet.

This sort of widespread issue is rife for abuse, and this is exactly what happened. Worse yet, the source code for Mirai was publically released. Now we have an example of a very successful DDoS attack, source code for the attack’s tool, and a huge number of unpatched devices.

How huge? Some estimates say that in 2016 there were slightly less than 18 billion IoT devices. Yes, that’s billion with a b. Now we begin to see where the problem lies.

Worse yet, the DDoS attacks that were carried out using Mirai only had a bit over 100,000 devices to work with, according to some estimates. This is a small fraction of the devices out there and had a very significant impact.

A Little Research

The first stop in figuring out if my overly-chatty device was to do a bit of research. Not all IoT devices are problematic, and the list of devices being co-opted by Mirai and its variants was a relatively short one. From a little searching on the topic, there are a couple ways to go about searching for which devices are problematic; we can try interrogating the network for devices that that have issue, hoping to find them directly, or we can research a bit to find out if we might have a problem based on the particular hardware in general.

From a purely research perspective, finding the devices that might be problematic isn’t terribly difficult. A bit of brief googling turned up a list from the Krebs on Security website that were targeted by Mirai. The list did turn up a device from the same vendor as mine, but not the exact same device. The listed hardware did, however, have the same default credentials as mine. I always change the credentials on my devices when I stand them up, so this shouldn’t be a problem.

No really, I’m totally positive I changed them. I was standing several up at a time, surely I changed them all? Uh oh?

The next research bit before diving into the devices would be try out a couple of the multitude of IoT scanners that have popped up, all claiming to be able to suss out problem devices. After a bit of googling, I decided to try a couple of them; one a web-based scanner and one a script.

The web-based scanner was from Imperva, a well known security tool company. It has a simple ‘press go’ interface and automatically scans the address you are browsing from.

graphic-IoT-image1.png

Its results, however, are not what I would call conclusive:

graphic-IoT-image2.png

I take a certain amount of umbrage to statements, like the above, which imply that the absence of information means that the results are negative. Well, perhaps the next scanner will be better.

The second scanner, also from a well-known security company, is a script-based tool from Rapid7[1] and hosted on GitHub. This is a little more my speed. Furthermore, the tool is written in Perl, so cracking it open was like running into an old friend. Running the tool required a quick download of a few modules from CPAN, and away we go.

graphic-IoT-image3.png

Unfortunately, no great results here either. After poking about in the configuration file that the script uses, it turns out that it was looking for devices from a specific list of vendors with the default administrative credentials set.

One of the devices in question was indeed still using the default login and password, but the script failed to figure out what the device was due to looking for a very specific bit of text in the source of the login screen. Nonetheless, I had a device on my network that very neatly fit the pattern for being abused by an IoT botnet, and this bore further investigation.

Investigating

There are a number of routes that we can take to investigate a specific device and see what it is up to. In the absence of historical log data, we can scan it externally, try to get onto the device itself and see what we can see, and we can look at the traffic inbound to and outbound from the device.

Nmap

One of the easiest and most informative tools that we can use to look at our device is Nmap. We can do a quick nmap -A against the device, which will give us OS detection, version detection, script scanning, and traceroute, resulting in:

graphic-IoT-image4.png

So here we can see the expected web server, running lighthttpd on ports 80 and 443, rtsp running on port 554, presumably to support streaming video from the device, and Dropbear running an ssh server. One thing to notice here is the lack of a Telnet server immediately visible on port 23. It could be on another port, but this does put us out of the immediately dangerous spot as far as Mirai and other variants go.

Internally

If we can get on the system itself (where system = tiny golfball-sized IP camera running linux), we should be able to interrogate it and find out exactly what it is that is using up so much bandwidth.

SSH

Since we saw an SSH server running from the nmap scan, we can try to connect to that. Using the still default vendor credentials of ubnt for a username and password we can ssh ubnt@10.0.0.197 and voilà:

graphic-IoT-image5.png

So here we have a standard BusyBox linux, common to many embedded systems. This gives us a somewhat limited but common set of tools that we can use to check things out a bit further.

Netstat

First and foremost, we can make use of netsat. Running netstat -anputw will give us a solid idea of who we are presently talking to. These options will list all UDP (u), TCP (t) and RAW (w)  connections on all sockets (a) with numeric hosts (n) and includes the associated program (p):

graphic-IoT-image6.png

Nothing much of interest here. We can see the SSH connection from 10.0.0.117 and the various camera specific connections to the recording back end on 10.0.0.197. Nothing unusual talking or listening.

Poking About the System

While logged in, we have the opportunity to poke around the system in general. This device is a very typical stripped-down linux BusyBox. In /var/log/ there are system logs, as we might expect, which are largely full of the camera specific bits doing camera-specific things. We can also look at dmesg and learn a bit more about the specific hardware and environment:

graphic-IoT-image7.png

One interesting thing to discover here, for gee whiz purposes, is that the system appears to be running on OpenWrt, which is somewhat cool for a consumer IP camera. Overall, nothing too interesting or unusual.

It is worth noting, for purposes of looking for signs of Mirai, that there would also not be anything here. Mirai maintains a connection through the open telnet server and does not, at this point, have a mechanism for persistence. It is also, however, worth noting, that putting such a mechanism in place would be fairly trivial and we should expect its arrival/discovery at any point.

Examining the Traffic

There are a couple bits that we can look at for purposes of examining the specific traffic externally to the device. We can look at packet captures and we can look at netflow data.

Packet Captures

In this case, I asked the pfSense box to pull a packet capture of all traffic destined to or from our questionable camera. Since the camera was attached to a wireless network and, from an infrastructure perspective, it would have been a bit of a pain to get monitoring in place an isolate traffic out for the particular device, this was by far the easiest route.

In our case, unfortunately (or fortunately, rather), nothing of interest. While this doesn’t mean that there would never be anything of interest, it does mean that the device is not spending all of its time actively attacking and/or communicating with odd devices on the internet.

graphic-IoT-image8.png


NetFlow

In order to get a better idea of what exactly the device is doing, we can also look at the NetFlow data. We can pull NetfFow from a variety of places on the network. NetFlow was originally developed by Cisco and gives us  variety of information about the source and destination traffic on the network.

In my case, I have a pfSense firewall and can make use of the softflowd package to collect the netflow data. From there, I need to send it somewhere to aggregate and analyze the data. There are a huge number of tools available for this task, many of them commercial and either free or with a free trial period. I do, however like to use open source tools whenever possible, so we’ll go with ntop.

After configuring pfSense to send the data and analyzing it with nopt, again, a whole lot of… nothing. This is not unexpected after the data from the packet capture, but it does give us a greater level of detail on the nothing that we previously saw. Graphically detailed nothing is, on occasion, very useful when presenting investigative results to management, so this is a worthwhile exercise in learning how to collect data in this format.

Given a much longer period of time in which to collect the netflow data, we would likely see the odd unexpected connection occur, possibly for firmware updates or a phone home to the vendor. Nonetheless, we again have confirmation that the device is not doing anything terribly unexpected.

Conclusion

Ultimately, my device ended up not doing anything terribly suspicious, but I feel better for giving it the once over. The large amount of traffic that I was seeing was likely rtsp streaming back to the Network Video Recording (NVR) software on one of the other systems on my network.

Yes, it was overly chatty, yes it was bogging down the network, no it was not a compromised device trying to take down Amazon. I was reasonably sure that this was the case relatively early on, but working through the process and tooling to run down suspicious activity is a valuable exercise in and of itself.

The IoT world is a very dark and scary place with twisty and crooked little paths through it. We can find internet enabled anything, from toasters, to light bulbs, to cameras, and, being a relatively new phenomenon, the security around these devices is practically nil.

While the IoT security problem is clearly bigger than a breadbox, and we can expect to see DDoS attacks and the like from swarms of these devices for the foreseeable future, these issues do bring to mind the growing pains of another similar set of devices — wireless access points.

The early days of wireless access points saw them shipping with common default passwords, disabled encryption settings, and firmware rife with vulnerabilities, manufacturer backdoors, and other such bugaboos. While it did take some time for the industry to mature sufficiently for these problems to be mitigated across the majority of vendors, it did eventually happen. Now all we need to do is hold on long enough for this to happen for IoT, and hope the wheels don’t come off in the meanwhile.

[https://blog.komand.com/investigating-our-technology-internet-of-things-or-internet-of-threats]

Incident Investigation: It’s All About Context

When security operations centers or security teams have data output from our security devices or from threat intelligence sources, it all too often lacks any sort of reasonable context on which to base an investigation.

When we have Indicators of Compromise (IoCs) that define a particular type of attack, often largely IP addresses and file hashes, this can make a very difficult starting place; inefficient at best, paralyzing at worst. Data with no intelligence lacks context and we need context in order to succeed. Let’s use a real world example to demonstrate this.

Grizzly Steppe

A somewhat recent example that we can look to for contextual issues can be seen in the attacks that occurred during the 2016 US election, which came to be known by the code name Grizzly Steppe. These attacks, purported to have been carried out by the Russian Intelligence Services (RIS), attempted to influence the course of the election through exfiltration and exposure of sensitive information, undermining of confidence in the election process, and disruption of critical infrastructure, all backed by a variety of phishing attacks, malware, stolen credentials, and a host of other means.

In an effort to combat these attacks, the US Department of Homeland Security (DHS) and the Office of the Director of National Intelligence (DNI), released a Joint Analysis Report (JAR) in December of 2016. Accompanying the report was a file containing IoCs that might be used to detect the presence of the Grizzly Steppe attack groups, Fancy Bear (APT28) and Cozy Bear (APT29), in a given enterprise.

The Data, As Released

The IoC file contained:

  •      1 URL
  •      10 FQDNs
  •      876 IPv4 IP addresses
    • 1 YARA rule
  • 24 file hashes, 7 with accompanying file information

Unfortunately, much of the information was very general, and a large portion of the included IPs pointed to the address space of large cloud companies such as Google, Yahoo, and Twitter. While this is not necessarily incorrect, as the attackers were indeed making use of these services, it is not useful from an investigative standpoint.

If we were to query a security device, such as a Security Information Event Monitoring (SIEM) tool, with these IoCs, we would be returned an enormous number of results in most environments, as we would be asking for information about very common user activity. In particular, the results in the Grizzly Steppe IoC was so general that it would return literally millions of results when querying the common set of security devices in a large enterprise environment.

Given this, is there anything we can do to get to a place where we have something that our Incident Response team can actually sink their teeth into? Fortunately, yes there is.

Gaining a Little Context

In order to get something useful from the general information that we have, we can do a bit of analysis on it using various tools. Many of these analysis tasks are very similar in nature, so one we have tooling for our efforts, we can continue to reuse and refine it whenever we have reason to apply it again. Here we will be focusing on the IP addresses and file hashes.

IP Addresses

We can query Shodan about the IP information that we have, making use of the command line client. Let’s take a quick look at the first IP:

Shodan host 167.114.35.70
ir-context-image01.png

We can see a bit of basic information about that one — nothing much interesting there, but let’s look at all the rest of them. For this, we’ll want to use a script, and there are a host of existing examples that we can make use of. We’ll pull the IPs out of the IoC file into their own text file called hosts, and use a script from John Matherly, the developer of Shodan, to process the IPs and download the information:

python ./shodan-ip-download.py hosts hosts.json.gz

ir-context-image02.png

We will end up with a json formatted file, suitable for parsing with the Shodan command line tool, like so:

shodan parse –separator , hosts.json.gz

This will give us a CSV file which we can then browse at our leisure.

Upon doing so, one of the interesting points of information is that many of the IP addresses, about 20%, are Tor exit nodes.

File Hashes

To handle the file hashes, we can likely submit these to an anti-malware platform in our environment. However, it might be nice to know a bit more about them. After all, if the files listed in the IoC are malware, we’ll likely see new variants of them shortly, and it might be nice to have some idea what family we should be looking out for in general. For this, we can turn to VirusTotal.

For the very small number of files that we are dealing with, 24 as we mentioned earlier, we could just bang them through the VirusTotal search field, returning some interesting commentary:

ir-context-image03.png

This works, but doesn’t scale well. Fortunately, Didier Stevens has a Python tool to help us out with this very problem. We can provide the script with a list of all of the files hashes from the IoC file that we have, in a text file called samples, and it will dutifully ask VirusTotal (we’ll need an API key for this) about them, which will ultimately provide a CSV file back to us with all of the results:

./virustotal-search.py samples

ir-context-image04.png

Where do we go now that we have a bit more specific data? Now we go investigate a bit. We have a much more specific list of IP addresses to work with. Presuming that the use of Tor is not common in our environment, this is a good place to start.

From there, we can see what else the systems using Tor have been touching internally and investigate further. We also have a good set of malware, and potentially variants, to investigate. Both of these items are specific enough that we may get a more manageable set of results out of querying our SIEM.

Once we have a good look at the systems that have presented themselves as being outliers among normal system activity immediately, by communicating via Tor or displaying malware-like behaviors, in addition to catching the bad guys, we can also use these results to help refine our security tools. IDSs can be tuned, YARA signatures added to our anti-malware devices, and better querying and altering configured on our SIEMs. As we discussed earlier, through developing better tools for investigation, we ease the path for ourselves the next time we need to do this.

Conclusion

In our daily duties as security professionals, we need context in order to be successful. As the old saying goes, “garbage in, garbage out”. Fortunately, with a little elbow grease, some command line kung-fu, and just a quantum of clever, we can often get to a place of better context by processing the data through a few simple tools. Like a magician’s trick, it doesn’t seem like much once we know how it was done, but these are the tactics on which much of the security industry survives.

The next time you find yourself facing down a mountain of log file or a pile of useless-looking data, get a fresh coffee, roll up your sleeves, and see where you can get by crunching the data a bit to gain some context and hone in on the interesting bits.

[Originally posted to https://blog.komand.com/incident-investigation-its-all-about-context]

Powershell – Dump AD users to CSV

Dump all active users in a particular AD group to a CSV, with headers:

<code>get-aduser -filter * -properties * | where {$_.memberof -like “*ADgroupgoeshere*” -and $_.enabled -eq $true} | select @{name=”emailAddress”;expression={$($_.emailAddress)}},@{name=”firstName”;expression={$($_.givenname)}},@{name=”lastName”;expression={$($_.surname)}} | export-csv c:\scripts\users.csv -notypeinformation</code>