Even in the Quietest Moments ...

Even in the Quietest Moments ...
I recently had a migration from one internet uplink to another to do for a client. As with many organizations, they have about 40% of their workforce at head office, and 60% (and sometimes more) of their workforce operating remotely, so taking the Firewall and especially the VPN services offline is a very big deal. There is no good time to take things down given that their sales force has people in just about every time zone, there are just times that are "less bad" than others. Anyway, we settled on a weeknight, starting at around 6pm EST after their shipping was completed. As in most projects of this type, we finally got most of the "important" users off the network by about 7:30 and where ready to start. On logging into their firewall, I (as I always do), did a quick check of the real time logs, just because you never know what you'll see. Imagine my surprise when I saw that it was still pretty busy, and the traffic being logged was mostly this: So what is that? Their internal workstation network is 192.168.122.0/24, a nice respectable RFC1918 address range. However, we're seeing lots of outbound requests for 192.168.1.0/24, which I would expect to see more on home networks. Now I could see a simple, persistent series of requests to a single 192.168.1.x address, perhaps a home printer that is a user's default, or a home DLNA server, but this looks a whole lot like reconnaissance doesn't it? It's a sweep of the entire 192.168.1.0 network, using ICMP. Looking deeper at the packet, as you'd expect these are echo requests (Type 0, code 0), otherwise known as plain old "ping". So, what's so interesting about this you ask? My answer would be - recon of a network that's the default subnet for many home network routers is always suspect in an enterprise. In fact, recon of any type in an enterprise network should be considered suspect. People connect to what they need for to do their jobs, network sweeps are almost always an indicator of compromise, it's almost always malware looking for something else to infect, or an attacker looking for their next foothold. Happily, if the network being scanned isn't in the internal routing table, if it's not black-holed internally it usually shows up in the firewall logs. In this case, it was malware looking for PNP devices (printers, cameras, TVs, but especially home firewalls), which Johannes wrote up in January ( https://isc.sans.edu/diary/Exposed+UPNP+Devices/15040 ). Even though it's what you might consider "old", it sees sustained use ( https://isc.sans.edu/port.html?port=1900 ) in malware and sees continued success, mostly becuase almost nobody patches or fixes their home routers. It could also just as easily have been malware looking for default home router credentials or one of the home router backdoor vulnerabilities (which Manuel wrote up here https://isc.sans.edu/diary/Old+D-Link+routers+with+coded+backdoor/16802 ) It was just lucky that I caught this activity on film, the network was so quiet that the malware activity just popped up without looking for it. It's funny that when you have a quiet network, sometimes all that's left is the malware and attack traffic. Of course, when we did a reverse lookup on the workstation IP, it was their HR manager. You know, the HR manager who insisted that we remove the internet filters from their account so they could be active on social media? No risk at all having malware on that station !! But hopefully, this should illustrate why it's so important that part of your day-to-day tasks to secure your organization should be to look at your logs. And not just glancing at the logs scrolling by on your firewall - review the text logs that you store in disk, that's where you'll find those "gold" log entries. Make friends with the grep or findstr commands and do some mining for malware, it'll be the most productive time you spend most days! Heck, just looking at a directory of your logs by day is often enough - if yesterdays logs was 10 times the size you normally see, often that's an IOC (Indicator of Compromise) all on it's own. Relying simply on tools to alert you of problems and not looking at your logs is passing up the easiest and earliest detection of problems you'll ever get. As we say over and over - "there's gold in them thar logs!" Even reviewing the logs of your home network can give you valuable information ( https://isc.sans.edu/diary/Collecting+Logs+from+Security+Devices+at+Home/14614 ). You'll find the same indicators of compromise or attack in home logs as at work, and if nothing else, it's good practice for reviewing logs for enterprise gear - home gear and enterprise gear all log the same things, if they're configured correctly that is... By the way, did you catch the other problem that our captures showed? Yes, the NTP server that the firewall was synced to had been retired, so the date and time on the firewall was completely out of whack. If you've ever tried to correlate logs from multiple systems, perhaps to get a complete picture of how a compromise happened or what it did after it got it's toe-hold, you'll appreciate just how big a deal syncing your clocks is. If you've found malware with plain old text logs as a primary source or using a log analysis tool, please let us know using our comment form! =============== Rob VandenBrink Metafore	Rob VandenBrink 548 Posts ISC Handler Dec 3rd 2013
Thread locked Subscribe	Dec 3rd 2013 6 years ago
A couple of jobs ago I did a lot of work on the firewall logs, and found tons of malware, misconfigured systems, adware from browser helper objects, "funny behavior" (ie: every time you print to a certain printer, it opened a connection to a server on the internet). I found it all just by doing traffic analysis on the firewall and bluecoat logs. A lot of bad traffic I could find by just grouping source ip /destination ip/destination ports together and seeing what runs 24/7.	Kenneth 11 Posts
Quote	Dec 3rd 2013 6 years ago
Hi Rob, Thx for your diary, Don't remember checking icmp payload, few times it's interesting for finding what tool or another used... Regards @Rmkml	Rmkml 11 Posts
Quote	Dec 3rd 2013 6 years ago
Quoting comment#28616:Spam on the isc web site... Wow that's embarassing.... Update: Already erased. Nice...[/quote] Whenever I see something like that I remember two things: 1. The biggest part of my job is learning from the mistakes of others before they happen to us. Hopefully there will be an article about how it happened, how it was detected and how it will be prevented going forward. Because the two latter points are the important items. 2. "There but for the grace of God go I" because some day I will not have learned fast enough...	Anonymous
Quote	Dec 4th 2013 6 years ago
We reject about 100+ spam signups a day. Once in a while something slips through. In some cases they look like they are manually entered.	Johannes 4001 Posts ISC Handler
Quote	Dec 4th 2013 6 years ago
I work in an enterprise, so lots of devices. We have closed our firewalls for most outgoing traffic, including web. That has to go through a proxy. As a result, when I look at rejects on the inside FW, I see so much noise that it is impossible to get operations to clean the stuff up. It is printers, it is one machine talking to another on a decomissioned subnet, you name it, we got it. Even though our rules says to never route 192.168.x.y, we see traffic to those, and from those. Shows that anti-spoofing would make a difference even on an internal netwrk.	Povl H. 74 Posts
Quote	Dec 4th 2013 6 years ago
Yes, same here. The ops folks don't care about what their stuff is doing or how it's doing it as long as the system appears to be working and clueless managers don't get it either. Fairly clean attempted egress logs means it's a snap to see an infected or unauthorized machine. Just keep gently pushing because some day you'll need those CYA emails and meeting notes to demonstrate why your company was owned and you didn't notice in a timely manner. Nothing will change until managers adopt a performance goal of "no re-work" and they will go on arguing that they have better things to do than clean up the mess they never should have allowed in the first place. It's just another aspect of our chosen careers.	Anonymous
Quote	Dec 4th 2013 6 years ago
Quoting Povl H.:I work in an enterprise, so lots of devices. We have closed our firewalls for most outgoing traffic, including web. That has to go through a proxy. As a result, when I look at rejects on the inside FW, I see so much noise that it is impossible to get operations to clean the stuff up. It is printers, it is one machine talking to another on a decomissioned subnet, you name it, we got it.[/quote] Yup. I recently setup some snort rules to watch all outbound traffic from certain subnets (non-user nets) and it was moderately horrifying how much traffic there was that's just "normal noise". It also uncovered people who should know better surfing the web from servers (which is decidedly verboten here), lots of systems mis-configured and sending DNS queries to the wrong place, etc. The worst was the outbound UDP rule which showed a jillion or so reply packets from domain controllers going to various non-routable subnets in use at homes of VPN users (yeah, the original packet apparently didn't get tunneled properly as the source IP was not their VPN-assigned IP but the IP on their home network - sigh - something else for a TAC ticket one of these days). As another poster mentioned, though, keep letting people know that egress filters are important and keep implementing the ones you can. We pushed hard for at least blocking outbound DNS queries (except from our proxies) and that's not only helped prevent some malware from functioning but has made it easier to detect as well. With so many linux distros now using a full-blown dns proxy as a client (grumble) which are determined to try doing their own root server queries instead of just using IPs configured in resolv.conf I can't simply watch for and report every blocked outbound DNS query as a compromise, but I have a whole laundry list of known malicious DNS servers that we do watch firewall logs for (oak is a great tool for automagically watching logs for various patterns, BTW). And blocking outbound DNS also means people can't bypass our DNS filters by pointing at opendns or something silly.	Brent 123 Posts
Quote	Dec 4th 2013 6 years ago

I recently had a migration from one internet uplink to another to do for a client. As with many organizations, they have about 40% of their workforce at head office, and 60% (and sometimes more) of their workforce operating remotely, so taking the Firewall and especially the VPN services offline is a very big deal. There is no good time to take things down given that their sales force has people in just about every time zone, there are just times that are "less bad" than others.

Anyway, we settled on a weeknight, starting at around 6pm EST after their shipping was completed. As in most projects of this type, we finally got most of the "important" users off the network by about 7:30 and where ready to start.

On logging into their firewall, I (as I always do), did a quick check of the real time logs, just because you never know what you'll see. Imagine my surprise when I saw that it was still pretty busy, and the traffic being logged was mostly this:

So what is that? Their internal workstation network is 192.168.122.0/24, a nice respectable RFC1918 address range. However, we're seeing lots of outbound requests for 192.168.1.0/24, which I would expect to see more on home networks. Now I could see a simple, persistent series of requests to a single 192.168.1.x address, perhaps a home printer that is a user's default, or a home DLNA server, but this looks a whole lot like reconnaissance doesn't it? It's a sweep of the entire 192.168.1.0 network, using ICMP. Looking deeper at the packet, as you'd expect these are echo requests (Type 0, code 0), otherwise known as plain old "ping".

So, what's so interesting about this you ask?
My answer would be - recon of a network that's the default subnet for many home network routers is always suspect in an enterprise. In fact, recon of any type in an enterprise network should be considered suspect. People connect to what they need for to do their jobs, network sweeps are almost always an indicator of compromise, it's almost always malware looking for something else to infect, or an attacker looking for their next foothold. Happily, if the network being scanned isn't in the internal routing table, if it's not black-holed internally it usually shows up in the firewall logs.

In this case, it was malware looking for PNP devices (printers, cameras, TVs, but especially home firewalls), which Johannes wrote up in January ( https://isc.sans.edu/diary/Exposed+UPNP+Devices/15040 ). Even though it's what you might consider "old", it sees sustained use ( https://isc.sans.edu/port.html?port=1900 ) in malware and sees continued success, mostly becuase almost nobody patches or fixes their home routers.
It could also just as easily have been malware looking for default home router credentials or one of the home router backdoor vulnerabilities (which Manuel wrote up here https://isc.sans.edu/diary/Old+D-Link+routers+with+coded+backdoor/16802 )

It was just lucky that I caught this activity on film, the network was so quiet that the malware activity just popped up without looking for it. It's funny that when you have a quiet network, sometimes all that's left is the malware and attack traffic.

Of course, when we did a reverse lookup on the workstation IP, it was their HR manager. You know, the HR manager who insisted that we remove the internet filters from their account so they could be active on social media? No risk at all having malware on that station !!

But hopefully, this should illustrate why it's so important that part of your day-to-day tasks to secure your organization should be to look at your logs. And not just glancing at the logs scrolling by on your firewall - review the text logs that you store in disk, that's where you'll find those "gold" log entries. Make friends with the grep or findstr commands and do some mining for malware, it'll be the most productive time you spend most days! Heck, just looking at a directory of your logs by day is often enough - if yesterdays logs was 10 times the size you normally see, often that's an IOC (Indicator of Compromise) all on it's own. Relying simply on tools to alert you of problems and not looking at your logs is passing up the easiest and earliest detection of problems you'll ever get. As we say over and over - "there's gold in them thar logs!"

Even reviewing the logs of your home network can give you valuable information ( https://isc.sans.edu/diary/Collecting+Logs+from+Security+Devices+at+Home/14614 ). You'll find the same indicators of compromise or attack in home logs as at work, and if nothing else, it's good practice for reviewing logs for enterprise gear - home gear and enterprise gear all log the same things, if they're configured correctly that is...

By the way, did you catch the other problem that our captures showed? Yes, the NTP server that the firewall was synced to had been retired, so the date and time on the firewall was completely out of whack. If you've ever tried to correlate logs from multiple systems, perhaps to get a complete picture of how a compromise happened or what it did after it got it's toe-hold, you'll appreciate just how big a deal syncing your clocks is.

If you've found malware with plain old text logs as a primary source or using a log analysis tool, please let us know using our comment form!

===============
Rob VandenBrink
Metafore

Rob VandenBrink

548 Posts
ISC Handler

Dec 3rd 2013

A couple of jobs ago I did a lot of work on the firewall logs, and found tons of malware, misconfigured systems, adware from browser helper objects, "funny behavior" (ie: every time you print to a certain printer, it opened a connection to a server on the internet). I found it all just by doing traffic analysis on the firewall and bluecoat logs. A lot of bad traffic I could find by just grouping source ip /destination ip/destination ports together and seeing what runs 24/7.

Kenneth

11 Posts

Hi Rob,
Thx for your diary,
Don't remember checking icmp payload, few times it's interesting for finding what tool or another used...
Regards
@Rmkml

Rmkml

11 Posts

Quoting comment#28616:Spam on the isc web site... Wow that's embarassing....
Update: Already erased. Nice...[/quote]

Whenever I see something like that I remember two things:

1. The biggest part of my job is learning from the mistakes of others before they happen to us. Hopefully there will be an article about how it happened, how it was detected and how it will be prevented going forward. Because the two latter points are the important items.

2. "There but for the grace of God go I" because some day I will not have learned fast enough...

Anonymous

We reject about 100+ spam signups a day. Once in a while something slips through. In some cases they look like they are manually entered.

Johannes

4001 Posts
ISC Handler

I work in an enterprise, so lots of devices.
We have closed our firewalls for most outgoing traffic, including web. That has to go through a proxy.
As a result, when I look at rejects on the inside FW, I see so much noise that it is impossible to get operations to clean the stuff up. It is printers, it is one machine talking to another on a decomissioned subnet, you name it, we got it.

Even though our rules says to never route 192.168.x.y, we see traffic to those, and from those. Shows that anti-spoofing would make a difference even on an internal netwrk.

Povl H.

74 Posts

Yes, same here. The ops folks don't care about what their stuff is doing or how it's doing it as long as the system appears to be working and clueless managers don't get it either. Fairly clean attempted egress logs means it's a snap to see an infected or unauthorized machine. Just keep gently pushing because some day you'll need those CYA emails and meeting notes to demonstrate why your company was owned and you didn't notice in a timely manner. Nothing will change until managers adopt a performance goal of "no re-work" and they will go on arguing that they have better things to do than clean up the mess they never should have allowed in the first place. It's just another aspect of our chosen careers.

Anonymous

Quoting Povl H.:I work in an enterprise, so lots of devices.
We have closed our firewalls for most outgoing traffic, including web. That has to go through a proxy.
As a result, when I look at rejects on the inside FW, I see so much noise that it is impossible to get operations to clean the stuff up. It is printers, it is one machine talking to another on a decomissioned subnet, you name it, we got it.[/quote]

Yup. I recently setup some snort rules to watch all outbound traffic from certain subnets (non-user nets) and it was moderately horrifying how much traffic there was that's just "normal noise". It also uncovered people who should know better surfing the web from servers (which is decidedly verboten here), lots of systems mis-configured and sending DNS queries to the wrong place, etc. The worst was the outbound UDP rule which showed a jillion or so reply packets from domain controllers going to various non-routable subnets in use at homes of VPN users (yeah, the original packet apparently didn't get tunneled properly as the source IP was not their VPN-assigned IP but the IP on their home network - sigh - something else for a TAC ticket one of these days).

As another poster mentioned, though, keep letting people know that egress filters are important and keep implementing the ones you can. We pushed hard for at least blocking outbound DNS queries (except from our proxies) and that's not only helped prevent some malware from functioning but has made it easier to detect as well. With so many linux distros now using a full-blown dns proxy as a client (grumble) which are determined to try doing their own root server queries instead of just using IPs configured in resolv.conf I can't simply watch for and report every blocked outbound DNS query as a compromise, but I have a whole laundry list of known malicious DNS servers that we do watch firewall logs for (oak is a great tool for automagically watching logs for various patterns, BTW). And blocking outbound DNS also means people can't bypass our DNS filters by pointing at opendns or something silly.

Brent

123 Posts