I had a client call me recently with a full on service outage - his servers weren't reachable, his VOIP phones were giving him more static than voice, and his Exchange server wasn't sending or receiving mail - pretty much everything was offline.
So right away this looks like malware, broadcasting on UDP ports 137 and 138 (netbios name services and datagrams). You''ll usually have some level of these in almost any network, but the volumes in this case where high enough to DOS just about everything, I was lucky to keep my SSH sessions (see below) going long enough to get things under control. And yes, that was me that was behind Monday's post on this if this sounds familiar
On Cisco: Where I can, I try to do this in packets per second, so that the discussion with the client can be "of course we shut that port down - there's no production traffic in your environment that should generate more than 100 broadcasts per second." With that done, I now could get to the syslog server. What we needed was a quick-and-dirty list of the infected hosts, in order of how much grief they were causing. First, let's filter out the records of interest - everything that has a broadcast address and a matching netbios port in it - it's a Windows host, so we'll use windows commands (plus some GNU commands): type syslogcatchall.txt | find "172.xx.yy.255/13"
But we don't really want the whole syslog record, plus this short filter still leaves us with thousands of events to go through cut -d " " -f 7 Unfortunately, that field also includes the source port, so let's remove that by using "/" as the field delimeter, and take only the source ip address (field one) cut -d "/" -f 1
Use sort and uniq -c (the -c gives you a count for each source ip) type syslogcatchall.txt | find "172.xx.yy.255/13" | cut -d " " -f 7 | cut -d "/" -f 1 | sort | uniq -c | sort /r > infected.txt
This gave us a 15 line file, sorted so that the worst offenders were at the top of the list, with a record count for each. My client took these 15 stations offline and started the hands-on assess and "nuke from orbit" routine on them, since their AV package didn't catch the offending malware.
and, in related but not directly related lessons ...
=============== |
Rob VandenBrink 578 Posts ISC Handler Apr 4th 2014 |
Thread locked Subscribe |
Apr 4th 2014 8 years ago |
Sign Up for Free or Log In to start participating in the conversation!