It's a note that many of us have received.  If we're unlucky, it's a note that your (not-a-packet-expert) boss has received and we've had to explain it.  It usually goes like this:

We recently received 1 complaint(s) regarding the article below. This issue appears to have originated from your domain (IP address: x.x.x.x).
 
Please take the appropriate action with your customer or the account in question, enforcing your Acceptable Use Policy.
 
Regards,
 
Sincerely,

Bob the Administrator

Internet Abuse Team
Some ISP Communications
Email abuse@someisp.com

Please include your original email to SOMEISP.com in any replies.
 

We have traced activity to your connection that is likely caused by Botnet traffic. Botnets are programs that allow virus writes to take control of other computers. Please take all appropriate steps to eliminate this virus. The communication request may include attempted synchronization between the virus and the controller, spamming activities, network scanning, or spreading the virus to other connections.

Here is a general description of botnets: http://en.wikipedia.org/wiki/Botnet

We have included the name of the bot below where possible.

--Incident--
IP: x.x.x.x Port 41425
Timestamp: 2014-08-17 00:10:02 GMT
Infection: zeus-p2p

-

A note like this takes a fair amount of explanation when discussing it with your client (or manager), who usually isn't a packet guru.  Usually the narrative goes something like this:

While (maybe) it's a good thing that an ISP is looking at traffic to find malicious patterns, unfortunately, in this case the ISP just plain gets it wrong.  Their entire conclusion is based on the source port.  Unfortuntately, the source port in most TCP communications is what is called an "ephemeral" port - it's either the next free port available after 1024, or a random free port above 1024. 

Add to that, in outbound communications, most TCP sessions are run through a firewall that NAT each session ( NAT = Network Address Translation), so that each of these ephemeral ports is mapped to yet a different ephemeral port.

As you can see, from the outside network's perspective, the source port could be just about anything.  So if you are looking for a "suspicious" port, you'll find it as a source port on any corporate firewall most days.

OK, so should the abuse@ folks look at the port at the other end?  Umm- sort-of.  Say someone is sending you an email or browsing to your website - in this case, the ephemeral port is at their end, and the target port is at your end.

Really, what you want to do is look at the ports at both ends - but not to make a final determination.  You might use that method to narrow down your search, before you then look at the contents of the actual packets.  An even better approach is to *start* by looking at the contents of the packets - but if you're an ISP that adds up to a lot of packets which needs a corresponding amount of CPU and disk to deal with it.

If you are an organization who needs to get a handle on outbound traffic (which is just about every organization), consider implementing an IPS on the traffic to/from the inside interface of your firewall - at that location you can see the true IP addresses and ports of all traffic, both source and destination. 

Another great thing to look at implementing is a Netflow collector (or sFlow or jFlow or IPFIX, depending on your gear) - a tool like this helps you to slice and dice your network traffic statistics to get quick answers, or in many shops  netflow graphical display is the most used feature, allowing a quick, visual way to identify "odd" traffic or trends.

Also an egress filter goes a long way to containing problems - if you only allow permitted traffic and have a default deny policy for all "mystery" traffic, you'll find that your problems are more likely to stay contained within your network, and if you are looking at your logs you'll be alerted much earlier in the process.

What you'll find with proper monitoring like this (IPS, Netflow, egress filtering and log monitoring) is that very likely you *do* have infected machines here and there.  For instance we did the "log check" thing last week at a client site and found 1 station trying to infect the neighbors with shellshock.  The difference between your logs and the ISP's logs though is that your logs will properly identify the problem workstation, and the alerts in your logs will be a lot less likely to be a false positive.

The sad thing that I've found is that once you get a false positive note like this from an ISP, you're generally in for a few weeks of them before you can escalate to a support person who understands enough to know that they're getting it wrong, and has the clout to fix the ISP's process.

Please, use our comment form to let us know if you've gotten a note like this, and if it was a false positive or not. 

===============
Rob VandenBrink
Metafore