Packet Analysis - Where do you start? - SANS Internet Storm Center

Packet Analysis - Where do you start?
We had a reader who sent an email to us tonight asking for some guidance "when tearing into packets". They are wanting to expand their skills at packet analysis. Since Guy was asking for packets earlier in the evening, it was a timely question. For many of people, packets are a mystery. I think back many, many (its been a while) years ago when I first started looking at packets. I was so anxious to learn it and become really good at it but had no idea where to start. Over my years of looking at packets, I have become completely convinced that packet analysis is well and truly art form (and alot of learning). Each person will have their own style and approach to looking at packets and traffic. However, there are some fundamental things to start with. The reader submitted an question and initial list of things he/she thought they should look at first. I'm going to include that question and list, then add to it. Question submitted from reader: "What are the top 10 or so questions (what & why ask) you would ask yourself when looking at packets you suspect contain evil?" Two things with this question. First, the indicators you received, alerts, users calling, etc. are a good way to point yourself in the right direction of things to look at initially. Start narrow and work your way wider when looking at the packets. Second, I would modify the thought process when looking at packets. Don't automatically assume evil! Why you might ask? Because I have watched people try to force packets into being malicious in nature to support their theory, when in reality something was just broke or it was normal behavior! Yet they were convinced something evil was going on and had management all spun up. Look at packets from a clean slate of what are they telling you! Don't assume one way or other. The packets are innocent until proven guilty:) Here's a past diary example of suspicious behavior being "normal": Here is what the reader submitted: Look for what IP went to an IP of interest --> Who followed a link, uploaded some data, etc. Look for a file that was downloaded --> Looking for malicious files to see if machine got infected Look for when (time) a file was downloaded --> To see if data exfiltration or strange behavior started soon after Look for the dst IP and port used --> To review the data going out the door Check DNS TTL --> Low TTLs might be an indication of malicious intent or Flux botnet This is really tough, because there are so many things that you can look for in packets. First and foremost, I can't stress the importance of learning normal!!! Once you learn normal, the abnormal just jumps out at you. The RFCs are a great place, but have coffee handy. There are alot of website that have good information about packets, but beware, I have ran across incorrect data too. Here are a few more things to look at. I can tell you why I look at it, but its the totality of information that helps you figure out what is going on. What protocol is being used? Are we dealing with TCP, UDP, ICMP? It will help focus where to start based on why you went looking in the first place. What made you suspect the traffic and what protocol would it use. Do you think the issue is at the application layer or is something going on in the transport layer? What are your indicators to help narrow your focus? Look at the source port --> Is it staying constant or is it changing? A source port should stay constant for each session, but change (usually incremental) for the next session. A constant source port can indicate a poor coding (or deliberate) in a tool or malware. It is a characteristic that might eventually help you ID what tool/malware it might be. Look at the sequence numbers --> Are they changing like they should or are they staying constant. It can indicate many things depending on the scenario. I look at the TTLs (same IP but widely varying; etc), IPID/fragID (fixed or changing), flag fields (IP and TCP headers), etc. Any of the packet fields to see if the traffic is behaving normally. (if its a large capture, this is much more difficult and time consuming) If you use wireshark to do your packet analysis, it has some great features to give you statistics about your packet capture. This can help you find interesting pieces of traffic to start your analysis. What's in the payload is always of interest. Sometimes, you find glaringly obvious things like: "This program cannot be run in DOS mode" which is a good indicator for an executable being downloaded. As a note for my analysis approach, I like to line up fields up as much as possible. Sometimes a little work up front to do this pays off in a quick visual inspection of the data. You can see anomalies better. Packet analysis is such a amazing, interesting and fun field of interest. I really enjoy seeing and understanding how other people approach the analysis. If you have any tips or advice for becoming good at packet analysis, please send them in! There are alot of young and old analysts out there who would appreciate it.	Lorna 165 Posts ISC Handler
Reply Subscribe	Jan 29th 2017 2 years ago
Thanks for the post. Are there any good books on the topic!	Anonymous
Reply Quote	Jan 31st 2017 2 years ago
Quoting Anonymous:Thanks for the post. Are there any good books on the topic! You should read Stephen Northcutt's "Network Intrusion Detection" and Richard Bejtlich's "The Practice of Network Security Monitoring: Understanding Incident Detection and Response". I learned intrusion detection from Northcutt, and I can't recommend him enough. I have read Bejtlich's "The Tao of Network Security Monitoring" (which I reference often), but not "The Practice..." The latter book is more recent and has a slightly different approach (as far as I can tell from the table of contents), but either book will serve you well. -- JB	Anonymous
Reply Quote	Feb 1st 2017 2 years ago
Well, one thing seems to have been left unmentioned - you need to understand how NORMAL traffic appears in packet captures! I'm a network/software engineer, and I can't tell you how many times people have brought perfectly normal traffic to my attention because "I think there's a network problem." The problem is that they don't know what "normal" looks like. For example - if you think you're looking at a potential HTTP exploit, you'd better know what normal HTTP traffic looks like, right? You have a few resources available that can help: 1) Wireshark. Get Wireshark from http://www.wireshark.org and start learning its power. Its dissectors do a great job of protocol analysis, and you can drill WAY down into most protocols. 2) Sample captures. The Wireshark site includes a healthy library of sample captures. Don't know what a normal DHCP exchange looks like? They have a sample pcap file you can examine for yourself. 3) Capture your own traffic! After all, you know exactly what you type/hit/select/do and when, so it's easy to capture 30 minutes' worth of your own activities and go digging. (I do this periodically, just to keep tabs on what my installed software is REALLY doing...once you know what your NORMAL workflow looks like, it's easy to spot new/unusual/questionable stuff.) You might also keep an eye on the Wireshark Q&A community at http://ask.wireshark.org - lots of good protocol-layer information can be found in those discussions.	Anonymous
Reply Quote	Feb 3rd 2017 2 years ago
When performing any packet analysis I would suggest that you approach it from an 'innocent until proven guilty' point of view. New analysts often see mal intent in every packet when often the majority of the traffic may be unrelated to the alert, attack, or infection that necessitated the investigation. Having a skeptical but thorough mind set and the ability to recognize what constitutes ‘known good’ are paramount. The more exposure you get to pcap data the better. Resources I recommend: https://github.com/USArmyResearchLab/Dshell - very powerful python tool for offline pcap analysis https://github.com/omriher/CapTipper - another python tool, only for HTTP http://forensicscontest.com/puzzles - good practice http://bytepuzzles.com/products/network-forensics-puzzle-2014 - good practice http://www.netresec.com/?page=PcapFiles – little outdated, still has good resources I'd recommend reading 'HTTP: The Definitive Guide' and 'DNS and BIND' to start.	Mike 1 Posts
Reply Quote	Feb 13th 2017 2 years ago
I find that if you are troubleshooting with PCAPs, it almost always because someone else has decided that "it's the network" or "it's the firewall". The goal is always to uncover the real source of the problem - people are never ready to hear you when you say "there's no 30 second delay on a switched network, it sounds maybe like a hostname typo in your app causing a dns delay" (just to pull an example out of the air) without ironclad proof. Half or more of network troubleshooting seems to be diagnosing non-network problems, so the goal (aside from solving the problem) is to minimize the MTTI (Mean Time To Innocence) - getting to the answer as quickly as possible. If we're trading resources, there are lots of excellent "how to maximize Wireshark" (and other analysis tools) papers in the SANS Reading Room -> sans.org/reading-room/	Rob VandenBrink 489 Posts ISC Handler
Reply Quote	Feb 13th 2017 2 years ago

We had a reader who sent an email to us tonight asking for some guidance "when tearing into packets". They are wanting to expand their skills at packet analysis. Since Guy was asking for packets earlier in the evening, it was a timely question. For many of people, packets are a mystery. I think back many, many (its been a while) years ago when I first started looking at packets. I was so anxious to learn it and become really good at it but had no idea where to start. Over my years of looking at packets, I have become completely convinced that packet analysis is well and truly art form (and alot of learning). Each person will have their own style and approach to looking at packets and traffic. However, there are some fundamental things to start with.

The reader submitted an question and initial list of things he/she thought they should look at first. I'm going to include that question and list, then add to it.

Question submitted from reader: "What are the top 10 or so questions (what & why ask) you would ask yourself when looking at packets you suspect contain evil?" Two things with this question. First, the indicators you received, alerts, users calling, etc. are a good way to point yourself in the right direction of things to look at initially. Start narrow and work your way wider when looking at the packets. Second, I would modify the thought process when looking at packets. Don't automatically assume evil! Why you might ask? Because I have watched people try to force packets into being malicious in nature to support their theory, when in reality something was just broke or it was normal behavior! Yet they were convinced something evil was going on and had management all spun up. Look at packets from a clean slate of what are they telling you! Don't assume one way or other. The packets are innocent until proven guilty:) Here's a past diary example of suspicious behavior being "normal":

Here is what the reader submitted:

Look for what IP went to an IP of interest --> Who followed a link, uploaded some data, etc.
Look for a file that was downloaded --> Looking for malicious files to see if machine got infected
Look for when (time) a file was downloaded --> To see if data exfiltration or strange behavior started soon after
Look for the dst IP and port used --> To review the data going out the door
Check DNS TTL --> Low TTLs might be an indication of malicious intent or Flux botnet

This is really tough, because there are so many things that you can look for in packets. First and foremost, I can't stress the importance of learning normal!!! Once you learn normal, the abnormal just jumps out at you. The RFCs are a great place, but have coffee handy. There are alot of website that have good information about packets, but beware, I have ran across incorrect data too. Here are a few more things to look at. I can tell you why I look at it, but its the totality of information that helps you figure out what is going on.

What protocol is being used? Are we dealing with TCP, UDP, ICMP? It will help focus where to start based on why you went looking in the first place. What made you suspect the traffic and what protocol would it use.
Do you think the issue is at the application layer or is something going on in the transport layer? What are your indicators to help narrow your focus?
Look at the source port --> Is it staying constant or is it changing? A source port should stay constant for each session, but change (usually incremental) for the next session. A constant source port can indicate a poor coding (or deliberate) in a tool or malware. It is a characteristic that might eventually help you ID what tool/malware it might be.
Look at the sequence numbers --> Are they changing like they should or are they staying constant. It can indicate many things depending on the scenario.
I look at the TTLs (same IP but widely varying; etc), IPID/fragID (fixed or changing), flag fields (IP and TCP headers), etc. Any of the packet fields to see if the traffic is behaving normally. (if its a large capture, this is much more difficult and time consuming)
If you use wireshark to do your packet analysis, it has some great features to give you statistics about your packet capture. This can help you find interesting pieces of traffic to start your analysis.
What's in the payload is always of interest. Sometimes, you find glaringly obvious things like: "This program cannot be run in DOS mode" which is a good indicator for an executable being downloaded.

As a note for my analysis approach, I like to line up fields up as much as possible. Sometimes a little work up front to do this pays off in a quick visual inspection of the data. You can see anomalies better.

Packet analysis is such a amazing, interesting and fun field of interest. I really enjoy seeing and understanding how other people approach the analysis. If you have any tips or advice for becoming good at packet analysis, please send them in! There are alot of young and old analysts out there who would appreciate it.

Lorna

165 Posts
ISC Handler

Thanks for the post. Are there any good books on the topic!

Anonymous

Quoting Anonymous:Thanks for the post. Are there any good books on the topic!

You should read Stephen Northcutt's "Network Intrusion Detection" and Richard Bejtlich's "The Practice of Network Security Monitoring: Understanding Incident Detection and Response". I learned intrusion detection from Northcutt, and I can't recommend him enough. I have read Bejtlich's "The Tao of Network Security Monitoring" (which I reference often), but not "The Practice..." The latter book is more recent and has a slightly different approach (as far as I can tell from the table of contents), but either book will serve you well.

-- JB

Anonymous

Well, one thing seems to have been left unmentioned - you need to understand how NORMAL traffic appears in packet captures!

I'm a network/software engineer, and I can't tell you how many times people have brought perfectly normal traffic to my attention because "I think there's a network problem." The problem is that they don't know what "normal" looks like.

For example - if you think you're looking at a potential HTTP exploit, you'd better know what normal HTTP traffic looks like, right?

You have a few resources available that can help:

1) Wireshark. Get Wireshark from http://www.wireshark.org and start learning its power. Its dissectors do a great job of protocol analysis, and you can drill WAY down into most protocols.

2) Sample captures. The Wireshark site includes a healthy library of sample captures. Don't know what a normal DHCP exchange looks like? They have a sample pcap file you can examine for yourself.

3) Capture your own traffic! After all, you know exactly what you type/hit/select/do and when, so it's easy to capture 30 minutes' worth of your own activities and go digging. (I do this periodically, just to keep tabs on what my installed software is REALLY doing...once you know what your NORMAL workflow looks like, it's easy to spot new/unusual/questionable stuff.)

You might also keep an eye on the Wireshark Q&A community at http://ask.wireshark.org - lots of good protocol-layer information can be found in those discussions.

Anonymous

When performing any packet analysis I would suggest that you approach it from an 'innocent until proven guilty' point of view. New analysts often see mal intent in every packet when often the majority of the traffic may be unrelated to the alert, attack, or infection that necessitated the investigation. Having a skeptical but thorough mind set and the ability to recognize what constitutes ‘known good’ are paramount. The more exposure you get to pcap data the better.

Resources I recommend:

https://github.com/USArmyResearchLab/Dshell - very powerful python tool for offline pcap analysis

https://github.com/omriher/CapTipper - another python tool, only for HTTP

http://forensicscontest.com/puzzles - good practice
http://bytepuzzles.com/products/network-forensics-puzzle-2014 - good practice
http://www.netresec.com/?page=PcapFiles – little outdated, still has good resources

I'd recommend reading 'HTTP: The Definitive Guide' and 'DNS and BIND' to start.

Mike

1 Posts

I find that if you are troubleshooting with PCAPs, it almost always because someone else has decided that "it's the network" or "it's the firewall". The goal is always to uncover the real source of the problem - people are never ready to hear you when you say "there's no 30 second delay on a switched network, it sounds maybe like a hostname typo in your app causing a dns delay" (just to pull an example out of the air) without ironclad proof. Half or more of network troubleshooting seems to be diagnosing non-network problems, so the goal (aside from solving the problem) is to minimize the MTTI (Mean Time To Innocence) - getting to the answer as quickly as possible.

If we're trading resources, there are lots of excellent "how to maximize Wireshark" (and other analysis tools) papers in the SANS Reading Room -> sans.org/reading-room/

Rob VandenBrink

489 Posts
ISC Handler