Threat Level: green Handler on Duty: Daniel Wesemann

SANS ISC: SANS Internet Storm Center SANS Internet Storm Center


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!

Latest Diaries

Network Forensics on Azure VMs (Part #1)

Published: 2021-06-17
Last Updated: 2021-06-17 14:40:22 UTC
by Daniel Wesemann (Version: 1)
3 comment(s)

The tooling to investigate a potentially malicious event on an Azure Cloud VM is still in its infancy. We have covered before (Forensicating Azure VMs) how we can create a snapshot of the OS disk of a running VM. Snapshotting and then killing off the infected VM is very straight forward, but it also tips off an intruder that he has been found out. Sometimes, it makes sense to first watch for a while, and learn more, for example about compromised accounts, lateral movement, or other involved hosts.

And to that end ... we would like to capture the network traffic to and from our affected VM.  In an "old-fashioned" on-premises datacenter, we would in such a situation likely deploy an RSPAN port to mirror the traffic to a collector. A similar functionality was once in the making for Azure with "Virtual Network Tap" but this solution has been put on ice by Microsoft, and is not currently available.

Of course we could also log into the VM and start Wireshark or tcpdump ... but this wouldn't be exactly subtle, and if the machine indeed were compromised, doing so would for sure tip off the attacker.

So ... which options are there?

Option 1:  Network Watcher Packet Capture

This delivers a full PCAP, and Microsoft amazingly even managed not to "improve" the PCAP format. Which means that the PCAPs collected with Network Watcher can in fact be opened and analyzed Wireshark. I'm sure this was just an oversight on Microsoft's part, and they are currently busy re-implementing packet capture to produce a JSON or Excel file instead :)

Joking aside, the main downside of this Network Watcher PCAP is that the capture duration is limited to 18000 seconds aka five hours. So you need to add some sort of logic to (or have an operator manually) restart the capture every couple hours, to make sure no data is lost. The second downside is that the PCAP is written to an Azure Blob Storage account, and the caching/timing of those writes is mightily unpredictable. If you expect that you can "watch live" what is going on on the VM, think again. The PCAPs show up with delays of half a hour or more.  

The main advantage is ... well, it is full PCAPs, and packets rule! Another advantage is that Network Watcher PCAP can be deployed onto a VM interface from within the Azure Portal, without touching the VM itself, and while the VM keeps running. An intruder who has an eye on the processes within the VM would notice though - while the watcher gets deployed via the virtualization layer, it does result in running "Network Watcher" processes that are visible from within the VM.

To enable Network Watcher for your VM, search for "Network Watcher" from the main Azure Portal search bar. Enable the Watcher for the Azure region(s) where you have resources of interest. Then open Network Watcher, and click on "Packet Capture" in the menu on the left.


Option 2: Flow Logs on the Network Interface Security Group (NSG)

This is one of the older techniques. Flow logs don't provide full packet captures, but create a log of source/destination IPs and ports, timestamp, and bytes transferred in each direction. Better than nothing. The main disadvantage is that in this case, Microsoft did reinvent the format, and it is a puke pile of JSON where, to their shame, even Microsoft's own recommendation seems to be to import the logs into Elastic or Splunk, or - yikes - to evaluate them using PowerBI. In a pinch, the format can though also be parsed with a bit of Powershell. https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview explains the structure.

To enable Flow Logs, open the NSG attached to the interface or VNET in question. The option "NSG Flow Logs" in the menu on the left, under "Monitoring", is where you enable the forwarding of flow logs to an Azure Storage Account.


In tomorrow's diary, we're going to look at a third option, using Azure Monitor Insights.

 

 

 

3 comment(s)

If you have more information or corrections regarding our diary, please share.

Recent Diaries

June 2021 Forensic Contest
Jun 16th 2021
1 day ago by Brad (0 comments)

Multi Perimeter Device Exploit Mirai Version Hunting For Sonicwall, DLink, Cisco and more
Jun 15th 2021
2 days ago by Johannes (0 comments)

Update: mac-robber.py
Jun 13th 2021
4 days ago by Jim (0 comments)

Fortinet Targeted for Unpatched SSL VPN Discovery Activity
Jun 12th 2021
5 days ago by Guy (0 comments)

Sonicwall SRA 4600 Targeted By an Old Vulnerability
Jun 11th 2021
6 days ago by Xme (0 comments)

Keeping an Eye on Dangerous Python Modules
Jun 11th 2021
6 days ago by Xme (0 comments)

View All Diaries →

Latest Discussions

Dshield Sensor
created Jun 8th 2021
1 week ago by Rick (0 replies)

API port data
created Apr 25th 2021
1 month ago by JJ (1 reply)

RSS feed containing non-XML compatible characters
created Apr 14th 2021
2 months ago by Anonymous (1 reply)

Handler's Diary (Full text) RSS Feeds stopt working due to a typo
created Mar 5th 2021
3 months ago by bas.auer@auerplace.nl (0 replies)

port_scan issue in Snort3
created Feb 23rd 2021
3 months ago by astraea (0 replies)

View All Forums →

Latest News

Top Diaries

Maldocs: Protection Passwords
Feb 28th 2021
3 months ago by DidierStevens (0 comments)

An infection from Rig exploit kit
Jun 17th 2019
2 years ago by Brad (0 comments)

Qakbot infection with Cobalt Strike
Mar 3rd 2021
3 months ago by Brad (0 comments)

Adversary Simulation with Sim
Mar 2nd 2021
3 months ago by Russ McRee (0 comments)

Fun with DNS over TLS (DoT)
Mar 1st 2021
3 months ago by Rob VandenBrink (0 comments)