Threat Level: green Handler on Duty: Didier Stevens

SANS ISC: Monitoring Virtual Machines - SANS Internet Storm Center SANS ISC InfoSec Forums


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
Monitoring Virtual Machines

In the past month or so, I have had more than one discussion with different friends on the monitoring of virtual machines(VMs).  Some of the conversations I have had centered on:  What tool(s) should I use? Should I monitor all communications between VMs?  What about an IDS?  How about Firewalls? etc.  It seems there are a lot of questions about keeping things secure in a virtual world environment. 

Virtualization has allowed us to do some wonderful things and it has also created a nightmare from a security perspective if not done thoughtfully.  Why a nightmare?  Let's say it's an organization with many different departments securely separated:  Financial, Human Resources, Research and Development, Operations, Legal, Security etc.  To consolidate, save money and take advantage server space, the company decides to use virtual machines.   To efficiently maximize the use of available resources, some departments ended up together on the same server, while others stayed on separate servers.  However, just because they are in the same department, does not mean they are allowed to communicate.  Some R&D projects are not allowed to have access to the other for example.  The real question becomes how do you to protect and monitor.

Do you invest in tools to monitor on the server between the VMs?  Do you just monitor outside the servers to ensure what actually leaves?  As an example, one IDS and firewall could be used to monitor and control communications between multiple servers.  However, when you collapse them into VMs, the monitoring ability from that one IDS and firewall has been significantly degraded.   With that said, I also encountered the the other argument that virtual machines can be isolated by the software, so there is no need to worry.  The worry is that you have lost the visibility to monitor that you once had, unless something is done.  In this scenario, you are relying on the virtualization to keep it secure, but what about monitoring to ensure it is providing the security you are expecting?

I believe it is a combination of both VM level monitoring and network level monitoring.  It really depends on the sensitivity of the information on or processed by the VMs as to how you handle it.  There may still be a compelling argument for segregation.  However, if you're in a environment that collapsed servers to save money, you may be find yourself in the position to have to demonstrate the need to spend more money on security and explain why you cannot rely on the existing security architecture.  Virtualization has changed the traditional approach to monitoring and introduced variables that may not have even been considered yet by an organization moving to a virtual world. The emphasis needs to be on having the same view into your systems as you did before.  The existing security architecture and monitoring efforts were put in place for a reason and need to be carefully preserved. 

 

What approach and techniques have you used to ensure you can monitor and secure the virtual environment?

 

Lorna

165 Posts
ISC Handler
We do not host production VM instances for different applications on the same VM server.

We ended up deciding we don't like VMs anyway after a few performance disasters.
Anonymous
At my day job, we've taken the approach that a VM server can't straddle different security boundaries. So the DMZ has a VM server that is physically distinct from the production VM server cluster connected to our various LAN segments. There have been too many bugs where one VM can touch another VM's memory and/or I/O.

We've also separated our Development and Production VM environments so if something runs amok on the Dev environment, it shouldn't be able to affect anything in Prod.

Of course, as this article mentions, this doesn't address the issue of not being able to easily monitor traffic between VMs running on the same physical hardware - that's still a problem...
Brent

120 Posts
Joshua's comment is also spot-on. We found that while it's possible to get decent performance with VMs (provided one is very careful about SAN setup and using good SAN storage), performance *is* definitely an issue.

It's no accident that our ERP database, for instance, is running on a decent standalone SGI server. We also don't put all of our IT eggs in one VM basket either. :-) So for instance in one geographic region, two of our three DNS servers may be on VMs but at least one will always be running on real hardware.
Brent

120 Posts
"The emphasis needs to be on having the same view into your systems as you did before. The existing security architecture and monitoring efforts were put in place for a reason and need to be carefully preserved."

No... the monitoring efforts should change; you need an entirely different view. The view you have before for a physical infrastructure has limitations, and is not well suited to a virtual infrastructure.

Virtualization requires a _more_ detailed view, because you have some new risks as well.

Risks such as one VM sharing resources on the same piece of metal being utilized to facilitate a DoS attack against another VM.

Suddenly monitoring performance metrics and system utilization has become a lot more important for monitoring security.

Virtualization puts you in the position of needing to determine if a spike in CPU usage or disk utilization is legitimate workload, or anomolous, potentially a compromise, or guest OS misbehavior.

As for designing for monitoring networks traffic between virtual machines; that's really easy -- there are many options to address the desire for IDS or application domains between VMs; Private vLans; assigning different groups of VMs to different port groups, with different subnets and VLAN tags; vDistributed Switch + SPAN; vShield Zones, etc, and various network virtualization schemes that involve mapping a dedicated physical NIC directly to a particular VM.

You can also use 'sniffing VMs' on a host.
Overhead toll there is high, but there are a multitude of options -- for network traffic, the solution space is much more massive than the problem space.

VMs' security challenges are not so much 'the network' or 'the contents of the network traffic'.

Mysid

146 Posts
As for access to VM traffic on the Virtual Switch, Cisco's 1000 series soft switch for VMWare may be your answer.
Mysid
1 Posts
@Mike
Answer to what? Too much uptime? -> http://www.vmware.com/security/advisories/VMSA-2011-0002.html

scnr :-)
Alex

13 Posts

Sign Up for Free or Log In to start participating in the conversation!