Monitoring VMWare logs

Published: 2012-05-02
Last Updated: 2012-05-02 08:54:06 UTC
by Bojan Zdrnja (Version: 1)
1 comment(s)

Virtualization is so popular today that there is almost no company that does not use a virtualization platform. VMWare is definitely the most popular one (at least the most popular one I seem to be running into).

It is also not uncommon to see VMWare farms growing exponentially as people tend to throw more hardware and just create new VMs. In such cases, controlling what your administrators do is a must yet I also see that organizations auditing their VMWare farms (and especially administrator’s activities) are pretty rare.

One of the problems is that reviewing VMWare logs can be complex so it is not easy to setup the whole log collection and analysis system correctly; this is something a lot of SIEM’s and similar log collection and analysis tools fail at. So let’s see what we have to work with here and how we can improve things.

System components

For the sake of this diary, I’ll write mainly about the “typical” setups today that consist of ESXi (or ESX, for older setups) host servers and one or more vCenter management servers.

ESXi is VMWare’s host operating system that actually runs the virtual machines. It is highly optimized and has a footprint of only 150 MB. This is what is usually installed on those big servers that today run 20+ virtual machines.

Of course, when you have more than one ESXi server, you want to manage it centrally, not only to make management easier but to also allow some more sophisticated processes such as vMotion and similar. This management is done through a vCenter server.

vCenter basically just runs on a normal Windows operating system machine that itself can be a VM as well. Administrators normally use the VMWare vSphere client application to connect to vCenter and to manage virtual machines (of course, depending on their role and permission).

The same client (vSphere client) can be used by an administrator to connect directly to an ESXi server and to manage VMs that are hosted on that server. As you can probably guess, this creates problems for activity auditing since, in this case, any changes are performed directly on the ESXi host server so vCenter will not see those activities directly.

Finally, if you are trying to troubleshoot some problems, you can allow SSH access directly to the ESXi hosts – this access is disabled by default, but I found it quite often that organizations enable it and leave it enabled.

Log collection

We can see that there are multiple system components that generate logs that we should be collecting. While vCenter keeps its own logs and allows reviews from the console, ESXi hosts will also independently keep their logs that should be audited. Actually, when an administrator modifies something in vCenter, a task will be created that will cause vCenter to connect to the target ESXi host and issue the change.

At the moment I’m usually recommending clients to collect logs from the following components:

What I’ve found is that the VMWare SDK API allows much easier retrieval of logs that will be nice and structured but, if your SIEM does not support it directly, you will have to code a script to retrieve such logs yourself.

Of course, do not forget about the OS logs as well as the database logs – since this server is the most important one, make sure that you’ve protected it accordingly and that you collect all other log files that might be important.

  • ESXi host logs are also very important since an administrator can connect directly to them (unless this has been prevented). With ESXi there aren’t many options and probably the best one is to configure a local Syslog to send logs to the central Syslog server, as shown in the picture below.

VMWare Syslog settings


Keep in mind, though, that VMWare creates many multi-line logs which will eventually be broken due to size limits of Syslog so correlating them on the server side might be quite a bit difficult, if not impossible.

By using Syslog we will also take care of SSH logins, since these will be logged by the console and sent through Syslog to the central server.

Auditing

Now that we have all the logs at one place, we can correlate them and setup alerts on suspicious activities.
Regular log reviews are very important. One of the things you should particularly take a look at is console access. For example, if the administrator that accessed a server’s console through vCenter forgot to logout, any other vCenter administrator can access that server’s console (if he has vCenter permissions to access it, of course).

Good log collection and correlation (remember to collect both vCenter logs as well as logs from all your guest servers) can tell you which server’s consoles were accessed as well as if the administrator had to log in or not.

So check your VMWare environments today and see if you can answer these questions: who, from where and when logged in to my vCenter console, which VMs were migrated and which consoles have been accessed by which administrator in last 30 days?

Let us know what your experiences with collecting and analyzing VMWare logs are and if you did something you’d like to share with our readers so everyone can benefit from your work.
 

--
Bojan
INFIGO IS

 

Keywords: logs vcenter vmware
1 comment(s)
ISC StormCast for Wednesday, May 2nd 2012 http://isc.sans.edu/podcastdetail.html?id=2506

Comments

What's this all about ..?
password reveal .
<a hreaf="https://technolytical.com/">the social network</a> is described as follows because they respect your privacy and keep your data secure:

<a hreaf="https://technolytical.com/">the social network</a> is described as follows because they respect your privacy and keep your data secure. The social networks are not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go.

<a hreaf="https://technolytical.com/">the social network</a> is not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go. The social networks only collect the minimum amount of information required for the service that they provide. Your personal information is kept private, and is never shared with other companies without your permission
https://thehomestore.com.pk/
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> public bathroom near me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> nearest public toilet to me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> public bathroom near me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> public bathroom near me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> nearest public toilet to me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> public bathroom near me</a>
https://defineprogramming.com/
https://defineprogramming.com/
Enter comment here... a fake TeamViewer page, and that page led to a different type of malware. This week's infection involved a downloaded JavaScript (.js) file that led to Microsoft Installer packages (.msi files) containing other script that used free or open source programs.
distribute malware. Even if the URL listed on the ad shows a legitimate website, subsequent ad traffic can easily lead to a fake page. Different types of malware are distributed in this manner. I've seen IcedID (Bokbot), Gozi/ISFB, and various information stealers distributed through fake software websites that were provided through Google ad traffic. I submitted malicious files from this example to VirusTotal and found a low rate of detection, with some files not showing as malware at all. Additionally, domains associated with this infection frequently change. That might make it hard to detect.
https://clickercounter.org/
Enter corthrthmment here...

Diary Archives