Logs - The Foundation of Good Security Monitoring

Published: 2011-08-21
Last Updated: 2011-08-22 02:12:57 UTC
by Lorna Hutcheson (Version: 1)
10 comment(s)

To build a good security monitoring program, logs are critical.  They feed almost everything we do in monitoring from event correlation to auditing.  We need logs from things such as security tools, network devices, servers, databases and applications in order to have an understanding of what is happening.  Any SIEM, analyst, auditor, SSA etc. is only as good as the data available for analysis. Think garbage in and garbage out.

Networks and systems today are growing ever larger and more complex.  With that, almost all devices can generate logs and there are many different levels of logging that can be configured for each device.  It crucial to ensure you get the right logs from the right devices and at the right level.  Without this you can and will miss events of interest.   However, getting logs from all devices and systems can lead into thousands of devices sending logs and requiring storage of those logs.  How do you know what you are getting and if you even getting the right kinds of logs?  Here are some key points to consider when setting up or evaluating your logging infrastructure:

1.  Know your budget.
The more important the data, the more money companies are *generally* going to be willing to spend on it.  I know money is tight and sadly security is often times the first place they make cuts.  However, you can only work within the means of the funding you have available.  This is important because the more logs you take in requires a more robust logging infrastructure as well as storage for those logs.  You may have to choose smartly about the logs you can process from which devices.  Know the budget you have to work with when starting up.  For those doing an evaluation, its a good time to see if you need to grow and how much that will cost.

2.  Determine the devices you should have logging.  
There is no simple answer to that question.  Ideally its everything, realistically that won't work.  How much storage do you have available? How many messages per second can your infrastructure support?  How big are the different logs from the different systems you'll be receiving?  You also need to know what you are trying to protect and where it lives. Once you answer those questions, it becomes easier to look at your infrastructure for the key devices/systems that you will need logs from that are there to protect your data or will give you information about what is going on with the network and systems surrounding your data.  You want logs coming in that will allow you to paint as complete of a picture as possible.

3.  Determine what level of logging is needed and document it.
This should be a group decision.  What groups in your organization use the logs to support their various jobs?  It is those groups who need to have a say in what that level would be be. It is equally important to document this and the logging level for the different device types for future reference.   For example, a Cisco router has eight different logging levels and each of these will provide more granular information resulting in more log entries.  If you set the level to debug, you can fill your central log storage up pretty fast if you have alot of devices logging.  If you set it to alerts then you may not get the information you want.  You can even mix and match the logging levels by having devices that are forward facing have more detailed logging levels and those devices that are more protected farther back in the network having less detailed logging.  Remember, that this can be changed if it becomes necessary.

4.  Know the log retention policy.
Not only do you have to have enough log storage capacity on your logging infrastructure, but you may also face the requirement of retaining those logs for a specified length of time.  If you have a log retention policy (most organizations do) you need to know what it is to ensure you have enough SAN or offline storage available to retain the logs you generate.  Just because your logging infrastructure may be able to capture the logs you are generating, doesn't mean that you have the necessary long term storage capabilities to meet the retention capabilities.  

5.  Monitor your log submissions.
How do you know are still getting the logs you asked for, at the level you want and nothing has changed?  This is probably one of the toughest areas and the one most often overlooked  My experience has been that people hand folks an SOP with how to send their logs, confirm they are getting logs and then that is it.  I have to say this is tough, especially in a large organization where you can have thousands of devices sending you logs.  How do you know if anything has changed?  How can you afford not to when so much is riding on the logs you get?

6.  Plan for the future.
If your network is going to grow, you need to ensure your logging infrastructure can grow with it.  This can include many areas such as log servers, appliances, SAN storage, offline storage, capacity of tools that are going to ingest these logs, additional personnel, etc.  You need to plan well in advance of when you are going to need to expand.

As you should see by now, your logging team has to have a pulse on everything going on with the network.  A logging infrastructure takes a lot of work, but the benefit is worth it.  It provides the foundation of your security monitoring and deserves more time than it is often given. If your analysts or auditors check the logs for a specific issue and don't see anything ask yourself if because its not there or could it be because they aren't getting the logs or information they need.  

If you any lessons learned or comments for setting up or managing an effective logging infrastructure...please let us know!


10 comment(s)


Disk is ridiculously cheap. $2000 on newegg.com will get you many TB of RAID5 disk on a decent quad-core machine. Log everything. If you are using the right solution, your searches should still be fast.
For receiving, use Splunk or Loggly if you have serious cash, otherwise check out Logstash, Logzilla, and my project, enterprise-log-search-and-archive.googlecode.com as free options. Even a properly configured stock syslog-ng or rsyslog will do very well out of the box if you're willing to use grep. A single syslog-ng instance can log 50-100k messages/second, sustained.
For sending, the best free Windows syslogger is eventlog-to-syslog.googlecode.com --it will handle Win2k-2k8 beautifully.
For logging redundancy, remember that your standard Cisco routers and firewalls can load balance port 514 just as well as 80, so you can leverage your existing Cisco gear to create a highly-available syslog setup.
Lastly, check out Anton Chuvakin's many blog posts at chuvakin.blogspot.com for deployment recommendations.
Anyone using Splunk?
We use Splunk and to get logs from Windows systems we use Snare from Intersect Alliance.

I hadn't heard of eventlog-to-syslog.googlecode.com
I tried Splunk a couple times but found it slow and bloated for my ten servers. I had an email conversation with them where they said it needed a dedicated machine.

I am still looking for my Windows log surfing nirvana.

I want something that will let me pull logs for a dozen servers and apply basic filters from a contextual menu then save those filters across sessions.

That way I can screen out all the "Java didn't start" "Error" messages but make sure I don't miss any disk related "Information" messages.
Sorta funny how things work out. Today I released a new version of Sagan (v 0.2.0). We're actually taking a bit of a different approach with logs. That is, we're not trying to replace projects like Splunk/Logzilla/Loganalyzer/etc. We're more about the detection of bad things as they happen. Basically, Sagan acts similar to the "Snort" engine, only with log analysis. That is, it has a rule set similar to that of Snort (which makes it compatible with pulledpork/oinkmaster) and can write out to unfiied2 output formats. What does this mean? You can store critical events, in real time, into a Snort back end. This also means that your IDS/IPS data and log data can reside in the same location. This makes better for correlation. Using tools like Snorby (Frontend for Snort), you can view and analyze/generate reports against that data. For more information, check out our main web site:

My organisation (Aust Fed Govt Agency) has used Snare from its very beginnings. For us, the beauty of Snare is that it's the ultimately customisable solution. You need a specific kind of report from an odd system; Snare can handle it. We've hooked to our AD infrastructure, mainframe, five different database systems, our personnel system, web proxies, etc. etc. We can not only get and analyse logs from every piece of kit we have, we can generate very meaningful reports as well.
We use splunk to search and find it fast enough. To get windows server logs we're using the splunkforwarder. I tried using snare for a while but... didn't care for it. But, splunk is not cheap....so I'm thankful for you guys bringing this important topic up because I'm going to look at the free solutions you mentioned.
But i'm sure splunk is going to be hard to beat because of the "app store." SplunkforSnort has come in handy for automated report building and notifications.
I recommend looking at QRadar Log Manager FE from Q1Labs. Really easy to use and powerful! Work well on VMWare ESX server. This version is free and support 50 EPS. This is enough for a small 20-30 servers + Firewall collection logs. Support Windows/Linux/Unix etc..
Often it's not just a matter of log collecting.
We would also need to correlate these information. The best way to manage logs generated from multiple device types (for real time correlated alerts, reports etc), from my point of view, is using a common log format: ArcSight, for example, uses CEF

why this solution has not been adopted by other SIEM yet ?
Security and complying with standards is essential when choosing the right log management software for a system. A free and open source software tool you can use and scale as you like is NXLog, which allows high-performance and enterprise scalability. You can find all details here: https://nxlog.co/products/nxlog-community-edition - don't hesitate to try if you need a professional solution with great support.

Diary Archives