To build a good security monitoring program, logs are critical.  They feed almost everything we do in monitoring from event correlation to auditing.  We need logs from things such as security tools, network devices, servers, databases and applications in order to have an understanding of what is happening.  Any SIEM, analyst, auditor, SSA etc. is only as good as the data available for analysis. Think garbage in and garbage out.

Networks and systems today are growing ever larger and more complex.  With that, almost all devices can generate logs and there are many different levels of logging that can be configured for each device.  It crucial to ensure you get the right logs from the right devices and at the right level.  Without this you can and will miss events of interest.   However, getting logs from all devices and systems can lead into thousands of devices sending logs and requiring storage of those logs.  How do you know what you are getting and if you even getting the right kinds of logs?  Here are some key points to consider when setting up or evaluating your logging infrastructure:

1.  Know your budget.
The more important the data, the more money companies are *generally* going to be willing to spend on it.  I know money is tight and sadly security is often times the first place they make cuts.  However, you can only work within the means of the funding you have available.  This is important because the more logs you take in requires a more robust logging infrastructure as well as storage for those logs.  You may have to choose smartly about the logs you can process from which devices.  Know the budget you have to work with when starting up.  For those doing an evaluation, its a good time to see if you need to grow and how much that will cost.

2.  Determine the devices you should have logging.  
There is no simple answer to that question.  Ideally its everything, realistically that won't work.  How much storage do you have available? How many messages per second can your infrastructure support?  How big are the different logs from the different systems you'll be receiving?  You also need to know what you are trying to protect and where it lives. Once you answer those questions, it becomes easier to look at your infrastructure for the key devices/systems that you will need logs from that are there to protect your data or will give you information about what is going on with the network and systems surrounding your data.  You want logs coming in that will allow you to paint as complete of a picture as possible.

3.  Determine what level of logging is needed and document it.
This should be a group decision.  What groups in your organization use the logs to support their various jobs?  It is those groups who need to have a say in what that level would be be. It is equally important to document this and the logging level for the different device types for future reference.   For example, a Cisco router has eight different logging levels and each of these will provide more granular information resulting in more log entries.  If you set the level to debug, you can fill your central log storage up pretty fast if you have alot of devices logging.  If you set it to alerts then you may not get the information you want.  You can even mix and match the logging levels by having devices that are forward facing have more detailed logging levels and those devices that are more protected farther back in the network having less detailed logging.  Remember, that this can be changed if it becomes necessary.

4.  Know the log retention policy.
Not only do you have to have enough log storage capacity on your logging infrastructure, but you may also face the requirement of retaining those logs for a specified length of time.  If you have a log retention policy (most organizations do) you need to know what it is to ensure you have enough SAN or offline storage available to retain the logs you generate.  Just because your logging infrastructure may be able to capture the logs you are generating, doesn't mean that you have the necessary long term storage capabilities to meet the retention capabilities.  

5.  Monitor your log submissions.
How do you know are still getting the logs you asked for, at the level you want and nothing has changed?  This is probably one of the toughest areas and the one most often overlooked  My experience has been that people hand folks an SOP with how to send their logs, confirm they are getting logs and then that is it.  I have to say this is tough, especially in a large organization where you can have thousands of devices sending you logs.  How do you know if anything has changed?  How can you afford not to when so much is riding on the logs you get?

6.  Plan for the future.
If your network is going to grow, you need to ensure your logging infrastructure can grow with it.  This can include many areas such as log servers, appliances, SAN storage, offline storage, capacity of tools that are going to ingest these logs, additional personnel, etc.  You need to plan well in advance of when you are going to need to expand.


As you should see by now, your logging team has to have a pulse on everything going on with the network.  A logging infrastructure takes a lot of work, but the benefit is worth it.  It provides the foundation of your security monitoring and deserves more time than it is often given. If your analysts or auditors check the logs for a specific issue and don't see anything ask yourself if because its not there or could it be because they aren't getting the logs or information they need.  

If you any lessons learned or comments for setting up or managing an effective logging infrastructure...please let us know!