When Good Logs Go Bad: Do You Understand Your Logs?

When Good Logs Go Bad: Do You Understand Your Logs?
This keeps happening over and over, and we aren't really covering this as much as we should: Readers finally heed our advise and look at their logs! Now this should make us proud and glad. But then the bad thing happens: They have no idea what they are looking at, and the logs look scary. So the conclusion is "I am hacked!". People stop working and their only goal is to get back a clean system which they find impossible to achieve. For some people, this even results in them becoming unemployed, or worse: They become security professionals. With this introduction, I got a challenge for you: Take a system that you reasonably believe to be "clean". Find some logs that make you think otherwise, and try to explain them. To get started, here some from my iMac desktop that I use to type this diary: `May 29 10:04:37 iMac.local com.apple.authd[57]: Succeeded authorizing right 'com.apple.ServiceManagement.daemons.modify' by client '/usr/libexec/UserEventAgent' [11] for authorization created by '/usr/libexec/UserEventAgent' [11] (12,0)` Even after a full 5 minutes with Google, I am kind of at a loss as to what this means. In my opinion it is nothing to worry about, but then again, that is just my "impression". May 29 10:46:16 iMac.local sandboxd[253] ([7255]): com.apple.WebKit(7255) deny file-read-data /Library/Preferences/com.apple.security-common.plist Seems like a coding bug in Safari to me. Why? Well, WebKit is the rendering engine behind Safari, and Safari runs inside a sandbox on OS X. But why does it try to read "com.apple.security-common.plist"? Looks bad. Maybe I am just doing this too long to still care about some of these messages. Sure looks dangerous to someone who still does care. So what are your favorite non-events? How do you figure out what is a problem and what isn't? Do we need a database of log messages with translations? And remember, �??Just because you're paranoid doesn't mean they aren't after you�?� (Joseph Heller, Catch 22). ------ Johannes B. Ullrich, Ph.D. SANS Technology Institute Twitter LinkedIn I will be teaching next: Intrusion Detection In-Depth - SANS Cyber Defense Forum & Training	Johannes 3950 Posts ISC Handler May 29th 2014
Subscribe	May 29th 2014 6 years ago
OSSEC HIDS Notification. 2014 May 28 21:49:33 Received From: xxxxxxxx->rootcheck Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)." Portion of the log(s): Files hidden inside directory '/dev'. Link count does not match number of files (4,88). This one always raises eyebrows.	Chavez243 15 Posts
Quote	May 29th 2014 6 years ago
Wow...I just had a major flashback to the SANS paper I wrote many years ago. I speak to this very point you highlight in this entry. http://www.sans.org/reading-room/whitepapers/logging/importance-understanding-logs-information-security-standpoint-200 Since I have written that paper, logging standards and formatting has improved greatly, but it is still a challenge. Log correlation tools like SPLUNK can help a lot, but at the end of the day, you still need to be able to interpret these. One line in an event log can mean the difference between identifying the trigger to an incident or missing it completely.	Stewart 1 Posts
Quote	May 29th 2014 6 years ago
I have yet to see any OS log that isn't full of useless noise, some of it downright hilarious. I did a search in my OS X 10.9.3 logs for "secur" and found this gem -- will I be safe in a crash without my seatbelt-profiles? com.apple.launchd.peruser.501[200]: (com.apple.EscrowSecurityAlert) Unknown key: seatbelt-profiles I always have a hard time convincing sysadmins to fix things that are just noise due to simple configuration issues, 'why bother?' is the usual response. Then when we have to do some forensics they bemoan the search taking so long through their ginormous logs....	Paul 44 Posts
Quote	May 30th 2014 6 years ago
Can we look at this from another angle? What events do we want to encourage sysadmins and developers to log? Sure, login/access failures are assumed to be part of the record but what else?	Sassan 4 Posts
Quote	May 30th 2014 6 years ago
Paul: I think this is a great example. The problem is that in some cases (too many cases?) these aren't fixable, and sometimes not even "errors" but notices. Sassan: I think it is not so much as to what to log, but how to log. I think there need to be well defined log levels ('debug','warnings','errors','security'...) that should be logged with the event. Next, there should be some documentation allowing a user to parse the logs.	Johannes 3950 Posts ISC Handler
Quote	May 30th 2014 6 years ago
Quoting Paul:... I always have a hard time convincing sysadmins to fix things that are just noise due to simple configuration issues, 'why bother?' is the usual response. Then when we have to do some forensics they bemoan the search taking so long through their ginormous logs.... Can't agree more. It's like trying to find the needle in the haystack all while having someone dump more hay on top of you.	cnorris 1 Posts
Quote	May 30th 2014 6 years ago
@Dr. J. "Do we need a database of log messages with translations?..." If you go to eventid.net the site will allow you to event a Windows event ID and then read what other users have said about that ID. It's not a comprehensive database but always a start for me. No I'm not affiliated with eventid.net just a happy user of their site.	PW 63 Posts
Quote	May 30th 2014 6 years ago
This is one of the primary points of Sagan (http://sagan.quadrantsec.com). The idea is to use a "collective" of knowledge to assist with detection. The example I like to give is a user with Splunk/Envision/etc looking for "authentication failures". In many cases, the user of these tools will search for terms like "authentication failure", "login incorrect", etc. However, if they have Oracle databases within their network, did they search for "ORA-1017" (Oracle Code for "authentication failure")? Not likely and _why_ would they know to search for this? Sagan has a lot of rules and is used to help security staff detect anomalies in real time. Sagan does _not_ attempt to replace tools like Splunk, which are incredibly useful. It's a tool to aid administrators in "understanding" logs. It is also an open source project :)	PW 9 Posts
Quote	Jun 4th 2014 6 years ago

This keeps happening over and over, and we aren't really covering this as much as we should: Readers finally heed our advise and look at their logs! Now this should make us proud and glad. But then the bad thing happens: They have no idea what they are looking at, and the logs look scary. So the conclusion is "I am hacked!". People stop working and their only goal is to get back a clean system which they find impossible to achieve. For some people, this even results in them becoming unemployed, or worse: They become security professionals.

With this introduction, I got a challenge for you: Take a system that you reasonably believe to be "clean". Find some logs that make you think otherwise, and try to explain them. To get started, here some from my iMac desktop that I use to type this diary:

May 29 10:04:37 iMac.local com.apple.authd[57]: Succeeded authorizing right 'com.apple.ServiceManagement.daemons.modify' by client '/usr/libexec/UserEventAgent' [11] for authorization created by '/usr/libexec/UserEventAgent' [11] (12,0)

Even after a full 5 minutes with Google, I am kind of at a loss as to what this means. In my opinion it is nothing to worry about, but then again, that is just my "impression".

May 29 10:46:16 iMac.local sandboxd[253] ([7255]): com.apple.WebKit(7255) deny file-read-data /Library/Preferences/com.apple.security-common.plist

Seems like a coding bug in Safari to me. Why? Well, WebKit is the rendering engine behind Safari, and Safari runs inside a sandbox on OS X. But why does it try to read "com.apple.security-common.plist"? Looks bad. Maybe I am just doing this too long to still care about some of these messages. Sure looks dangerous to someone who still does care.

So what are your favorite non-events? How do you figure out what is a problem and what isn't? Do we need a database of log messages with translations?

And remember,

�??Just because you're paranoid doesn't mean they aren't after you�?� (Joseph Heller, Catch 22).

------
Johannes B. Ullrich, Ph.D.
SANS Technology Institute
Twitter
LinkedIn

I will be teaching next: Intrusion Detection In-Depth - SANS Cyber Defense Forum & Training

Johannes

3950 Posts
ISC Handler

May 29th 2014

OSSEC HIDS Notification.
2014 May 28 21:49:33

Received From: xxxxxxxx->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."
Portion of the log(s):

Files hidden inside directory '/dev'. Link count does not match number of files (4,88).

This one always raises eyebrows.

Chavez243

15 Posts

Wow...I just had a major flashback to the SANS paper I wrote many years ago. I speak to this very point you highlight in this entry.

http://www.sans.org/reading-room/whitepapers/logging/importance-understanding-logs-information-security-standpoint-200

Since I have written that paper, logging standards and formatting has improved greatly, but it is still a challenge. Log correlation tools like SPLUNK can help a lot, but at the end of the day, you still need to be able to interpret these.

One line in an event log can mean the difference between identifying the trigger to an incident or missing it completely.

Stewart

1 Posts

I have yet to see any OS log that isn't full of useless noise, some of it downright hilarious. I did a search in my OS X 10.9.3 logs for "secur" and found this gem -- will I be safe in a crash without my seatbelt-profiles?

com.apple.launchd.peruser.501[200]: (com.apple.EscrowSecurityAlert) Unknown key: seatbelt-profiles

I always have a hard time convincing sysadmins to fix things that are just noise due to simple configuration issues, 'why bother?' is the usual response. Then when we have to do some forensics they bemoan the search taking so long through their ginormous logs....

Paul

44 Posts

Can we look at this from another angle?

What events do we want to encourage sysadmins and developers to log?

Sure, login/access failures are assumed to be part of the record but what else?

Sassan

4 Posts

Paul: I think this is a great example. The problem is that in some cases (too many cases?) these aren't fixable, and sometimes not even "errors" but notices.

Sassan: I think it is not so much as to what to log, but how to log. I think there need to be well defined log levels ('debug','warnings','errors','security'...) that should be logged with the event. Next, there should be some documentation allowing a user to parse the logs.

Johannes

3950 Posts
ISC Handler

Quoting Paul:...
I always have a hard time convincing sysadmins to fix things that are just noise due to simple configuration issues, 'why bother?' is the usual response. Then when we have to do some forensics they bemoan the search taking so long through their ginormous logs....

Can't agree more. It's like trying to find the needle in the haystack all while having someone dump more hay on top of you.

cnorris

1 Posts

@Dr. J.
"Do we need a database of log messages with translations?..."

If you go to eventid.net the site will allow you to event a Windows event ID and then read what other users have said about that ID. It's not a comprehensive database but always a start for me.

No I'm not affiliated with eventid.net just a happy user of their site.

PW

63 Posts

This is one of the primary points of Sagan (http://sagan.quadrantsec.com). The idea is to use a "collective" of knowledge to assist with detection. The example I like to give is a user with Splunk/Envision/etc looking for "authentication failures". In many cases, the user of these tools will search for terms like "authentication failure", "login incorrect", etc. However, if they have Oracle databases within their network, did they search for "ORA-1017" (Oracle Code for "authentication failure")? Not likely and _why_ would they know to search for this?

Sagan has a lot of rules and is used to help security staff detect anomalies in real time. Sagan does _not_ attempt to replace tools like Splunk, which are incredibly useful. It's a tool to aid administrators in "understanding" logs.

It is also an open source project :)

PW
9 Posts