Heartbleed, IE Zero Days, Firefox vulnerabilities - What's a System Administrator to do?

With the recent headlines, we've seen heartbleed (which was not exclusive to Linux, but was predominately there), an IE zero day that had folks over-reacting with headlines of "stop using IE", but Firefox and Safari vulnerabilities where not that far back in the news either.

So what is "safe"?  And as an System Administrator or CSO  what should you be doing to protect your organization?

It's great to say "Defense in Depth" and "The 20 Critical Controls", but that's easy to say and not so easy to do when you are faced with a zero day in the browser that your business application must have to run.  What can you do that's quick and easy, that offers some concrete protection for your community of 20, 200, 2,000 or 20,000 workstations?

I'm starting a list here, but this is by no means a well-researched, exhaustive and complete list, just a starting point:
Inventory your hosts and the software that you run.  Know your network, and know what's running on your servers and workstations.  (if this sounds like another way of saying "read the 20 controls, start implementing at #1 and work your way down the list", then you get my point).  After you inventory, read the list.  Expect a story on this in the next week or so.

Deploy EMET.  In the hype of "don't run IE" headlines, many of the folks who recommended this missed the fact that if you installed EMET, you were nicely protected against last week's 0-day.  Expect a story on this later today (yes, I'm on a bit of a tear).

Read your logs.  In every incident that I've worked, there's been *some* indication of the attack or compromise in logs somewhere - this is why you log after all.  This also holds true for system crashes and problems of any kind.  Troubleshooting always starts with your logs, but if you monitor logs for unusual things, you can expect less trouble to troubleshoot, because you'll see it before it gets to be a problem.  If there are too many logs (yes, log volumes are insane), deploy a tool (there are tons of free ones) that will help you with this.  ELSA (https://code.google.com/p/enterprise-log-search-and-archive/) is a decent starting point for log consolidation and prioritizing, but it's by no means the only solution - find what works for you and use it.

Control your network perimeter (if you can define a perimeter).  Put an egress filter that allows what you need, then has a deny any/any/log statement at the bottom.  The "log" part makes it simpler to add new list entries to satisfy new business requirements as they come up.

Also at your perimeter, have the ability to block some of the "problem" applictions when you know that things have gone particularly bad.  For instance, if there is a Flash zero-day, there's no shame in sending a note to your users to say "we have to block flash at the firewall for a few days until there's a fix".  Ditto that for ActiveX, Java, PDF files and whatever else you'd care to add to the list.  

Many of these settings are simple tick-boxes at the firewall, some are IPS signatures or some might need a proxy or web content filter product.  The key to all of these is to be prepared, know where the knobs are that you need to tweak, and know what you can and can't do both technically and within your organization.  If you're caught by surprise and put a "fix" together in a hurry, document what you did so you can re-use that next time, or improve it when you have an hour to spare.

Talk to your users.  Keep a steady flow of communication going, let them know what's going on.  Call this "Security Awareness Training" if that scores you points, but the object of the game is to keep your user community in the loop - the more they know about what they should do or not do (and why), the fewer problems will come your way from that direction.  Also, the more that IT and the Security Team is seen as helping and advising (in a good way), the more slack they'll cut you when you need it - for instance when you need to block Flash, PDF files or their favourite website for a day or three.

We've been saying this for years, but I still have clients who say "we trust our people, why would we do any of that".  My answer remains "so you trust their malware too?".

I'd invite you, our readers to add to this list in the comments.  This is meant as just a starting point - what have I missed?  What has worked for you?

Rob VandenBrink

Rob VandenBrink

582 Posts
ISC Handler
May 9th 2014
There's a saying when the wise points to the moon, the idiot looks at the finger. That may be my case right now... I keep asking sysadmins to never, *ever* browse the Internet with admin privileges, and especially not on the servers themselves.

Has this practice been generally adopted since, and I'm old school to mention it ?

14 Posts
Rob.. below are a few things that come to mind, though may sound like "brick building" they are not. As a SysAdmin for one controls company and Security Contractor for various companies, these are my hurdles.

Quoting Mr.Prontissimo:There's a saying when the wise points to the moon, the idiot looks at the finger. That may be my case right now... I keep asking sysadmins to never, *ever* browse the Internet with admin privileges, and especially not on the servers themselves.

Has this practice been generally adopted since, and I'm old school to mention it ?

Utopian idea! And I totally understand "old school" however it is impossible in a workplace environment to put on such restrictions. There are tools, "Tor/Onion" plug ins that forbid malicious code from being injected, No Script, Adblock, Ghostery, Qualsys and others all free that greatly minimizes exposure.

Quote: an IE zero day that had folks over-reacting with headlines of "stop using IE",

My take, run from I.E. period... it is my belief that nobody over-reacted from a security flaw that was versions old encompassing various OS's. Oddly enough it took this to have MS fix the issue. Really??!!?! Sadly, like Java, they only care when they lose the mighty "George Washington"

Quote:Deploy EMET.
Seems a bit backwards, MS creates an application to fix another MS flawed application instead of fixing the known problem years earlier?? Then conflicts of EMET crashing IE 10 and 11. https://social.technet.microsoft.com/Forums/security/en-US/60a337f8-edda-45a7-9674-4364a41a3c36/internet-explorer-11-emet-41-windows-81-sehop-has-to-be-disabled?forum=emet


Quote:Talk to your users. Keep a steady flow of communication going
How does one speak to sales that use it as a dagger against you? "My sales are down because I am blocked" Why do they care, they are chasing the $$$ for the company or use it for poor job performance. They care not, free wireless.. cool... ah.. who needs that pesky VPN application, takes too long to click on it.

It would be ideal to be in a 100% proactive mode, but we wind up in a 90% reactive mode. Only when it effects them personally they go.. humm.. that was dumb!

52 Posts
EMET, EMET, EMET. It's everywhere and would solve all your problems.

Unfortunately, it breaks things. Like a lot of things. Even IE and Word which you'd think would be tested.

So, how does one roll out EMET to 5,000 users in an environment with 200+ custom, in-house applications and a technology group with admin permissions so they can install whatever they feel like until a scan picks it up and we remove it (but they could always download it again)?

I'm honestly looking for solutions. I've been saying we need to put together a project for it for almost a year now, but nobody wants to be the technical lead on it. I know I don't. I have enough to do without spending hundreds of man hours troubleshooting every single application we run.

93 Posts
Deploy DNS filters. I build my own using the RPZ feature in bind and as many freely available datasets as I can lay my hands on (known hostile hostnames/domains, known compromised IPs, etc). It's also useful for when you detect a phish that made it through your spam filters but which is a PITA to remove from everyone's mailbox - update the RPZ zone to prevent that URL from resolving. I'm also a fan of OpenDNS (we're using them on top of my own DNS filters).

Deploy Web filters. Even if you don't fundamentally break SSL with (buzzword alert!) ssl interception, you can still block an awful lot of malware, typo-squatters, etc. More than one complaint about our recently deployed web filters turns out to be someone putting in the WRONG URL and landing on a typo-squat'd domain the user would either hand over their credentials to or infect their own systems from.

Either block outbound HTTP/HTTPS from your servers (damn near impossible these days with every piece of software wanting to phone home and a certain OS wanting to download updates or CRL lists from a bazillion new akamai IPs), or at least monitor it with your favorite NIDS tool. We have sharepoint consultants that I frequently catch surfing facebook/myspace/whatever from production servers! It's also eye-opening to see that backup software you're evaluating phoning home all the time - just what IS all that traffic and just how much do you trust them?

Watch your logs and cacti/pnp4nagios/rrdtool graphs. Can't stress that enough. Whenever I'm reacting to a problem and surfing through logs to find evidence of what happened, I usually follow up with a rule in one log-watching tool or another to look for that same sort of event again so the next time someone is notified sooner rather than later.

133 Posts
Quoting Jasey:
So, how does one roll out EMET to 5,000 users in an environment with 200+ custom, in-house applications and a technology group with admin permissions so they can install whatever they feel like until a scan picks it up and we remove it (but they could always download it again)?

In a word: Incrementally.

I'd roll it out in stages, and prioritize by what is at risk of 0-days (e.g., browsers first).

EMET doesn't need to be an all-or-nothing deployment. It's opt-in model allows for it to be configured to ignore everything except one single application.

Your internal 200+ custom apps are perhaps not at much risk of a 0-day? Leave adding them to EMET for later (or never).

Your issues with your technology group running rogue is one of company policy and management; that's not EMET's problem.

1) Configure a GPO for the first test OU(s) being deployed. Make it highly conservative so that it is all opt-in and only includes the target apps, such as browsers.
2) Deploy EMET via SCE, SCCM or WSUS (http://www.darkoperator.com/blog/2013/8/28/deploying-emet-40-in-small-to-medium-environments-using-wsus.html) to the target clients.
3) Deploy an EMET policy auto-refresh task to those same clients via GPO (http://blogs.technet.com/b/kfalde/archive/2014/03/13/automatically-refreshing-emet-gpo-s.aspx).
4) Monitor for issues, and expand deployment to other OUs.
5) Test and add application coverage via the GPO(s), one OU or group at a time. Monitor, tweak, and expand deployment.

I don't think EMET is ideal (the lack of inherent GPO refresh ability sticks in my craw, and a centralized console/reporting would be dandy), but deploying it internally here (granted, we are a much smaller operation than yours) has been fairly painless, using these steps. All Office apps (Win7/Office 2010) and all browsers (IE, Firefox, Chrome, Opera) are covered. We have not had a single issue from a user other than asking what was the new icon in the system tray (you can turn that off via GPO). I'm not sure what the Word/IE issues are you experienced with EMET, but we have not seen them. I've personally been running EMET since version 3.x at home and at work and can recall only one issue long ago with Excel that required a tweak of the EMET settings.

31 Posts
This was quoted in the 1973 Martial Arts movie, Enter the Dragon.

Bruce Lee: It is like a finger, pointing it's way to the moon.

Lao looks at the finger, gets slapped in the head.

Bruce Lee: Don't concentrate on the finger, or you will miss all that heavenly glory, do you understand.

21 Posts
I just noticed that on 4/30/2014 a new version of EMET (EMET 4.1 Update 1) was released:


Today, we are releasing EMET 4.1 Update 1, which contains improvements and bug-fixes. More details on the list of the introduced improvements are available at this KB article. These improvements are the outcome of the feedback you have given us and the forward thinking work we continue to do. We recommend all EMET 4.1 customers download this new version and install it, since the benefits of all these improvements are noticeable. The upgrade experience is seamless, as all the current settings can be kept as-is by choosing “Keep Existing Settings” option during the install process. We also recommend all EMET 3.0 and 4.0 customers to upgrade to EMET 4.1 Update 1 (remember EMET 3.0 will go out of support next June!).

31 Posts
IPS and antivirus vendors often deploy signatures for 0-day exploits before the patch for the underlying vulnerability is available. Although IPS and antivirus are not a definitive solution, which the patch should be, they can provide a substantial amount of protection / risk reduction during the time that you are waiting for, testing and deploying the vendor patch.

13 Posts

Sign Up for Free or Log In to start participating in the conversation!