Over the last number of weeks (after the Solarwinds Orion news) there's been a lot of discussion on how to detect if a server-based applcation is compromised. The discussions have ranged from buying new sophisticated tools, auditing the development pipeline, to diffing patches. But really, for me it's as simple as saying "should my application server really be able to connect to any internet host on any protocol". Let's take it one step further and say "should my application server really be able to connect to arbitrary hosts on tcp/443 or udp/53 (or any other protocol)". And when you phrase it that way, the answer really should be a simple "no".
For me, fixing this should have been a simple thing. Let's phrase this in the context of the CIS Critical Controls (https://www.cisecurity.org/controls/)
I know these first two are simple - but in your organization, do you have a list of software that's running on each of your servers? With the inbound listening ports? How about outbound ports that connect to known internet hosts?
What about my Linux servers you ask? Don't they need all of github and every everything in order to update? No, no they do not. To get the list of repo's that your server reaches out to for upgrades:
(this lists all sources, filters out comment lines, and looking for "deb" nicely filters out blank lines)
Refine this list further to just get the unique destinations:
So for a stock Ubuntu server, the answer is two - you need access to just two hosts to do a "direct from the internet" update. Your mileage may vary depending on your configuration though.
How about Windows? For a standard application server, the answer usually is NONE. You likely have an internal WSUS, SCCM or SCOM server right? That takes care of updates. Unless you are sending mail with that server (which can be limited to just tcp/25, and most firewalls will restrict to valid SMTP), likely your server is providing a service, not reaching out to anything. Even if the server does reach out to arbitrary servers, you can likely restrict it to specific destination hosts, subnets, protocols or countries.
With a quick inventory, creating a quick "stanza" for each server's outbound permissions goes pretty quickly. For each line, you'll be able to choose a logging action of "log", "alert" or "don't log". Think about these choices carefully, and select the "don't log" option at your peril. Your last line for each server's outbound stanza should almost without fail be your firewall's equivalent of "deny ip <servername> any log"
Be sure that your server change control procedures include a "after this change, does the application or server need any additional (or fewer) internet accesses?"
The fallout from this? Surprisingly little.
For most organizations with less than a hundred server VMs, you can turn this into a "hour or two per day" project and get it done in a month or so.
Will this catch everything? No you still need to address workstation egress, but that's a do-able thing too (https://isc.sans.edu/forums/diary/Egress+Filtering+What+do+we+have+a+bird+problem/18379/). Would this have caught the Solarwinds Orion code in your environment? Yes, parts of it - in most shops the Orion server does not need internet access at all (if you don't depend on the application's auto-update process) - even with that, it's a short "allow" list. And if the reaction is to treat denied packets seriously, you'd have caught it well before it hit the news (this was a **lengthy** incident). The fact that nobody caught it in all that time really means that we're still treating outbound traffic with some dangerous mindsets "we trust our users" (to not make mistakes), "we trust our applications" (to not have malware) and "we trust our server admins" (to not do dumb stuff like browse from a server, or check their email while on a server). If you read these with the text in the brackets, I'm hoping you see that this really should be mindsets we set aside, maybe we should have done this in the early 2000's! This may seem like an over-simplification, but really it's not - this approach really does work.
If you've caught anything good with a basic egress filter, please share using our comment form (NDA permitting of course).
Referenced Critical Controls:
CC1: Inventory and Control of Hardware Assets (all of it, if you haven't done this start with your server VLAN)
Feb 1st 2021
|Thread locked Subscribe||
Feb 1st 2021
1 year ago