I was recently in a client engagement where we had to rebuild / redeploy some ESXi 4.x servers as ESXi 5.1.  This was a simple task, and quickly done (thanks VMware!), but before we were finished I realized that we had missed a critical part - the remote managent port on the servers.  These were iLO ports in this case, as the servers are HP's, but they could just as easily have been DRAC / iDRAC (Dell), IMM or AMM (IBM) or BMC (Cisco, anything with a Tyan motherboard or lots of other vendors).  These "remote management ports are in fact all embedded systems - Linux servers on a card, booting from flash and usually running a web application.  This means that once you update them (via a flash process) they are "frozen in time" as far as Linux versions and patches go. In this case, these iLO cards hadn't been touched in 3 years.

So from a security point of view, all the OS version upgrades and security patches from the last 3 years had NOT been applied to these embedded systems.  What can you do with access to these remote managment cards?  Well, anything from take over the console keyboard and screen, mount a CD or DVD over the netowrk, to powering the server off or reconfigure or delete RAID arrays.  The goal of these cards is to enable you to do almost anything you can do from the console, but from a remote location.  Couple this with difficulty in patching them (or just forgetting to patch them), and youv'e got a serious exposure. 

How can you mitigate a situation like this?  The obvious answer is to patch as updates come out.  For many server vendors however, this means booting the server from a CD or DVD.  This is often a tough sell to management, as it's not only an outage for a production server, but if the firmware update fails or causes some new problem, that could cause another (unplanned) outage later, or in the best case a planned outage to back out the update.  Plus you need to convince them every time the topic comes up that you need remote management at all, which eventually starts to sound like too much work.  But *not* updating critical server components is a ticking time-bomb.  

And all this assumes that your server vendor actually fixes known security issues in their  OS or management interfaces. Since these interfaces are normally web-based, not only does this mean OS patches, but it's VERY common to see XSS, CSRF, authentication bypass and even command injection vulnerabilities built-in to the web interface - so you get your web vulnerabilities for free before you even deploy your own web application!  And server vendors aren't always keen to hear about or fix security issues in these interfaces - from their point of view they might be asked to fix an interface issue in a product they stopped selling years ago, so to the sales folks it might seem like lost money to fix these things.

What many folks do is put the remote management cards in their dedicated management VLAN, which has ACLs and other protections on it.  This certainly isolates these cards from attackers and targetted malware, but if that VLAN is ever breached, these cards become the low-hanging fruit for the attack, which can then be used as a pivot to attack more hardened interfaces such as the Hypervisor admin consoles or SAN management interfaces that are also commonly in these Management VLANS.  I'd suggest both patching your server hardware and segmenting these interface cards off, possibly on a dedicated VLAN just for them.

What other devices in your datacenter should be considered embedded systems, with the same risks? 

• The obvious ones in my mind are KVM (Keyboard / Video / Mouse) switches, which now often have IP interfaces for access, and in some cases operate over IP with dongles attached to the servers - in this case both the KVM unit and the individual dongles are all embedded systems. 
• On the network side - routers, switches and fiber channel switches all have these same risks.  These devices and risks are generally more well-understood though, and in most security conscious environments are patched annually or (hopefully) more frequently.  But in security assessments, it's not uncommon at all for me to find a core routing switch that the entire organization depends on running 10 year old code (just to pick an example from last month).
• Many of the more advanced SCSI RAID, SCSI HBAs or Fiber Channel HBAs (Host Bus Adapters) are now web-enabled, with their own IP addresses and management interfaces (no risk there!).  Folks tend to see these web interfaces as great features, and not make that next cognitive leap to see how they can easily be turned into silent killers.

Oh, one more thing - please change the passwords on all of these!  All the patching in the world won't help you if you're attacker can google for the administrative credentials.  I can't tell you how many SANs, Bladecenters or FC Switches I've seen with the default administrative credentials still in play.  If your admin password is still "password", it's time to change it!

The list of embedded devices in your datacenter goes on - door locks, lights and HVAC controls of course, but I'm sure there are other embedded systems that could be turned to evil in our server racks.  Take a walk down the cold aisle (or better yet, down the hot aisle) in your server room and take a look - please, post anything interesting that you might find in our comment form!
 

===============
Rob VandenBrink
Metafore