Hardly anybody has physical access to critical public facing servers. Usually, they are located in a data center, hours away from the system administrators charged with managing them. Doing a system reboot, or trouble shooting a system that has dropped offline, is accomplished by either calling some third party support person at the data center, or via remote consoles. Neither solution is perfect.

Using remote "hands" often requires that these individuals will need access to root/admin passwords, or other credentials used to access these systems. Managing these credentials isn't easy, in particular "under stress" if you systems are down. To get things done, bad decisions are made in a rush if no plan was defined ahead of time. 

The alternative (and my preferred option) is a solid "out-of band" management solution for the systems. This typically includes some form of remote power controller and a "KVM over IP" solution. Not sure if anybody still uses dialup lines to access these consoles, but of course, there should be some secondary IP connection to reach them. There are a number of different vendors that offer solutions for this, but in my experience, many of them suffer from some basic flaws:

Logging: No half way security conscious system administrator would turn off logging for failed and successful login attempts. However, many of these remote access solutions do not even have a logging option, and if they do it is often not configured. Many loging features only allow for limited loging on the device itself, and do not allow to simply send the logs to a syslog server. SMTP traps exist in some cases, but again have to be enabled. I can't remember seeing anything that I would consider "robust" logging, like logging over SSL, from any KVM or RPC device.

Access Control: Sometimes, you find features like RADIUS, which at least allows for some central management of credentials. But remember, you need to be able to use these systems if "things don't work right". So often it is necessary to have local credentials. Better systems at least allow for multiple users and offer different privilege levels. But in some cases, the user can not change their password after the administrator set it for them. This requires the device's administrator to communicate the password to the user which is less then ideal.

Out of data Java applets: Many modern KVMs use a Java applet to provide remote access. These applets are often signed with expired, or self signed, certificates. Newer versions of Java will not allow you to run these applets, so you need to keep an older version around just in case. Aside from having to accept the invalid signature, this puts your system at risk if this older Java version is vulnerable. Some vendors provide updates for the firmware fixing the Java signature issues, but this will only help you as long as your device is still supported.

SSL Configuration: It is usually possible to load your own SSL certificate, which is a minimum requirement to obtain a reasonable SSL connection (you don't want to use the default configuration!!!). But I have seen devices that do only allow MD5 signed certificates. No real certificate authority will provide them anymore due to the documentd weakness of MD5. It is possible to setup your own CA (and sometimes a good idea in cases like this), but you still probably should not use MD5 signed certificates. Aside from that, many of these devices still max out at SSLv3, which is less and less supported. 

So how do you control these "backdoors"? Any great ideas on staying up to date with patches, and limiting/logging access to them? I didn't even mention simple firmware vulnerabilities like authentication bypass that plague these devices.

---
Johannes B. Ullrich, Ph.D.
STI|Twitter|LinkedIn