Over 20 thousand servers have their iLO interfaces exposed to the internet, many with outdated and vulnerable versions of FW
Integrated Lights-Out (iLO) is a low-level server management system intended for out-of-band configuration, which is embedded by Hewlett-Packard Enterprise on some of their servers[1]. Besides its use for maintenance, it is often used by administrators for an emergency access to the server when everything "above it" (hypervisor or OS) fails and/or is unreachable. Since these kinds of platforms/interfaces are quite sensitive from the security standpoint, access to them should always be limited to relevant administrator groups only and their firmware should always be kept up to date.
About a month ago, I came across an analysis of interesting rootkit, which “hid” in the iLO platform[2]. This enabled it to interact with the infected system on a very low level. Since the iLO offers a web-based interface and the text of the analysis also made an allusion to multiple vulnerabilities that were historically discovered in it, it got me thinking on less sophisticated attacks that might target the platform. If remote attackers were able to get access to an iLO, they would basically have full control over the target server. This would certainly pose an issue on local networks, but would be much worse if any iLOs were accessible online and attackers might compromise them over the internet.
Given the aforementioned sensitivity of this interface, you wouldn’t steal a car…I mean expose your iLO to the internet, would you? Especially if HP explicitly says not to do so in their official documentation…[3]
Unfortunately, it seems that a not insignificant number of IT specialists were less security-minded than was optimal and decided they would (expose iLOs on their servers to the internet, that is, not steal a car… I hope).
But back to the beginning.
To determine whether any iLOs were "out there", I first looked for an image of the iLO login page (Google Image search for “ilo login page” returned plenty of relevant results) and then constructed the following search string which I thought might give me some idea of whether Google indexed any such pages.
ilo proliant "local user name" "password"
To my surprise, this search yielded over 24 thousand results.
A quick glance through the results seemed to indicate that they really represented internet-exposed iLOs, however the number seemed to be excessive to me, so I turned to Shodan for a “second opinion”.
In simple terms, Shodan periodically scans the entire public IP space for open ports, gathers data returned on those ports, and enables one to search through it in multiple different ways. This gives one a lot of data to play with, but since Shodan does not (primarily) index contents of websites, searching through it is usually not as straightforward as using Google.
After some initial trial and error, I ended up using mainly appropriate favicon hashes[4] in order to identify publicly exposed iLOs. I’ve managed to identify 5 different favicons that were used by different iterations of iLO (version 2 to the most current version 5) over the years, and an additional search string that would lead to only iLO version 1 results being returned by Shodan. Having covered all main iLO versions, I calculated MurmurHashes[5] for all of favicons, constructed relevant search strings and eliminated false-positives as best I could (you may find the resulting search strings near the end of this article). After I summed up all the results, they came to 22,120 public IPs.
It seemed that Google was not as far of the mark as I had hoped… And a further look at some of the identified FW version numbers made the situation seem even grimmer.
As we alredy mentioned, the HP Integrated Lights-Out platform has gone through 5 different iterations over the years (iLO, iLO 2, iLO3, iLO 4 and iLO 5). And, over time, multiple vulnerabilities were identified and patched in each of them[6]. Some of these vulnerabilities had fairly high CVSS scores, such as the 9.8 rated CVE-2017-12542[7] – a trivial-to-exploit authentication bypass that affected iLO 4 before firmware version 2.53[8].
Yet, the previously mentioned Shodan searches revealed a significant number of iLO 4s with (sometimes much) lower FW version numbers… And the situation was fairly similar for other vulnerabilities as well.
Overall, Shodan searches that I originally ran about two weeks ago returned the following numbers for different iLO iterations:
iLO 1 | 84 |
iLO 2 | 567 |
iLO 3, 4 and 5 (the most common favicon is used in all 3 iLO versions) | 21,469 |
Given the number of internet-exposed iLO interfaces I managed to find, many of which were running out of date/vulnerable versions of firmware (and none of which should be directly accessible from the internet in the first place, since the exposure alone goes against good industry practice and introduces unnecessary risk), I’ve decided to let the international CSIRT community know about my findings before I published them here. About two weeks back, I sent an e-mail to all full member teams of FIRST and TF-CSIRT with the description of the issue and a list of Shodan “dorks” that they might use to check for exposed iLOs in their own constituencies.
It seems that this effort bore at least some fruit since the number of iLOs detected by Shodan has fallen somewhat in the meantime. The number of results returned by Shodan for the same searches at the time of writing is as follows.
iLO 1 | 83 |
iLO 2 | 529 |
iLO 3, 4 and 5 (the most common favicon is used in all 3 iLO versions) | 20,911 |
Although January has not ended yet and the data on which Shodan computes trends might therefore not be complete, a small decrease in the number of iLOs in which the most common favicon is used can already be seen from the trend chart as well.
We can only hope that this trend continues in the future...
One further point which deserves a mention is the geographic distribution of identified iLOs. As you may see in the following chart, that shows 20 countries in which the most common iLO favicon was detected the highest number of times, the largest number of identified systems was in the United States, though it was not much higher than in the Netherlands or Russia. Although the inclusion of Netherlands at the top of the chart might look strange, since it has historically been one of the datacenter capitals of Europe[9], the high numbers do certainly make at least some sense.
If you would like to check whether your own public IP ranges hide any iLOs, you may use the following search strings along with the “net” operator (e.g.: “http.favicon.hash:958636481 net:192.168.1.0/24”). The third search returns 4 systems other than iLOs globally at the time of writing, all others should be false-positive-free):
- iLO 3, 4 and 5:
http.favicon.hash:2059618623
- iLO 3 and 4:
http.favicon.hash:-685753388
- iLO 4:
http.favicon.hash:-1912577989 -http.html:"Hi, I am ON from" -http.title:"latex"
- iLO 3
http.favicon.hash:958636481
- iLO 2:
http.favicon.hash:178882658
- iLO 1:
http.title:"HP Integrated Lights-Out" -http.title:"Integrated Lights-Out 2"
If you do discover any exposed iLOs in your public IP space, make sure to secure them. Otherwise, the slightly poetic name „Lights-Out“ might take on its literal meaning for your servers…
At a minimum, placing the iLO in a special VLAN with controlled access, and allowing remote access only over a VPN would certainly be a good start. But since I wanted to share some more detailed recommendations as well, I’ve reached out to the HP/HPE PSIRT and asked them for some. They responded with the following recommendation:
HPE recommends that customers follow the iLO security guidelines published at the following links:
- HPE Integrated Lights-Out (iLO) - Implementing Security Best Practices to Protect the iLO Management Interface
https://support.hpe.com/hpesc/public/docDisplay?docId=a00046959en_us&docLocale=en_US
- HPE iLO 5 Security Technology Brief
https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00026171en_us
If you have any HP servers with iLOs in your infrastructure, following the preceding recommendations would certainly be advisable...
Unfortunately, iLOs are only a top of the proverbial iceberg and even if we were able to make sure that all HP iLO interfaces disappeared from the internet, I’m quite sure that many other, similarly sensitive configuration interfaces would remain exposed. But if we want to change that, we have to start somewhere… And this is certainly not a bad place to start from.
[1] https://en.wikipedia.org/wiki/HP_Integrated_Lights-Out
[2] https://threats.amnpardaz.com/en/2021/12/28/implant-arm-ilobleed-a/
[3] https://support.hpe.com/hpesc/public/docDisplay?docId=a00046959en_us&docLocale=en_US
[4] https://isc.sans.edu/forums/diary/Hunting+phishing+websites+with+favicon+hashes/27326/
[5] https://en.wikipedia.org/wiki/MurmurHash
[6] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=Integrated+Lights-Out
[7] https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-hpesbhf03769en_us
[8] https://www.bleepingcomputer.com/news/security/you-can-bypass-authentication-on-hpe-ilo4-servers-with-29-a-characters/
[9] https://www.dutchdatacenters.nl/en/nieuws/dutchdatacenters2019-2/
-----------
Jan Kopriva
@jk0pr
Comments
Anonymous
Dec 3rd 2022
9 months ago
Anonymous
Dec 3rd 2022
9 months ago
<a hreaf="https://technolytical.com/">the social network</a> is described as follows because they respect your privacy and keep your data secure. The social networks are not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go.
<a hreaf="https://technolytical.com/">the social network</a> is not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go. The social networks only collect the minimum amount of information required for the service that they provide. Your personal information is kept private, and is never shared with other companies without your permission
Anonymous
Dec 26th 2022
9 months ago
Anonymous
Dec 26th 2022
9 months ago
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> nearest public toilet to me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> public bathroom near me</a>
Anonymous
Dec 26th 2022
9 months ago
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> nearest public toilet to me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> public bathroom near me</a>
Anonymous
Dec 26th 2022
9 months ago
Anonymous
Dec 26th 2022
9 months ago
https://defineprogramming.com/
Dec 26th 2022
9 months ago
distribute malware. Even if the URL listed on the ad shows a legitimate website, subsequent ad traffic can easily lead to a fake page. Different types of malware are distributed in this manner. I've seen IcedID (Bokbot), Gozi/ISFB, and various information stealers distributed through fake software websites that were provided through Google ad traffic. I submitted malicious files from this example to VirusTotal and found a low rate of detection, with some files not showing as malware at all. Additionally, domains associated with this infection frequently change. That might make it hard to detect.
https://clickercounter.org/
https://defineprogramming.com/
Dec 26th 2022
9 months ago
rthrth
Jan 2nd 2023
8 months ago