Using NMAP to Assess Hosts in Load Balanced Clusters
Recently I've been seeing more clients using DNS load balancing (GSLB) to load-balance traditional web, API and other services - it's an easy way to load balance between datacenters for instance. What I've noticed lately, especially with servers that use cloud "as a service" offerings, is that:
- Not all cluster members neccessarily have the same ports available
- Not all cluster members might even have the same web components installed
If there isn't a TLS front-end (and we're seeing less of this over time), that means that certificates are now maintained per-server, which means that one or more can easily expire "under the radar"
So, how do we work through this problem of "my DNS target is now multiple different hosts, each with their own IP", and add to that, now dozens or hundreds of other hosts (from other organizations) now reside on those same IP addresses?
By default, nmap will only assess the first IP returned for the DNS query against your hostname. In fact, it comes right out and tells you that when this situation comes up:
nmap -Pn -sT -v -p80,443,8443,9443 somehost.somedomain.com
Starting Nmap 7.92 ( https://nmap.org ) at 2022-05-18 14:08 Eastern Daylight Time
Initiating Parallel DNS resolution of 1 host. at 14:08
Completed Parallel DNS resolution of 1 host. at 14:08, 0.02s elapsed
Initiating Connect Scan at 14:08
Scanning somehost.somedomain.com (1.2.3.4) [4 ports]
Discovered open port 443/tcp on 1.2.3.4
Discovered open port 80/tcp on 1.2.3.4
Discovered open port 8443/tcp on 1.2.3.4
Discovered open port 9443/tcp on 1.2.3.4
Completed Connect Scan at 14:08, 0.08s elapsed (4 total ports)
Nmap scan report for somehost.somedomain.com (1.2.3.4)
Host is up (0.065s latency).
Other addresses for somehost.somedomain.com (not scanned): 5.6.7.8 13.12.11.10
rDNS record for 1.2.3.4: some-rev-dns.some-other-domain.com
PORT STATE SERVICE
80/tcp open http
443/tcp open https
8443/tcp open https-alt
9443/tcp open tungsten-https
Read data files from: C:\Program Files (x86)\Nmap
Nmap done: 1 IP address (1 host up) scanned in 0.23 seconds
This is of course less than optimal, the workaround is to use the "--resolve-all" switch, so that each IP is scanned (as the dns name provided). Because of how web servers work, if you just scan the IP address you are quite often not scanning the service you think you are - in many cases for instance you might be "looking" at the apache or IIS default server rather than your customer's web server.
nmap -Pn -sT -v -p80,443,8443,9443 --resolve-all somehost.somedomain.com
Starting Nmap 7.92 ( https://nmap.org ) at 2022-05-18 15:20 Eastern Daylight Time
Initiating Parallel DNS resolution of 3 hosts. at 15:20
Completed Parallel DNS resolution of 3 hosts. at 15:20, 0.04s elapsed
Initiating Connect Scan at 15:20
Scanning 3 hosts [4 ports/host]
Discovered open port 80/tcp on 1.2.3.4
Discovered open port 80/tcp on 5.6.7.8
Discovered open port 80/tcp on 13.12.11.10
Discovered open port 443/tcp on 13.12.11.10
Discovered open port 443/tcp on 1.2.3.4
Discovered open port 8443/tcp on 1.2.3.4
Discovered open port 443/tcp on 5.6.7.8
Discovered open port 8443/tcp on 5.6.7.8
Discovered open port 9443/tcp on 5.6.7.8
Discovered open port 9443/tcp on 13.12.11.10
Completed Connect Scan at 15:20, 0.14s elapsed (12 total ports)
Nmap scan report for somehost.somedomain.com (5.6.7.8)
Host is up (0.043s latency).
rDNS record for 5.6.7.8: some-rev-dns-5.some-other-domain.com
PORT STATE SERVICE
80/tcp open http
443/tcp open https
8443/tcp open https-alt
9443/tcp open tungsten-https
Nmap scan report for somehost.somedomain.com (13.12.11.10)
Host is up (0.044s latency).
rDNS record for 13.12.11.10: some-rev-dns-13.some-other-domain.com
PORT STATE SERVICE
80/tcp open http
443/tcp open https
9443/tcp open tungsten-https
Nmap scan report for somehost.somedomain.com (1.2.3.4)
Host is up (0.037s latency).
rDNS record for 1.2.3.4: some-rev-dns.some-other-domain.com
PORT STATE SERVICE
80/tcp open http
443/tcp open https
8443/tcp open https-alt
Read data files from: C:\Program Files (x86)\Nmap
Nmap done: 3 IP addresses (3 hosts up) scanned in 1.05 seconds
Key things to note:
- All 3 hosts got scanned
- All 3 hosts got scanned as the target service somehost.somedomain.com
- Not all 3 hosts had the same services - when I looked at this closer, the missing services were API interfaces, which were only fully implemented on the "primary" host. You might think that this is a pretty big "oops", but the load balancer might know about that, and will only load balance 8443 between the correct 2 hosts and 9443 between the correct (other) two hosts for that service. It's tough to say without a failover to test it, and that's all you can suggest to the client - "you should test the failover for those two services"
Using this same switch with nmap scripts can also be useful, especially the "SSL-" family of nmap scripts. ssl-cert and ssl-enum-ciphers in particular should always give you the same results for each cluster member. "Should" being the operative word!
Have you found some missed configs or "housekeeping opportunities" when scanning load balanced clusters? If your NDA permits, please share any war stories using our comment form!
===============
Rob VandenBrink
rob@coherentsecurity.com
Comments
Anonymous
Jun 4th 2022
2 years ago