SANS ISC: SANS.edu Internet Storm Center - SANS Internet Storm Center

• SANS Site Network
  • Current Site
  • SANS Internet Storm Center
  • Other SANS Sites Help
  • Graduate Degree Programs
  • Security Training
  • Security Certification
  • Security Awareness Training
  • Penetration Testing
  • Industrial Control Systems
  • Cyber Defense Foundations
  • DFIR
  • Software Security
  • Government OnSite Training

SANS.edu Internet Storm Center

Recently I've been seeing more clients using DNS load balancing (GSLB) to load-balance traditional web, API and other services - it's an easy way to load balance between datacenters for instance.  What I've noticed lately, especially with servers that use cloud "as a service" offerings, is that:

• Not all cluster members neccessarily have the same ports available
• Not all cluster members might even have the same web components installed

If there isn't a TLS front-end (and we're seeing less of this over time), that means that certificates are now maintained per-server, which means that one or more can easily expire "under the radar"

So, how do we work through this problem of "my DNS target is now multiple different hosts, each with their own IP", and add to that, now dozens or hundreds of other hosts (from other organizations) now reside on those same IP addresses?
By default, nmap will only assess the first IP returned for the DNS query against your hostname.  In fact, it comes right out and tells you that when this situation comes up:

nmap -Pn -sT -v -p80,443,8443,9443 somehost.somedomain.com
Starting Nmap 7.92 ( https://nmap.org ) at 2022-05-18 14:08 Eastern Daylight Time
Initiating Parallel DNS resolution of 1 host. at 14:08
Completed Parallel DNS resolution of 1 host. at 14:08, 0.02s elapsed
Initiating Connect Scan at 14:08
Scanning somehost.somedomain.com (1.2.3.4) [4 ports]
Discovered open port 443/tcp on 1.2.3.4
Discovered open port 80/tcp on 1.2.3.4
Discovered open port 8443/tcp on 1.2.3.4
Discovered open port 9443/tcp on 1.2.3.4
Completed Connect Scan at 14:08, 0.08s elapsed (4 total ports)
Nmap scan report for somehost.somedomain.com (1.2.3.4)
Host is up (0.065s latency).
Other addresses for somehost.somedomain.com (not scanned): 5.6.7.8 13.12.11.10
rDNS record for 1.2.3.4: some-rev-dns.some-other-domain.com

PORT     STATE SERVICE
80/tcp   open  http
443/tcp  open  https
8443/tcp open  https-alt
9443/tcp open  tungsten-https

Read data files from: C:\Program Files (x86)\Nmap
Nmap done: 1 IP address (1 host up) scanned in 0.23 seconds

This is of course less than optimal, the workaround is to use the "--resolve-all" switch, so that each IP is scanned (as the dns name provided).   Because of how web servers work, if you just scan the IP address you are quite often not scanning the service you think you are - in many cases for instance you might be "looking" at the apache or IIS default server rather than your customer's web server.

nmap -Pn -sT -v -p80,443,8443,9443 --resolve-all somehost.somedomain.com
Starting Nmap 7.92 ( https://nmap.org ) at 2022-05-18 15:20 Eastern Daylight Time
Initiating Parallel DNS resolution of 3 hosts. at 15:20
Completed Parallel DNS resolution of 3 hosts. at 15:20, 0.04s elapsed
Initiating Connect Scan at 15:20
Scanning 3 hosts [4 ports/host]
Discovered open port 80/tcp on 1.2.3.4
Discovered open port 80/tcp on 5.6.7.8
Discovered open port 80/tcp on 13.12.11.10
Discovered open port 443/tcp on 13.12.11.10
Discovered open port 443/tcp on 1.2.3.4
Discovered open port 8443/tcp on 1.2.3.4
Discovered open port 443/tcp on 5.6.7.8
Discovered open port 8443/tcp on 5.6.7.8
Discovered open port 9443/tcp on 5.6.7.8
Discovered open port 9443/tcp on 13.12.11.10
Completed Connect Scan at 15:20, 0.14s elapsed (12 total ports)
Nmap scan report for somehost.somedomain.com (5.6.7.8)
Host is up (0.043s latency).
rDNS record for 5.6.7.8: some-rev-dns-5.some-other-domain.com

PORT     STATE SERVICE
80/tcp   open  http
443/tcp  open  https
8443/tcp open  https-alt
9443/tcp open  tungsten-https

Nmap scan report for somehost.somedomain.com (13.12.11.10)
Host is up (0.044s latency).
rDNS record for 13.12.11.10: some-rev-dns-13.some-other-domain.com

PORT     STATE SERVICE
80/tcp   open  http
443/tcp  open  https
9443/tcp open  tungsten-https

Nmap scan report for somehost.somedomain.com (1.2.3.4)
Host is up (0.037s latency).
rDNS record for 1.2.3.4: some-rev-dns.some-other-domain.com

PORT     STATE SERVICE
80/tcp   open  http
443/tcp  open  https
8443/tcp open  https-alt

Read data files from: C:\Program Files (x86)\Nmap
Nmap done: 3 IP addresses (3 hosts up) scanned in 1.05 seconds

Key things to note:

• All 3 hosts got scanned
• All 3 hosts got scanned as the target service somehost.somedomain.com
• Not all 3 hosts had the same services - when I looked at this closer, the missing services were API interfaces, which were only fully implemented on the "primary" host.  You might think that this is a pretty big "oops", but the load balancer might know about that, and will only load balance 8443 between the correct 2 hosts and 9443 between the correct (other) two hosts for that service.  It's tough to say without a failover to test it, and that's all you can suggest to the client - "you should test the failover for those two services"

Using this same switch with nmap scripts can also be useful, especially the "SSL-" family of nmap scripts.  ssl-cert and ssl-enum-ciphers in particular should always give you the same results for each cluster member.  "Should" being the operative word!

Have you found some missed configs or  "housekeeping opportunities" when scanning load balanced clusters?  If your NDA permits, please share any war stories using our comment form!

===============
Rob VandenBrink
rob@coherentsecurity.com