Lessons Learned from Automatic Failover: When 8.8.8.8 "disappears". IPv6 to the Rescue?
A famous XKCD cartoon talks about the importance of the often taken for granted "8.8.8.8" Google DNS server. Like many, I use it often as a quick connectivity check. 8.8.8.8 is an anycast address that exists many times around the globe. I also started to use it for automatic failover on my OPNSense firewall/router.
I am using two uplinks. My primary connection uses Comcast, and recently I started using T-Mobile's 5G Home service as a backup. Overall, I was pleasantly surprised by the T-Mobile service. It competed well with my Comcast service for download speed. I even configured my router to use the T-Mobile service as "primary" for some movie streaming, as it does not have a data cap.
To detect if one of the connections is "down," I am using 75.75.75.75 for the Comcast connection (Comcast's default DNS server) and 8.8.8.8 for T-Mobile. Well... until one day, I had odd issues with the T-Mobile connection. OPNSense had marked it as "down," but the connection appeared to work fine for streaming and other data (also, DNS to 8.8.8.8 didn't work). Just "ping" had high packet loss and the couple of packets that made it had very high latencies:
% ping -c 100 8.8.8.8
...
--- 8.8.8.8 ping statistics ---
100 packets transmitted, 19 packets received, 81.0% packet loss
round-trip min/avg/max/stddev = 114.412/181.239/302.607/40.971 ms
Luckily, Google also runs DoH endpoints on its public DNS servers, so we can verify the results using hping3 and TCP SYN Packets:
% sudo hping -S -p 443 8.8.8.8 -c 100
HPING 8.8.8.8 (en5 8.8.8.8): S set, 40 headers + 0 data bytes
len=44 ip=8.8.8.8 ttl=115 DF id=36841 sport=443 flags=SA seq=0 win=65535 rtt=52.8 ms
len=44 ip=8.8.8.8 ttl=114 DF id=37609 sport=443 flags=SA seq=1 win=65535 rtt=50.9 ms
...
--- 8.8.8.8 hping statistic ---
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 32.2/64.6/96.4 ms
No packet loss! Reasonably latency! And a "speedtest.com" test showed download speeds of around 200 MBit/sec.
I did some more tests pinging a colocated server and got similar results. The colocated server allowed me to test various other packets:
- Empty ICMP packets (no ICMP header, no payload)
- ICMP error packets
- various random ICMP traffic (fragments, misc type and code combinations, different payload sizes)
None of the packets appear to make it. This is, in particular, an issue if ICMP errors are blocked. Like most LTE/5G ISPs, T-Mobile also does carrier-grade NAT, which adds additional "wrinkles" to their network. I could theoretically use IPv6 over T-Mobile, but the modem they provide only offers a /64 and no "Bridge" mode, so you cannot use IPv6 if you are using your own router with IPv6.
Connecting a system directly to the T-Mobile modem via Wi-Fi shows that ICMPv6 is not blocked:
% ping6 2001:4860:4860::8888 -c 100
PING6(56=40+8+8 bytes) 2607:fb90:be84:750f:285b:6b41:63c6:dc46 --> 2001:4860:4860::8888
16 bytes from 2001:4860:4860::8888, icmp_seq=0 hlim=114 time=101.936 ms
16 bytes from 2001:4860:4860::8888, icmp_seq=1 hlim=114 time=57.622 ms
16 bytes from 2001:4860:4860::8888, icmp_seq=2 hlim=114 time=55.386 ms
...
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 33.788/67.714/153.129/22.813 ms
For connection status detection, ICMPv6 will do for now, but there is no way to route T-Mobile IPv6 into my network, and IPv6 failover for a setup like this doesn't exist :(.
---
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu
Twitter|
Comments
www
Nov 17th 2022
6 months ago
EEW
Nov 17th 2022
6 months ago
qwq
Nov 17th 2022
6 months ago
mashood
Nov 17th 2022
6 months ago
isc.sans.edu
Nov 23rd 2022
6 months ago
isc.sans.edu
Nov 23rd 2022
6 months ago
isc.sans.edu
Dec 3rd 2022
6 months ago
isc.sans.edu
Dec 3rd 2022
6 months ago
<a hreaf="https://technolytical.com/">the social network</a> is described as follows because they respect your privacy and keep your data secure. The social networks are not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go.
<a hreaf="https://technolytical.com/">the social network</a> is not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go. The social networks only collect the minimum amount of information required for the service that they provide. Your personal information is kept private, and is never shared with other companies without your permission
isc.sans.edu
Dec 26th 2022
5 months ago
isc.sans.edu
Dec 26th 2022
5 months ago