Friday morning (at least it wasn't Friday afternoon), we got an alert that our database and web servers exceeded the expected load. Sometimes, this "happens." Often it is just some user innocently flooding our API with requests. We do use quite a bit of caching and such for requests, but it can happen that things pile up at the wrong time. So I took a look at the logs. In these cases, I first look at the top IPs sending requests to our API. The first IP that stood out was %ip:137.189.8.184%%. At the time, it had sent about 6,000 requests in 3 hrs. Nothing that would typically cause problems. But the requests themselves didn't make much sense. A small sample:

GET /api/ip/2001:14ba:1f8:c600:94ab:7958:3347:1faa?json HTTP/1.1
GET /api/ip/2001:14ba:7ff:d00:3575:c285:aaaf:4c5e?json HTTP/1.1
GET /api/ip/2001:14bb:150:2f72:7d26:2f5:b2f9:490a?json HTTP/1.1
GET /api/ip/2001:14bb:150:2f72:88c3:36b3:c5b8:507a?json HTTP/1.1
GET /api/ip/2001:14bb:150:2f72:d1f:d7cc:d8b4:6e36?json HTTP/1.1
GET /api/ip/2001:14bb:150:2f72:f196:6cce:b08e:ec1b?json HTTP/1.1
GET /api/ip/2001:14bb:170:ca0:4ca1:cc47:9f4f:5f9f?json HTTP/1.1
GET /api/ip/2001:14bb:170:ca0:901d:76cb:f861:67f4?json HTTP/1.1
GET /api/ip/2001:14bb:170:ca0:a4e7:9ed:8b8f:b8e?json HTTP/1.1

This API endpoint doesn't offer IPv6 data. We do not have enough IPv6 data yet to make this work well. Secondly, it looks like they are "scanning" random IPv6 addresses. There were likely a few more gazillion queries coming to complete this scan. In other words: This will take a while and doesn't make much sense. The user-agent they used curl/7.29.0pointed to a script and didn't comply with our guidelines. So I started by blocking this IP address, but the load on our systems remained high. 

So I started to look at other logs and ran into a flood of requests for https://dshield.org/ipinfo.html. These requests went to dshield.org, not isc.sans.edu, but both sites share the same backend. 

This is where things got a bit more interesting. A quick sample of the logs again:

124.94.188.217, /ipinfo.html?ip=0.25.231.227
59.58.87.149, /ipinfo.html?ip=0.25.232.38
183.164.240.168, /ipinfo.html?ip=0.25.232.18
175.43.172.240, /ipinfo.html?ip=0.25.232.15
114.106.137.211, /ipinfo.html?ip=0.25.232.44
49.85.188.200, /ipinfo.html?ip=0.25.231.222
106.110.200.27, /ipinfo.html?ip=0.25.232.46
183.166.125.155, /ipinfo.html?ip=0.25.232.13
27.156.185.139, /ipinfo.html?ip=0.25.232.40
27.156.211.146, /ipinfo.html?ip=0.25.232.41

This snapshot is after the attack was in progress for a while. The IP addresses appeared to be incrementing. Targeting "ipinfo.html" was likely because this is one of our slower pages. It is even slower if you hit random non-existing IPs. All requests used the same user agent again:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.0.0 Safari/537.36

The user agent looks normal for a slightly older version of Google Chrome. So nothing unusual. By spreading the requests over multiple IPs, it was more challenging to identify the offending requests, but the user-agent was helpful. Looking at the signature of requests to ipinfo.html from this user agent, we counted about 2 million requests against dshield.org but less than 100 against the same URL at isc.sans.edu, the more popular website. So we got a culprit.

More than 100k source IPs participated in the attack. But the most requests we got from a particular IP address were about 350. So there was no clear "winner" to block easily. But looking at the IP addresses, I noticed they came from the same subnets. Summarizing by /16s, only 506 subnets remained. And some of them originated a significant number of requests. 117.69/16 was the most active network, followed by 27.156/16.

Based on whois data:

117.64.0.0 - 117.71.255.255: China Telecom
27.156.0.0 - 27.156.127.255: Broadband network, the city of fuzhou domain, Fujian Province

It turned out the requests causing problems originated exclusively from China. So I started blocking requests from China in our proxy/waf that protects DShield. The load on our servers almost immediately recovered. I already had country lookups configured on the server. We are using Nginx as a proxy, and blocking by country is pretty straightforward:

First, load the geoip_country lookup with data from Maxmind

geoip_country /usr/share/GeoIP/GeoIP.dat;

next, set an "$allowed_country" variable based on the country of origin returned by the country lookup (this will make it easier to block multiple countries)

map $geoip_country_code $allowed_country {
         default yes;
         CN no;
}

Finally, as part of the location directive, return an error if the country is not allowed

location / {
          if ($allowed_country = no) {
                return 444;
          }
...

(I started with 666, but figured for China, 444 is more appropriate).

How much traffic was it overall? Here is a quick graph of the traffic over the weekend from our proxy:

Note how it all stopped this morning. Likely whoever started it came into work and noticed the attack no longer having any effect. I will keep you updated as to what we see next.

Lessons learned:

• Always nice to have a few scripts ready to summarize your logs quickly. Either on the command line or in your fancy log monitoring system. 
• Often, you need to make a quick decision to block an attack accepting some false positives but allowing most of your traffic to succeed. It buys you time and breathing room to find a more specific block.
• I'm not too fond of country-level blocks. But in this case, it kept things going again, and we will remove the block soon after the attack stops.

Some final words:

• This was not a significant DDoS attack. It was pretty small. In the end, you have to "buy your way out of a DDoS attack." For DShield.org, I am willing to accept a bit of downtime like this if I can learn from it.
• Who was behind this attack? I have no idea. All requests came from China. But this could be Chinese hacktivists or just "for the LOLs."
• Were there > 100k systems attacking us? I doubt it. The rate was too low for 100k systems. A lot of the IPs were mobile systems. I suspect they had very dynamic IP addresses, and some devices may show up with multiple IP addresses. My best guess is that these were infected mobile gateways. We see a lot of Mirai-like activity against systems like this. But who knows? I haven't had a chance to look into this closer.
• Am I giving away too much by writing this up? I doubt it, and our mission is to share information and to educate. You do not hear much about these little nuisance DDoS attacks in the news, but they are a big problem for many small businesses.

For a write-up about a more interesting Chinese DDoS in the past, see this attack that involved the Chinese Great Firewall.

[our comments are currently broken, unrelated to the topic of this post. We will have them back shortly. You can reach us via our "Contact Us" form.]

---
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu
Twitter|