Threat Level: green Handler on Duty: Brad Duncan

SANS ISC: Abnormal DNS Volumes - SANS Internet Storm Center SANS ISC InfoSec Forums


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
Abnormal DNS Volumes
On Jan. 05, we recorded a DNS volume about 4 times normal. It then fell back to normal volumes for 3 days, before climbing steadily every day thereafter. We implemented a DNS Filter that filtered out unsupported query types and duplicate requests, yielding these results on Jan. 15, 2017:
Requests received by filter - 49,905
Requests passed onto DNS - 6,075
Requests dropped by filter - 43,830 or 88%

The requests still continued to increase, so we analyzed the queries being made and added a dynamic block list that was triggered on a common unsupported property. Because there was a possibility that a legitimate server might get added to the block list, we cleared it every 4 hours. This gave rise to the following peek statistics on Feb. 25, 2017:
Total queries processed - 977,501
Queries forwarded to DNS - 834
Queries dropped by filter - 976,667 or 99.9%
1133 unique IP's filtered = 862 queries per IP per day

At first, we thought that the source IP address must be spoofed, but we soon started to have some doubts. Following up on a suggestion that analyzing the TTL could help to determine which requests are coming from spoofed addresses, I started to log the TTL on IP addresses as they were added to the block list. The TTL is decremented by one as it passes through each router, but not all operating systems use the same starting TTL by default. Unix/Linux systems default to 64, Windows systems default to 128, and Solaris systems default to 255. Most of the start values used seemed to be either 64 or 192, with most of the 64 based queries belonging to Level3, and most of the 192 based queries belonging to Amazon. There did not appear to be any that defaulted to 128. This fact, coupled with the fact that none of the filtered addresses belong to Microsoft, would suggest that Windows systems are not involved. I was able to verify the hop count using a ping on some of the 64 based queries, but I could not find any of the 192 based addresses that would respond to a ping.

So it appears that the source address on most of these queries has NOT been spoofed, and they represent the actual source. What this suggests is that the Linux devices on the other end have been compromised. Is this possible in such a large number?

J.A. Coutts
Anonymous

J.A.,

It appears that you have done quite a bit of research on this DNS issue. Do you have packets that you can share with the Handlers? We love to have packets in cases like this.

Thanks for reaching out and supporting the SANS Internet Storm Center!

Russell
Russell

88 Posts
ISC Handler
Sorry, but I don't have the actual packets, as I operate this server remotely. I can however provide the log file which currently contains the source IP, TTL & Time. I can also add anything you would like to the log file. Here are the historical stats, which appear to tapering off (again).

02/15/2017 (Block List not implemented)
Total queries processed - 49,905
Queries forwarded to DNS - 6,075
02/20/2017 (Block List implemented)
Total queries processed - 103390
Queries forwarded to DNS - 1095
02/21/2017
Total queries processed - 133025
Queries forwarded to DNS - 1056
02/22/2017
Total queries processed - 240771
Queries forwarded to DNS - 601
02/23/2017
Total queries processed - 669611
Queries forwarded to DNS - 1111
02/24/2017
Total queries processed - 511333
Queries forwarded to DNS - 842
02/25/2017
Total queries processed - 977501 |2724 lines = 358 queries/IP
Queries forwarded to DNS - 834 |1133 unique IP's = 862 queries/IP
02/26/2017
Total queries processed - 873405
Queries forwarded to DNS - 843
02/27/2017
Total queries processed - 627842
Queries forwarded to DNS - 1507
02/28/2017
Total queries processed - 346460
Queries forwarded to DNS - 1075
03/01/2017
Total queries processed - 741147
Queries forwarded to DNS - 1099
03/02/2017
Total queries processed - 654581
Queries forwarded to DNS - 1169
03/03/2017
Total queries processed - 421518
Queries forwarded to DNS - 1132
03/04/2017
Total queries processed - 214636
Queries forwarded to DNS - 924
03/05/2017
Total queries processed - 395508
Queries forwarded to DNS - 878
03/06/2017
Total queries processed - 190853
Queries forwarded to DNS - 991
03/07/2017
Total queries processed - 301691
Queries forwarded to DNS - 1300
03/08/2017
Total queries processed - 595027
Queries forwarded to DNS - 972
03/09/2017
Total queries processed - 636251
Queries forwarded to DNS - 1130
03/10/2017
Total queries processed - 333075
Queries forwarded to DNS - 1178
03/11/2017
Total queries processed - 174978
Queries forwarded to DNS - 964
03/12/2017
Total queries processed - 163656
Queries forwarded to DNS - 1013
03/13/2017
Total queries processed - 160345
Queries forwarded to DNS - 1135
03/14/2017
Total queries processed - 147113
Queries forwarded to DNS - 1212
03/15/2017
Total queries processed - 113694
Queries forwarded to DNS - 1124
03/16/2017
Total queries processed - 153091
Queries forwarded to DNS - 1044
03/17/2017
Total queries processed - 93988
Queries forwarded to DNS - 971
03/18/2017
Total queries processed - 96491
Queries forwarded to DNS - 403
03/19/2017
Total queries processed - 63851
Queries forwarded to DNS - 975
03/20/2017
Total queries processed - 93342
Queries forwarded to DNS - 1171
03/21/2017
Total queries processed - 513545
Queries forwarded to DNS - 1231
03/22/2017
Total queries processed - 591397
Queries forwarded to DNS - 970
03/23/2017
Total queries processed - 711868
Queries forwarded to DNS - 1124
03/24/2017
Total queries processed - 367652
Queries forwarded to DNS - 1242
03/25/2017
Total queries processed - 118355
Queries forwarded to DNS - 1238

J.A. Coutts
PS. I also duplicated the original post in General Discussions because I could not locate this post for the longest time. Sorry.
Anonymous

It turns out the problem originates with EDNS. I logged the entire request packets for a brief period, and analyzed the results. The extra length of the OPT record (11 bytes), caused a miscalculation of the Question Length. It was not a problem for the question itself because the Name string ended in a null character, but the Question Length was also used to locate the Question Type. The extra length resulted in the DO flag being detected as the Question Type (32768).

00 60 08 9C BD F4 48 F8 B3 76 22 5E 08 00 45 00
00 4B 1E 5D 00 00 AD 11 AD 5B 36 5C 49 E2 C0 A8
01 03 C8 4C 00 35 00 37 F1 7B 75 A8 00 10 00 01
00 00 00 00 00 01 03 6E 73 31 0A 79 65 6C 6C 6F
77 68 65 61 64 03 63 6F 6D 00 00 1C 00 01 00 00
29 10 00 00 00 80 00 00 00

00 60 08 9C BD F4 (Destination MAC)
48 F8 B3 76 22 5E (Source MAC)
08 00
45
0 1 0 0 - (Version 4)
0 1 0 1 - (IHL = 5x4 = 20 bytes)
00 (Type of Service)
00 4B (Total Length 75)
1E 5D (Packet ID)
00 00 Flags & Fragmentation offset)
AD (TTL = 173)
11 (Protocol = 17 UDP)
AD 5B (Checksum)
36 5C 49 E2 (Source IP)
C0 A8 01 03 (Destination IP)
C8 4C (Source Port)
00 35 (Destination Port)
00 37 (Pseudo Length 55)
F1 7B (Pseudo Checksum)
75 A8 (ID)
00 10
0 (QR - Query)
0 0 0 0 (Opcode - Standard Query)
0 (AA - Authoratative Answer Flag)
0 (TC - Truncation Flag)
0 (RD - Recursion Desired)
1 (RA - Recursion Available)
0 0 0 (Z - Reserved)
0 0 0 0 (Rcode
00 01 (Question Count)
00 00 (Answer Count)
00 00 (Authority Record Count)
00 01 (Additional Record Count)
03 6E 73 31 (3 char - "ns1")
0A 79 65 6C 6C 6F 77 68 65 61 64 (10 char - "yellowhead")
03 63 6F 6D 00 (3 char - "com" & terminating null)
00 1C (Question Type - AAAA)
00 01 (Question Class - IN)
00 (Name - MUST be 0)
00 28 (TYPE 41 - OPT)
10 00 (CLASS 4096 - Payload Size)
00 00 80 00 (TTL - RCODE and flags)
00 00 (RDATA 0)

All of the queries that got blocked showed the RA (Recursion Available) bit set, which I found rather strange for a query, and there is no mention of it in RFC 6891. Instead of reverting to TCP, all but one of the many blocked sources just kept pounding away from many different addresses.

After correcting the code error, DNS volumes have returned to relative normal. But I still have no explanation as to why the volumes got so high.

J.A. Coutts
Anonymous

Sign Up for Free or Log In to start participating in the conversation!