Investigating an Odd DNS Query

Published: 2019-05-23
Last Updated: 2019-05-23 17:00:31 UTC
by Johannes Ullrich (Version: 1)
3 comment(s)

I have been asked this question a few times, and figure it may be worthwhile to document this in a quick diary. This is typically the result of watching for odd DNS queries (and I highly recommend that). But not all DNS queries are created equal, and sometimes you will see odd, or even malicious, hostnames and domain names in your logs without any wrongdoing on your end.

The latest example I just ran into: . IR being the country level domain for Iran, and I am currently not doing business with Iran, which certainly makes this a bit suspect if it bubbles up to the top of the "odd domain list".

The queries for this domain came in at a rate of 100-150/5min in my Zeek logs:

Next, let's break down all the queries for the "" domain

You can click on the image to get a larger view. But the queries are essentially A/AAAA queries for ns[1-4] To add to this: they all came from my DNS server. Now the DNS server's query log would usually be my next step. But in this case, the query log does not show any queries for * I also didn't see any queries from any of my hosts to the name server for * . The reason for these queries was that a prior query returned these hostnames as authority records. This triggered my name server to do a lookup for these hostnames. So I need to search for answers that contain

It turned out that a prior reverse lookup by the mail servers spam filter returned the authority record, and as a result, the name server then kept looking for ns[1-4] So why did the mail server try to reverse resolve the IP address over and over? My first guess was spam, but it turned out to be a brute force attack against the server:

May 23 16:47:35 mail postfix/smtpd[3420]: connect from unknown[] May 23 16:47:42 mail postfix/smtpd[3420]: warning: unknown[]: SASL LOGIN authentication failed: authentication failure May 23 16:47:42 mail postfix/smtpd[3420]: disconnect from unknown[] May 23 16:47:58 mail postfix/smtpd[3420]: connect from unknown[] May 23 16:48:05 mail postfix/smtpd[3420]: warning: unknown[]: SASL LOGIN authentication failed: authentication failure May 23 16:48:05 mail postfix/smtpd[3420]: disconnect from unknown[]

So at least not entirely a "false positive", but also not terribly exciting. Mail servers are probably the main source of odd DNS queries. They tend to do a lot of reverse lookups for anti-spam, and they also use various DNS based anti-spam and email validation features that often look very much like data exfiltration. You will also see a lot of less common record types in DNS queries from mail servers (TXT, SPF..).

Johannes B. Ullrich, Ph.D. , Dean of Research, SANS Technology Institute

3 comment(s)


Those grahs look nice, what tool are you using to generate them? Is that out of a SIEM?
[quote=comment#42552]Those grahs look nice, what tool are you using to generate them? Is that out of a SIEM?[/quote]

It looks like it came from kibana. I've become a huge fan of elasticsearch and kibana. At my previous day job, we couldn't afford a proper SIEM (I was still trying to afford things like intrusion detection!) and even the commercial logservers were out of reach for us. We had stupid amounts of log traffic to deal with. I looked at the ELK stack (elasticsearch, logstash and kibana) but the documentation was atrocious and I couldn't figure out how to get logstash to parse and index my log data for the things I wanted to log.

A Christmas project to learn nodejs led to a syslog daemon written in nodejs and logging to elasticsearch. :-) The result was better than the log server we had and it eventually became our production logserver. Kibana makes doing basic searching and statistical analysis of logs easy. Making ad-hoc queries or even ad-hoc visualizations of log data very easy. And I found that making new visualizations of my log data VERY useful in finding "weird stuff" going on in my networks.

Actually, I made a (now pretty old) video demonstrating how I used kibana to find a botnet infection and then back-track through the logs to identify exactly when and how the infection occurred. Feel free to take a peek:

It's much improved now and using newer versions of elasticsearch and kibana, though now I'm just running it at home (my employer was bought by a bigger company with bigger budgets). But I made another video explaining some of the under-the-hood design concepts here:

But have a look at logstash too. It's probably got improved documentation now too. I just liked the idea of using my own syslog daemon because it lets me glue together multiple sources of log data. For instance, I use DHCP and VPN logs to map IPs to usernames and hostnames, and then add that data to other logs like DNS logs. I also map IPs to specific geographies so all my logs contain lat/lon coordinates, city names, office names, etc. I *think* there may be some of that in logstash now but not to the same degree. I've got some rudimentary log analysis stuff going on too and my intent is to someday make that a lot better, but this has mostly been for fun and to scratch the programming itch. :-)
I also found things like squid proxies are often the source of weird DNS queries, but not to the same degree as mail servers. In fact, I made my syslog daemon parser for snort logs log both the url and the hostname and the hostname part of the url is logged using the same "query" attribute as DNS logs use. That way when I'm searching all logs for anything related to a particular hostname, I get dns logs AND squid logs.

ssh servers and/or honeypots exposed to the outside world or webservers (depending on if they're configured to do in-addr lookups on IPs that connect to them) also can be sources of weird DNS queries. This can be good and bad. It's often a nice way to pick up new indicators (ie, your ssh honeypot querying squid/dns for hostnames hosting malware payloads). But they can often be really noisy too so be sure you have a good way to filter out noise coming from externally facing systems. :-)

Diary Archives