Incredibly slow DNS resolution times

[u3b3rg33k]

As of last night, shortly after turning off some 1:1 NATting and setting up some new forwards to make up for it, my DNS resolution took a dump.


for example:
computer:~ u3b3rg33k$ dig untangle.com

; <<>> DiG 9.6-ESV-R4-P3 <<>> untangle.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3425
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 4

;; QUESTION SECTION:
;untangle.com. IN A

;; ANSWER SECTION:
untangle.com. 300 IN A 74.123.28.23

;; AUTHORITY SECTION:
untangle.com. 172783 IN NS dns2.untangle.com.
untangle.com. 172783 IN NS dns.untangle.com.

;; ADDITIONAL SECTION:
dns.untangle.com. 172783 IN A 74.123.28.4
dns.untangle.com. 172783 IN AAAA 2607:f3a0:13::250:56ff:fe96:4b6
dns2.untangle.com. 172783 IN A 74.123.29.4
dns2.untangle.com. 172783 IN AAAA 2001:470:810d:21:230:48ff:fe86:9b29

;; Query time: 4213 msec
;; SERVER: 192.168.0.1#53(192.168.0.1)
;; WHEN: Sat Jan 25 17:54:19 2014
;; MSG SIZE rcvd: 171

if I set the DNS server locally, this happens (different name chosen to avoid cached IP lookup)

computer:~ u3b3rg33k$ dig forums.untangle.com

; <<>> DiG 9.6-ESV-R4-P3 <<>> forums.untangle.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6142
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;forums.untangle.com. IN A

;; ANSWER SECTION:
forums.untangle.com. 300 IN A 74.123.28.16

;; Query time: 87 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Jan 25 18:00:40 2014
;; MSG SIZE rcvd: 53

computer:~ u3b3rg33k$


I'm at a bit of a loss. just for kicks, I rebooted the untangle box, and of course, it made no difference. (FWIW, it's a newish xeon with ECC ram). There are no other performance issues, but I can't figure out what's going on here. 4 seconds is completely ridiculous.
Everything (including untangle) is currently using google's 8.8.8.8 and 8.8.4.4, after switching from my ISP's DNS servers to see if that was the problem. No change.

I can work around this for servers and workstations, but manually setting computers that move (laptops) can cause other problems.

[thassler]

Could the primary DNS server listed on the untangle box be timing out? Maybe try reversing your Primary and Secondary and see what happens.

[u3b3rg33k]

I swapped them around a few times. Even with 8.8.8.8 & 8.8.4.4, or 8.8.4.4 & 8.8.8.8, Its still horribly slow. if I set this manually on client devices, things work perfectly.

[holynger]

Could setup another DHCP server and turn DHCP off on the untangle server..
set the new DHCP server to give out the DNS server of your ISP

See if that fixes it.

Sounds like it might be DNS relay on the untangle server is slowing you down some how.

[milen]

Getting a Wireshark dump of what is going on at the client side is critical.
You may have the "FQDN" forward lookup syndrome, where it adds the connection suffix to everything (e.g. trying to open google.com results in the client sending google.com.mydomain.whatever as a DNS, and then when it doesn't get anywhere, tries to search for google.com.mydomain, and then nails it with a DNS request for google.com). This is a wild guess, and that's why getting a proper dump of what's going on from the client side is critical.
Second important bit is doing the same on the Untangle appliance. Use something like:
tcpdump -i eth0 host W.X.Y.Z and udp port 53
W.X.Y.Z is the address of the DNS server you're supposed to query. Do it on both interfaces - inside and outside, and compare results. It's important that all machines have synchronized clocks using NTP for the measurements to have a meaning.
This is supposed to give us all a better picture of what is going on.

[u3b3rg33k]

That's worth a look - thanks for the idea.

[bluesky]

I've had something similar happen to me since the 10 upgrade. It occurred after making routine changes setting clients to "static DHCP" clients in the router. I did this a few times without issue, then suddenly had the DNS problem happen. If I put a manual DNS in a workstation computer, it worked perfectly. But there seemed to be DNS problems that could only be resolved with a config-restore. When this first happened, I tried for over an hour doing different DNS settings on both ISPs that we run through Untangle. Also tried leaving the ISP's own DNS as well. Neither made a difference.

Worked fine again for a couple more weeks until today, when the same issue happened again. I was again setting a client IP to be a static DHCP in the advanced settings, and Untangle DNS again took a dump. I had to do a config-restore again.

Never had the issue prior to 10.

[u3b3rg33k]

OK, a little update:
I still haven't figured out why what's going on is going on, but I have figured out that untangle appears to be ignoring the DNS servers set for the external interface, and instead referring to an internal DNS server (a legitimate one) that was not set to respond to requests except for sites that it is hosting (because otherwise they don't work). now that the internal server will accept requests from the UTM, things work again (it appears to be forwarding the request to a forwarded port?):

compooter:~ u3b3rg33k$ nslookup cnn.com
Server: 192.168.0.1
Address: 192.168.0.1#53

Non-authoritative answer:
Name: cnn.com
Address: 157.166.226.26
Name: cnn.com
Address: 157.166.226.25

compooter:~ u3b3rg33k$


Anyways at least now I am no longer required to manually set DNS servers for any machine in the building, so that's nice.

[sky-knight]

cat /etc/dnsmasq.conf

Look for server=, if you don't see lines for each DNS server on External. Something might be very wrong.

Untangle always uses itself to resolve DNS, and DNSMasq is how it resolves it. So it's rather important that service is working.