No traffic after putting UT into production

[neiby]

Ok, further testing is showing that DNS is breaking with UT in place. I can ping 4.2.2.1 from UT and from a PC inside on the network, but even with all rack units disabled, DNS is not working. I noticed that the Attack module seemed to be blocking lots of stuff, so I disabled it. I have no idea what that thing was doing.

Regardless, with UT in place, DNS does not work. Our PCs query an internal DNS server which in turn queries an external server. That seems to be breaking with UT in place.

Any thoughts?

[neiby]

I'm working with support on this. We tried changing the UT box so that it used 4.2.2.1 for DNS instead of our internal server. It still was not able to resolve names even though it could ping that site. Weird!

[neiby]

To make this even stranger... Even though DNS is still not working, the box intercepted (and delivered) a spam message even though it is not configured to be filtering spam at the moment. WTF?

The eSoft filter was on and Spam is one of the blocked categories, but I didn't think it would actually intercept mail like that. I suppose that's the only way it could filter spam, though. It still struck me as odd.

[dmorris]

its gonna filter whatever mail goes through it if spam blocker is on

[neiby]

its gonna filter whatever mail goes through it if spam blocker is on

Spam blocker is not on, though. At the time, only eSoft was on. I removed Spam blocker from the rack completely before I put UT into production. But a spam email made it through, coincidentally to our email admin, and she brought it to my attention. She was looking at the headers and saw that our UT box was listed in there, which I thought shouldn't have happened if Spam blocker was not running.

[neiby]

We're stumped. I tried moving it back to the way we had it during testing and it worked:

laptop -> UT -> 6509 -> FW -> Internet

Basically, we moved the UT external NIC over to the 6509 and I connected a laptop to the internal interface. Everything works perfectly. Yet when I connect it the way it needs to be in production, DNS will not pass through the box:

6509 -> UT -> FW -> Internet

We tried setting UT to go to 4.2.2.1 for DNS, to no avail. It can ping it just fine, but DNS fails. I thought perhaps something funky was happening inside our firewall and maybe it didn't like something, so I made all these changes and rebooted the firewall. Didn't help: no DNS.

I think the support engineer is stumped, too. This really doesn't make any sense at all.

[sky-knight]

Nope, it's your Cisco.

The bridge is a b-router, this means that all the packets after traversing the UT have the UT's mac address. Your firewall is probably picking this up as an ARP spoof, and halting the packets.

[neiby]

If UT is in transparent bridge mode, why would packets that traverse it have UT's MAC address? I hesitate to ask, but are you sure about that?


EDIT: After thinking about this, I don't understand how that could be happening. If every packet picked up the MAC address of the UT box, no return traffic would ever know where to go. At the ethernet level, it would only have the MAC address of the UT box to send to, so nothing would ever get delivered. UT would have no way of knowing which traffic was which. Unless, I suppose, it is keeping a massive table of all MAC address mappings and then rebuilding the Ethernet frames dynamically with the correct MAC address based on the IP address inside the IP packet.

[neiby]

Also, it does not appear to me that we have Dynamic ARP Inspection enabled on the ASA. Regardless, the ASA still might not like what it's seeing for whatever reason.

[neiby]

I just tested this idea. I put UT back inline and cleared the ARP cache on the firewall. It then populated again with the proper MAC addresses, not the MAC of the UT box. I liked your idea because that would explain what was happening. But that doesn't seem to be the case. I also saw no errors in the firewall logs that might point toward a problem. I'm truly stumped.

I'm done for the night, though. I'll ponder this on Monday.