ESXi 5 and Untangle

[sky-knight]

I've tested it, and gave you the result. If you want to test it yourself, go for it.

I'm telling you that the use of vmxnic will give you issues in the long run. It isn't worth the effort. Untangle isn't running a stock kernel, and sometimes the ko just won't load. You want that in production?

[NetJunkie]

I'll give it a shot. Worst case I roll it back. This is for testing in my 3-node vSphere lab cluster so it's not going to impact many people should it fail. To me some of the changes are more about how it impacts other VMs, not necessarily about how it helps Untangle.

[sky-knight]

If you've got the resource pools configured correctly, there won't be any impact to other VMs because Untangle owns the resources it will use. I also recommend you dedicate physical interfaces to the Untangle for the same reason.

Not much more embarrassing than watching the entire LAN fall offline because someone pulls a file out of your SBS VM at gigabit speeds.

[NetJunkie]

Okay, I'm not going to mess with VMXNET3. When I install the latest VMware Tools on a new install it causes X to start locking up. Underlying system is fine..just X. Not worth the hassle.

My network config is solid. Each node in the cluster has 4 Intel NICs. I have them in a lazy config of all 4 active on the same vDS uplink. Hashing type is Load Based Teaming so it'll actually balance the NICs. Again, not prod so should be just fine.

[sky-knight]

I only point out the resource allocation so when you start feeling random network hiccups you know where to look.

I've done what you're doing on several occasions, I always end up taking UT back to bare metal in the end.

[NetJunkie]

Nevermind. Appears it was a Layer 2 problem. It didn't like the internal and external NICs in the same L2 domain, even on different L3 segments. Not sure why...might investigate that later but I was doing initial testing. Now that I've split them to diff networks it's behaving just fine.

[sky-knight]

That does seem a bit odd, I've never deployed UT in a bridge mode inside a VM environment other than to test. I have seen odd things if the vSwitch the Untangle "Internal" interface is attached to doesn't allow promiscuous mode. That however, generally only applies to bridge mode testing. Also, if you put two bridged interfaces into the same layer 2 domain, things get rather interesting. Broadcast storm much?

You don't have more vNICs than two do you? If so, beware that DMZ and upward by default bridge to external. This will cause issues if you aren't careful. It's best to not connect any vNICs until after you've configured them in Untangle.

[NetJunkie]

That's the thing...I'm not in bridge mode. I'm routing. And yeah, I saw the DMZ NIC being bridged to External. That's set as disconnected. But yeah, I'm routing, not briding...I'd get it if I was bridging. Odd. But it seems fine now.

[sky-knight]

Odd indeed... I've had countless two NIC Untangle routers attached to the same vSwitch.

You aren't sharing a physical NIC for management with a VM enabled vSwitch are you? The performance penalties if you do that are quite painful, and may explain this particular oddity.

[NetJunkie]

I have four physical NICs in a single "team" using Load-Based Teaming. Not sure how familiar you are with vSphere networking but there should be no I/O starvation there. If vSphere sees any single NIC above 75% utilization for 30 seconds it'll flip the MAC addresses from VMs/vmkernels to another NIC and balance the load. It's a lot better than standard hashing so it's not an I/O starvation issue...I'm watching the NIC load.

I think it's an issue with how I was testing. I also noticed some odd things with QoS when I had it like that. Now that I'm setup to flip the network over it's working as expected.

Doing performance testing right now. I have it set with 2 vCPUs. Seeing what it hits at full load (50Mb) with a lot of sessions open. I'd like to get it down to 1 vCPU so I can enable Fault Tolerance. That way if one of my hosts fails it'll instantly flip to the mirror VM with zero downtime....much better than a reboot with VMware HA.