We just switched from pfSense to Untangle after the changes (bugs) in 2.2 were more than we could tolerate.

We run multiple Untangle firewalls running 11.0.1~svn20150118r39522release11.0-1wheezy in router mode at 10 difference branches in an IPSec over Internet mesh configuration. We are only a few days in and IPSec works pretty good so far. Most of our branch connections are at 50Mbps or lower, so we have no trouble there, but we have a site to site that has a 400/400Mbps connection on one side and a 150/150Mbps connection on the other and for some reason we are only ever able to get up to 80Mbps through this one site to site IPSec VPN tunnel where on the same hardware we used PFSense to saturate the link at 150Mbps.

We have untangle running on VMWare ESXI 5.5 with each side having four core of E5-2690 2.9GHz Xeon processors, 4GB RAM, Intel 10Gb Nic (using e1000) and running on a SSD based RAID 10 datastore. The hardware is barely being utilized, to the point that I think I can knock the CPU down to one or two cores, but I can't get the IPSec tunnel over 80Mbps. We are bypassing IPSec traffic and I have verified that with the session viewer. Here are the settings on each side.

Phase 1-
DH: 2

Phase 2-

Testing was first done with iperf -P 8 and iperf -w 130K -P 8 which both gave me results in the 72-78Mbps across the IPSec tunnel. I tried a Veeam backup across the same connection at it also hovered around 70-79Mbps with a 5 minute average of 76.5Mbps. QoS is currently turned on and set to the correct throughput settings (380/380Mbps and 145/145Mbps but even with it turned off it makes no difference. CPU load is never over 0.21 and checking my VMware stats, the datastore read average at peak transfer is less than 2ms and write is less than 12ms. I have also turned off all of the applications just in case it might help, but it didn't.

I reviewed my notes about setting up PFSense and I was reminded that one thing I had to do on PFSense was set mss clamping to about 1380 mtu in order to get more than 90Mbps on PFSense, but I don't see this option in Untangle and I assume (hopefully) that going from Untangle to Untangle that the Path MTU discovery works out of the box.

I guess I am looking for a sanity check. My resultant questions are--

1. Anyone have an Untangle to Untange IPSec tunnel running over 80Mbps? I am sure this is a yes.

2. Anyone have an VMWare virtualized Untangle to VMWare virtualized Untange IPSec tunnel running over 80Mbps?

3. Does the MSS clamping on Untrangle work automagically?

4. If the answer to 3 is no, is there an accepted way or proven "hack" that allows me to set the MSS clamping for the IPSec VPN tunnel?

5. Any suggestions to speed up the throughput? Overall we are happy with the throughput, but our offsite backups window is pretty tight so the extra 50+Mbps would be really nice.

Required network diagram is below. Although it only shows 3 branches, we have a total of 10, with the Untangle having a 400Mbps WAN and Untangle having the 150Mbps WAN.

IMG_0424 - Copy.JPG