IPSec site-to-site 30Mbps max throughput

[dmor]

Has anyone else seen this problem? Last Nov we decided to setup a lab to really test the IPSec site-to-site throughput in a perfect world (no ISP issues, etc.).

What we found was the max throughput we could obtain was 30Mbps.

During the tests, we ran top (or htop) on both Untangle servers & made sure no system resources were maxed out. System resource usage was low.

This lead us to question whether StrongSwan has some built-in limitation, or perhaps Untangle devs hard coded it in order to avoid having system resource issues.

I would like to know whether anyone else has seen this issue, and whether Untangle support is aware of it? Untangle support guys, can you tell me the max throughput you have seen between 2 Untangles?


FYI as I know it will be of interest, here is how the tests were performed:

1. Setup a PC on the LAN side of each Untangle using a directly-connected Ethernet cables.
2. Connected the Untangle WANs to each other using a single ethernet cable.
3. As a point of reference, we tested with no NAT or VPNs. This was done by turning off NAT on the Untangle WAN interfaces, then adding a static route to each Untangle for the other UT's LAN via the adjacent UT's WAN. Copying files between the 2 PCs maxed out at 100Mbps (this was the wire speed of the network).
4. We then did the same test over an IPSec tunnel by:
   
   1. Removing the static routes.
   2. Re-enabling NAT on the WANs.
   3. Installed the IPSec app on each UT and used the most basic configuration (didn't do the advanced/custom stuff). Copied the files again. 30Mbps was the peak.
5. We also repeated this test with an OpenVPN site-to-site tunnel and found the OpenVPN throughput to vary wildly. It was very jumpy/bursty and would sometimes be faster than IPSec and sometimes slower.

Any thoughts on this?

Now I have a customer asking me for a high-throughput site-to-site tunnel and I'm not sure that UT is the right solution, unless UT support can acknowledge this was a problem and assure me it has been fixed.

Thanks,
-
Doug

[jcoffin]

The limitation is probably more due to which type encryption method chosen and the hardware's ability to encrypt/decrypt.

[sky-knight]

No system specifications on the test hardware leaves us all guessing.

VPN performance is a function of CPU.

[dmor]

No system specifications on the test hardware leaves us all guessing.

VPN performance is a function of CPU.

Rob one of the units was an appliance I bought from you a few years ago (Intel D510 CPU). The other was a faster box, I believe running a 4-core or 8-core Silvermont Atom.

[jcoffin]

Atoms have fewer internal data pipelines, narrower data buses, slower clocks, and smaller caches.

[dmor]

Atoms have fewer internal data pipelines, narrower data buses, slower clocks, and smaller caches.

Do you guys know of any hardware with which you've successfully seen 100Mbps throughput over an IPSec tunnel running on 2 Untangles?

Have you guys seen that level of throughput on UT IPSec tunnels at all?

Thanks,
-
Doug

[sky-knight]

That's strange... because our Atom boxes have NICs that offload encryption. But I have to admit I've never actually tested the IPSec, I tested L2TP and OpenVPN, and I was able to get velocities higher than 30mbit.

But if you need IPSec to scream, I wouldn't use less than an i3 based device. You really do need that CPU.

[jcoffin]

Using gateways Untangle u500 and u50, I can get 99.6 Mbps over IPsec in the lab.

PC1 --> u500 --- IPsec tunnel --- u50 <-- PC2


jcoffin@debian-pc:~$ scp 192.168.10.28:./500MB.zip ./
500MB.zip 100% 500MB 12.5MB/s 00:39

https://www.untangle.com/untangle-ng...ll/appliances/

[sky-knight]

^^^^

That's what I've seen as well. The limiter being the u50's platform not being able to go much faster. Two u500 or equivalents should be able to get wire speed out of IPSec.

[dmor]

Cool. Thanks for the lab testing guys. I feel better about that. Still not sure why top didn't show much CPU utilization when we did the testing in the past and bottlenecked at 30Mbps.