Questions about Bandwidth Control setup

[Synical]

Hi all,

I've been doing some rules cleanup and such on my home Untangle, and thought it was finally time to give the Bandwidth Control app a whirl. As my background is more on the admin/security side than network shaping and flow, I figured I should review the wiki before jumping right in. As I make my way through the wiki and FAQ, I have a few questions.
First, I see the most important setting is the WAN bandwidth rates. I know what we pay for likely isn't what the actual speed is, so I followed the recommendation to do some tests during off-peak time. This ended up taking me down a rabbit hole, as the first couple of results came back in the neighborhood of:

Ping: 11 - 13 ms
Download: 403 - 501 Mbps
Upload: 40 - 41 Mbps

I was perturbed, because that's about half the speed we pay for. On the bright side, it's a chance to play around some more! I set up a rule to bypass my desktop, and ran the tests again. Pretty big difference as now it showed:

Ping: 12 ms
Download: 934 Mbps
Upload: 42

I know some loss is to be expected as Untangle does all the nifty things it does, but I didn't expect that. When running the test normally (not bypassed), CPU Load never exceeded 1.25 with a little over 40% memory used. Untangle is installed on an SSD, and I added an Intel server 10/100/1000 NIC to avoid any "on-board NIC" chokepoints.

So... back up the rabbit hole, should I use the "real" bandwidth rate of 400-500 Mbps as the base for the 95-100% calculation of WAN bandwidth as that's what I'm seeing on the interior network when Untangle is doing it's thing, or 900+ Mbps, as that's what Untangle will see (I presume) on it's external interface?

Additionally, is there anything I should be looking at to see about reducing that bottleneck?

Thanks, and happy weekend!

[Jim.Alles]

Good morning!

• Tell us more about your hardware, to include the Motherboard, CPU, type & amount of RAM, model # of each NIC.
• Tell us more about the NGFW Apps you have installed.
• Are you running any VPNs?
• What is the flavor of the ISP Connection (is PPOE involved)?
• Is this instance of NGFW in a VM/Hypervisor environment?

This isn't likely a limitation of Bandwidth Control. Certainly your results so far indicates that Untangle has a fair amount of processing to do. But that CPU Load % number doesn't tell the whole story.

And for me, picking a culprit off the top of the heap, the PCI bus would be my prime suspect. (like how many lanes?)

To answer the question,

So... back up the rabbit hole, should I use the "real" bandwidth rate of 400-500 Mbps as the base for the 95-100% calculation of WAN bandwidth as that's what I'm seeing on the interior network when Untangle is doing it's thing, or 900+ Mbps, as that's what Untangle will see (I presume) on it's external interface?

I would say use no more than 95% of the 934 Mbps you measured. I might measure it again at 7pm when everyone in the neighborhood is home and streaming videos, for a worst case.

You can also take measurements at #config/network/troubleshooting/download, definitely switch to one of the 50mb tests. You may have to do some math with this one.

[Synical]

Good morning!

• Tell us more about your hardware, to include the Motherboard, CPU, type & amount of RAM, model # of each NIC.
• Tell us more about the NGFW Apps you have installed.
• Are you running any VPNs?
• What is the flavor of the ISP Connection (is PPOE involved)?
• Is this instance of NGFW in a VM/Hypervisor environment?

This isn't likely a limitation of Bandwidth Control. Certainly your results so far indicates that Untangle has a fair amount of processing to do. But that CPU Load % number doesn't tell the whole story.

And for me, picking a culprit off the top of the heap, the PCI bus would be my prime suspect. (like how many lanes?)

Hi Jim! My hardware is a bare metal custom build, and here's the other info you asked for:

• SuperMicro X7SPE-HF-D525 board with an Intel Atom D525 Pineview-D (Quad core) CPU, and 4 GB of DDR3 for RAM. Untangle is installed on a Kingston SATA III SSD.
• NIC in use is an Intel Pro/1000 PT Dual Port adapter (Intel 82571EB Controller) that has a PCIe x4 Lane interface (2.5 GT/s). There's also dual onboard Gigabit LAN ports (Intel 82574L controllers) that aren't currently in use; I plan on using one of the ports for an internal, "lab-only" network I'm getting ready to stand up (to keep isolated from our home network)
• Installed, active apps are Web Filter, Virus Blocker, Application Control, Firewall, Ad Blocker, Reports, Intrusion Prevention, and Configuration Backup. Inactive/disabled apps are Web Cache, Bandwidth Control, and SSL Inspector.
• No VPNs in use.
• ISP is XFinity cable modem, with 1 Gbps down / 40 Mbps up service

To answer the question, I would say use no more than 95% of the 934 Mbps you measured. I might measure it again at 7pm when everyone in the neighborhood is home and streaming videos, for a worst case.

You can also take measurements at #config/network/troubleshooting/download, definitely switch to one of the 50mb tests. You may have to do some math with this one.

Math's not my forte and some of what you said went over my head, which means more fun time reading up on it! Untangle is in-line between the cable modem and our switch, so I used the switch (Netgear GC728X) run a cable test on the cables from my desktop to Untangle and all came back good.

Hopefully that provides a bit of clarity on what I'm looking at, and thanks for the quick response.

[Jim.Alles]

ok, I am out at a site right now, so this will be brief.

I have a problem.
Your hardware appears to be stellar.
That makes troubleshooting this one harder.

[Jim.Alles]

We are doing several different things here, getting your WAN Bandwidth values set up, getting familiar with the Bandwidth Control App, and troubleshooting a loss of bandwidth.

First, I see the most important setting is the WAN bandwidth rates.

Correct.

     WAN Bandwidth: The most critical setting to configure correctly. This should be set to 
85-95% of your actual line speed - we recommend contacting your ISP to get the proper numbers 
and testing to verify.

    Finding the right settings for the WAN Bandwidth may take some experimentation. 
If the bandwidth limit is set too high QoS will have no effect at all. 
If the bandwidth setting is too low, traffic will be unnecessarily limited to a lower bandwidth. 
Remember, traffic only receives preferential treatment when the set bandwidth limit is saturated. 

    The QoS limits are configured correctly when the network has slightly less throughput than 
when QoS is disabled entirely. Depending on your hardware, the QoS settings may not match the 
ISP-provided or testing Mbit numbers exactly. Experimentation is required.

It is the percentage Limits & Reservations of these two values that get divided up as the priorities.

When you first configure the App, it configures entries under QoS. It is good to be aware of that because you can make updates there directly, without running the Bandwidth Control Setup Wizard.

wan bandwidth.png

Also see: http://wiki.untangle.com/index.php/QoS

Don't make the math hard here, initially. As a general rule, your ISP specifies your bandwidth as Mbps. That translates to a million bits per second. QoS wants kbps entered, or thousand bits per second. Just multiply the ISP's number by 1,000.

1 Gbps down / 40 Mbps up service

And, we back that off a bit to 90%
1,000 * 900 = 900,000 kbps down
40 * 900 = 36,000 kbps up

This gives you some sane values for initial set-up. It is a good start, but there might be an advantage in battling bufferbloat by making it even lower on the upload side. The math after measurement generally gets trickier, however.

I know what we pay for likely isn't what the actual speed is, so I followed the recommendation to do some tests during off-peak time.

What is your testing process? It would be a help to me to use something I am familiar with.
http://www.dslreports.com/speedtest
I have been using it for a long time, and I like it in particular because they now evaluate for a Bufferbloat score.

Since you have already done some testing:

Ping: 12 ms
Download: 934 Mbps
Upload: 42

And we are using fq_codel, I would set the WAN Bandwidth as follows:
943 * 950 = 900,000 download
42 * 850 = 35,000 or even 34,000 upload.

That is 95% of actual download speed, and 85% of actual upload speed. These numbers came in pretty close to our initial sanity check. The goal on the download side is to just barely feel the restriction of NGFW.
Upload speed is less, because this is where we can control bufferbloat effectively. Also, you should notice the greater restriction of NGFW to a lesser extent on the upload side.

I was perturbed, because that's about half the speed we pay for.

Indeed.

TROUBLESHOOTING

For troubleshooting, I am going to assume this is not a hardware issue (although this may not be a correct assumption).
And you can do some step-by step troubleshooting (similar to what is known as half-splitting) that we might need to come back around to.
But I am going to jump ahead of this, because I have a hunch.

• Do your speed testing with everything enabled, not bypassed, network quiet (no streaming).
• Then turn off ('power down') the Web Filter App in NGFW.
• test again, tell us the results.

EDIT: Or not. see the post below.

So... back up the rabbit hole, should I use the "real" bandwidth rate of 400-500 Mbps as the base for the 95-100% calculation of WAN bandwidth as that's what I'm seeing on the interior network when Untangle is doing it's thing, or 900+ Mbps, as that's what Untangle will see (I presume) on it's external interface?

To answer this question directly, It should be something close to the 900 Mbps value, as seen at the modem.

You can also do (limited) download speed testing in NGFW, as another check. THAT requires the extra math, and I will save it for some other post.

[Jim.Alles]

There isn't any reason for suspense.

While confirming tests on this thread, I saw that my download speed was low.
I have been having trouble with the cable ISP's physical plant, to include them throwing a new modem at me.
They took the network down at midnight last night for 'maintenance' (unannounced, as usual).

So looking at my test results, I was preparing myself to make the dreaded phone call.

Then I got around to exploring my hunch. I took QoS WAN bandwidth to full, advertised, line rate.

did a test. got about half of the rated download.

Powered down the Web Filter App. and did a test. Got to something like 90% of line rating.
Powered up the Web Filter App. and did a test. Got to something like 90% of line rating.

This was an upgrade from 15.0 to 15.1 on a u25xw box.

I had never noticed the hurt.
Maybe, cycling 'power' on the App is all it takes. YMMV.

Oh, and this was confirmed on the #config/network/troubleshooting/download cachefly test.
2.15 MB/s initially, over multiple tests.
5.33 MB/s after cycling [Web Filter is enabled.]

EDIT:
tHAT HAS NOTHING TO DO WITH wEB fILTER.
I changed two things at the same time - bad troubleshooting technique.
My assumption was invalid.
It turns out the large content size is prioritized to 'Medium', which has a 50% download limit.
This likely affecting the O.P.s observation, as well.

you can un-click enable on this rule for testing:

deprioritize.png

[jcoehoorn]

NIC in use is an Intel Pro/1000 PT Dual Port

Just the one? Untangle should use at minimum two cards: one up to the internet, and the other down to the rest of the network. One card usually means a VM situation, and with a VM on an Atom I'm surprised things run as well as they do.

[Jim.Alles]

Si.

...Dual Port adapter

[Jim.Alles]

Bandwidth Control

As such, we've changed (from the) approach (of) always prioritiz(ing) port 80 traffic at the Medium 
level or higher. The problem with this approach (was) that large web downloads or streaming port 80 
traffic will receive that same priority which is obviously not latency sensitive content. 
To combat this, we've added a "HTTP: Content Length" matcher which allows large downloads to 
be prioritized lower while actual interactive web content is prioritized highly. 
Similarly, "HTTP: Content Type" can be used to deprioritize any specific content types that should 
not receive a high priority.

(mostly) From:
http://wiki.untangle.com/index.php/9.4.0_Changelog#Bandwidth_Control

[Jim.Alles]

You can also do (limited) download speed testing in NGFW, as another check. THAT requires the extra math, and I will save it for the next post.

Here is a way to help with the math conversion between bits and MegaBytes:
http://www.matisse.net/bitcalc