Page 1 of 2 12 LastLast
Results 1 to 10 of 13
  1. #1
    Master Untangler
    Join Date
    Nov 2009
    Posts
    106

    Thumbs up Proper TCP Tuning = huge speed improvement

    Your Internet connection is underperforming. I can say that with near absolute certainty because Untangle is doing ugly things to TCP parameter tuning. I know why they're doing it, and will explain later.

    They are turning off Autotuning by setting a fixed TCP Receive Window Size, and disabling Window scaling, Selective Acks, and Time Stamps. This is a sort of a bad idea (at least for a subset of their user base) and I have the proof. The good news is that this is a relatively easy fix that will take you a few minutes, tops.

    The Linux kernel we're using has fairly mature TCP adaptive tuning built-in and enabled by default. When it's working as intended (not like it is now in your Untangle boxes), it can make HUGE differences in performance.

    First a bit of background. TCP operates "closed-loop". That is, it verifies packet arrival and tells the source "Got it". For obvious reasons, it would be a bad idea to acknowledge each packet as it arrives, so what happens is incoming packets fill a buffer (the Receive Window) that, when filled, generates an acknowledgement. Now when the source is nearby (low latency) those acknowledgements propagate quickly and transmission continues. The problem arises when the source and destination are far apart (high latency). It takes a LONG time for those acknowledgments to reach the source, which causes dead periods and consequent low transmission rates. (Note to you sticklers - yes, I know this is an oversimplification. If you want to delve deeper into the phenomenon Google "Bandwidth Delay Product" aka BDP) Now, one way around this problem is to make the Receive Window larger for high-latency connections. It's not a perfect solution because it makes each ack packet more "valuable" should it be lost, but it's basically the primary knob we have to turn to improve performance.

    So how to go about it? First off, we need to consider MTU (Maximum Transmission Unit, aka how much can we stuff into a packet?) It's 1460 bytes. Basically it's the packet size of 1500 minus the header (unless you're a DSL user in which case it's smaller. You can Google for the numbers). Therefore the Receive Window should be a multiple of 1460. The default (un-scaled) window size maximum is 65536, but since the window has to be a multiple of MTU, the optimal unscaled size turns out to be 44 * 1460 = 64240. This number will appear later.

    The next question is then How big can/should the Receive Window go? Well, it uses some memory as it grows, and is created for each socket instance, so some estimation is in order. Its value is an order of 2 multiple of 64240, so you can choose values for yourself. I've set mine to 2055680. The tradeoff is obvious. If you expect a high number of simultaneous TCP sessions, this memory usage can get large, fast. For a few dozen clients, you can probably safely set a value as I did. For a few hundred, you need to think about how much memory you can afford.

    To make these changes work, you'll need to enable three other TCP options: timestamps, selective acknowledgements, and window scaling. I'll show you how to do this below.

    That's it.

    What is the effect? Well, if you're running speed tests to your local speedtest server (Speedtest.net is a valuable resource) the Untangle defaults may appear to be okay. That's because the server is local and latency is low. So go ahead and get a number for your connection. Then try it again but with a server that's a few thousand miles away, say on the opposite coast. Your 50mbit/sec connection that you were so pleased with now slows to 3mbit/sec. Or less. That's not acceptable performance, as far as I'm concerned. The problem isn't your connection, it's BDP.

    Now you can't make the proverbial silk purse, but you can make a dramatic improvement. For example, I'm in PA. A Speedtest.net server in Palo Alto ran at ~19mbit/sec with default values and 34mbit/sec with properly adjusted TCP parameters. The latency of that server is 120msec. For a server with twice the latency the improvement is similar, 4.7mbit/sec to 8.6mbit/sec.

    The point of this is, of course, that you're spending $$ for a fast connection, but not getting the corresponding performance from sources more than a few tens of msec away.

    If you want to fix this, here's what you do:

    1) edit /etc/sysctl.conf to include the following:

    Code:
    net.core.rmem_default = 64240
    net.core.rmem_max = 2055680
    net.core.wmem_default = 64240
    net.core.wmem_max = 2055680
    net.ipv4.tcp_timestamps = 1
    net.ipv4.tcp_sack = 1
    net.ipv4.tcp_window_scaling = 1
    Choose your own value for rmem max and wmem max. These are workable for, say, 10-20 workstations and enough memory in your Untangle box.

    2) (The ugly part) Set a cron job to keep these values active.
    As root, run "crontab -e" and add the line:

    Code:
    1,16,31,46 * * * * /sbin/sysctl -p
    Check it with "crontab -l"

    This forces reset of the TCP parameters to your selected values every 15 minutes. This is necessary because Untangle comes along every so often and resets them back to the default values.

    That's it. You're done. Enjoy your increased speed. I suggest that you run some benchmark tests with Speedtest.net at local and distant servers prior to making these changes, and then again at the same sites afterward. It might be helpful for others to post up your results. You can also see before/after parameters at http://www.dslreports.com/tweaks

    Now here's the part that's going to make the Untangle folks squirm. I know why they set the defaults as they do: it's to accommodate large numbers of clients without driving memory usage in the Untangle box through the roof. But here's the thing, if you have roomfuls of people all piped through a single Untangle box, you are essentially FORCED to kill your bandwidth in order to provide connections for all of them. So you have to make a tradeoff of:
    1) Number of users on one box
    2) Available memory
    3) Effective bandwidth to moderately-distant to faraway sites.

    So, if you're in that situation with a high user count, you're basically committed to sub-par performance, even with high-bandwidth connections. IOW, you're not getting what you're paying for. Improve the situation by stuffing as much memory in Untangle as you can and then increasing the Receive Window.

    For those users such as myself with a very low user count, the adjustments can run toward the maximum values without any concern.

    As always, feedback is welcome. Enjoy.

  2. #2
    Untangle Ninja sky-knight's Avatar
    Join Date
    Apr 2008
    Location
    Phoenix, AZ
    Posts
    25,071

    Default

    Except I am getting what I'm paying for, more than what I'm paying for actually. And I didn't reset anything on my box to do it.

    Speedtest.net is NOT a valuable resource, it's an engine to sell anti-spyware software. There are only a handful of servers in that mess that actually have the bandwidth to properly test a connection. To date, I've found ONE server that can actually push things fast enough to test my 10/100 DOCSIS 3.0 cable, Palo Alto.

    If you've got 20mbit or less on your connection, sure knock yourself out on Speedtest.net. If you've got faster, you'll run into its limitations very quickly.
    Rob Sandling, BS:SWE, MCP
    NexgenAppliances.com
    Phone: 866-794-8879 x201
    Email: support@nexgenappliances.com

  3. #3
    Master Untangler
    Join Date
    Nov 2009
    Posts
    106

    Default

    Look, I don't give a rat's ass what speed test server you or anyone else uses. Use whatever. That's not the point. And for that matter, I don't see anything at all about spyware or whatever you're all in a knot about. The servers they have in Philadelphia, NJ, and Brooklyn all can run my connection at full speed -- 60mbit/sec. I don't know how fast the one in Palo Alto is, but it's at least 35mbit/sec at that distance. I WAS USING IT FOR ILLUSTRATION, JEEZ!!! Lighten the freak up.

    You've totally missed the point because you're so wrapped up in something incidental.

  4. #4
    Master Untangler Louisd's Avatar
    Join Date
    Jan 2008
    Location
    Montreal, QC
    Posts
    168

    Default

    GeneralEclectic;

    You might find that your otherwise very valuable contribution is undermined by your aggressive tone.

    It is OK to be in complete disagrement with someone else's comments, but is is not OK to use aggressive tone or improper language (not saying specifically that you did, but I sense that we are going down that road fast).

    One great characteristic of this forum is the fact that most people remain civil in their disagreement, and seldom deride newbies. Please let's keep it that way.

    LD

  5. #5
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,747

    Default

    +1 Louisd.

    It worth saying that these options are disabled for a reason. So I'd take some of his "proper" and "ugly" comments with a grain of salt.

    Enabling TCP scaling and selective ACK proved problematic for many users so we disabled it by default. We may enable it by default in the future as we are now on a later 2.6 kernel which handles them better, but buyer beware. Don't call support if you change this and have issues.

    I don't know why you'd turn on tcp timestamps. You'd want to turn this off if you're nitpicking and want truly maximum speed.

    Also, note this doesn't effect your overall bandwidth or throughput, it just changes the behavior or singular TCP streams (which are used to measure throughput in the case of speedtest.net)
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  6. #6
    Master Untangler
    Join Date
    Apr 2007
    Posts
    641

    Default

    So basically the point of this thread is how to optimize tcp to get the max number per your connection on speedtests...

  7. #7
    Untangler memothejanitor's Avatar
    Join Date
    Nov 2009
    Posts
    97

    Default

    Errr...happy Mother's Day Untangle moms! (if we have any)

  8. #8
    Untangle Ninja
    Join Date
    May 2008
    Posts
    1,286

    Default

    Quote Originally Posted by bigdessert View Post
    So basically the point of this thread is how to optimize tcp to get the max number per your connection on speedtests...
    That seems to be a very important thing to a lot of people around here. Just read all the threads. May be only cosmetic but might help if you have high latency for other things to like remote vpn to a cororate network or something.

    Anybody else tested this? I don't see any gains that are significant. I do have high latency to almost everywhere since it takes 80 to 90 msec to get out of my isps network. I will try for a week or so to see if I get any feeling that things have speeded up or not. Thanks for the effort in looking at this.

    DM: I think ugly was refering to his own hack.

    Don
    Last edited by donhwyo; 05-08-2011 at 01:20 PM.

  9. #9
    Untangle Ninja sky-knight's Avatar
    Join Date
    Apr 2008
    Location
    Phoenix, AZ
    Posts
    25,071

    Default

    But that's just it, the speed tests provide by Speedtest.net are patently inaccurate. I discovered this fact the hard way with my own DOCSIS 3.0 connection. What I posted before wasn't meant to undermine the point of the TCP tuning, it was meant to point out that the service provided by speedtest.net specifically isn't scientific enough to test what is being adjusted. I also wanted to point out that I have several servers in service that get great performance on the connections they use, and not a single one of them has such aggressive hacks made to the underlying operating system.

    Your mileage my vary of course, but I have to point out that the settings used by Untangle are not "ugly" as indicated in the OP. Untangle's default settings in this department are based on years of research and testing.
    Rob Sandling, BS:SWE, MCP
    NexgenAppliances.com
    Phone: 866-794-8879 x201
    Email: support@nexgenappliances.com

  10. #10
    Untangle Ninja YeOldeStonecat's Avatar
    Join Date
    Aug 2007
    Posts
    1,548

    Default

    I used to tune Windows all the time...up through and including XP. Tuning the TCP stack is what my friend Philip started doing for early Windows 95 back when broadband was getting popular, and he started www.speedguide.net. Since Windows was tuned to work on either dial up, or LANs...but not in between.

    But since Vista came out...with auto tune..results of tweaking are greatly diminishing.

    I never focused on tweaking *nix much....but setting your RWIN for 2 million...you should be on a gigabit connection for that. 20-50 megs or so, 512k is more proper...multiple of MSS to be precise, but the optimal setting also depends on the average latency of your connection.


    http://www.speedguide.net/analyzer.php

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2