Results 1 to 8 of 8

Thread: Scary

  1. #1
    Master Untangler jcoehoorn's Avatar
    Join Date
    Mar 2010
    Location
    York, NE
    Posts
    712

    Default Scary



    I'll wait for a moment while that sinks in....

    And this was on it's way down: I caught the load listed at 252 at it's peak for the 1 minute average! I took this screen shot because it's where the 15 minute average maxed out. I had the 5 minute average over 100 briefly. The server is no slouch, either: 4 real cores+ hyperthreading, 8gb ram. But then, it's disk i/o that's causing this.

    Needless to say I'll be calling support in the morning (this is from just a few minutes ago). I've already checked the usual suspects (attack blocker, etc) and I didn't see anything stand out to cause this kind of mayhem. Support already applied the auto-vacuuming patch. That seemed to fix everything... for just over a day

    Now for honesty's sake I'll disclose that it was up that high because I was running du on the box, but in my defense the reason I was running du is that 5 min load was already up in the 40's and 1 min in the 60's. It looked like it was all due to disk i/o, and I wanted to see where it was going/coming from. So I may as well post the interesting results from that du:

    /var/spool/squid 11GB (I'm not using Web Cache right now on any rack)
    /var/lib/postgresql 17GB (the auto-vacuum patch was just recently installed, but disk use is still high here)
    Last edited by jcoehoorn; 01-26-2012 at 08:22 PM.
    Five time Microsoft ASP.Net MVP managing a Lenovo RD330 / E5-2420 / 8GB with Untangle 10.1-1 to protect 60+40Mbits for 450+ residential college students and associated staff and faculty

  2. #2
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Mateo, CA
    Posts
    12,640

    Default

    There is no "auto-vacuuming patch."
    I made some changes but just tuning changes.

    I'd guess something is using all your memory, because havin 7260megs used is WAY too much.
    Its probably saturating the disk i/o because it is swapping.
    Did you email support with your UID?

    DO NOT REBOOT.
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  3. #3
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Mateo, CA
    Posts
    12,640

    Default

    Does your UID start with 84c?

    If so, I looked at that server again (it wasn't rebooted YAY!)
    Unfortunately it wasn't using all that memory like it was before and had recovered, but I think I was able to figure out what happened.
    I've applied a change that should correct it if my theory is right.
    Please email support@untangle.com or my @ dmorris@untangle.com if it happens again as soon as you notice.
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  4. #4
    Untangle Ninja
    WebFooL's Avatar
    Join Date
    Jan 2009
    Location
    Sweden (Eskilstuna)
    Posts
    4,478

    Default

    That is what I call good support!
    Keep it up Dmorris you are a role-model for all of us

  5. #5
    Master Untangler jcoehoorn's Avatar
    Join Date
    Mar 2010
    Location
    York, NE
    Posts
    712

    Default

    dmorris, you rock! It's running much better so far this morning. The true test won't come until after lunch, though: there's normally not nearly as much traffic in the morning.

    The reason memory was lower was because I stopped the uvm and let the network run unfiltered for about 1/2, until the 15 min load average was down to something reasonable again, before starting up the uvm. Can you share with me what you changed?
    Five time Microsoft ASP.Net MVP managing a Lenovo RD330 / E5-2420 / 8GB with Untangle 10.1-1 to protect 60+40Mbits for 450+ residential college students and associated staff and faculty

  6. #6
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Mateo, CA
    Posts
    12,640

    Default

    Well, I'm just glad you didn't reboot!
    I've still got an SSH window open to it, and I'm keeping an eye on it.

    I will PM with you with details.
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  7. #7
    Untangle Ninja dwasserman's Avatar
    Join Date
    Jun 2008
    Location
    Argentina
    Posts
    4,137

    Default

    Quote Originally Posted by dmorris View Post
    Well, I'm just glad you didn't reboot!
    I've still got an SSH window open to it, and I'm keeping an eye on it.

    I will PM with you with details.
    Knowledge is power
    The world is divided into 10 kinds of people, who know binary and those not

  8. #8
    Master Untangler jcoehoorn's Avatar
    Join Date
    Mar 2010
    Location
    York, NE
    Posts
    712

    Default

    Just to follow up for anyone who finds this thread in the future... it's just over a month later and things are running much better now, and have been consistently for most of that period. From what I understand the fixes they made for this box are set to be included in the stock release.

    I will say that my memory use is still consistently high: it shows 6543 used right now. But I'm okay with that, because load is consistently low and latency introduced by untangle has been minimal. I'd still like a way for the daily reports to show me some information about the average latency introduced to packets as they pass through (or wait to pass through) the uvm, perhaps broken down by QoS level, as this would have much more meaning to me than the load or memory metrics.
    Last edited by jcoehoorn; 03-08-2012 at 09:35 AM.
    Five time Microsoft ASP.Net MVP managing a Lenovo RD330 / E5-2420 / 8GB with Untangle 10.1-1 to protect 60+40Mbits for 450+ residential college students and associated staff and faculty

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2