Results 1 to 7 of 7
  1. #1
    Untangler
    Join Date
    Jan 2021
    Posts
    94

    Question Java causing high Swap usage

    I received some alerts about high swap usage. I SSH'd in and ran 'top' and saw java was using an abnormal amount of memory and causing high swap usage. I killed the process and it eventually restarted itself. Any ideas what could have caused this?

    System-Swap_Usage-28.08.2022-2021.png

  2. #2
    Untangle Ninja jcoehoorn's Avatar
    Join Date
    Mar 2010
    Location
    York, NE
    Posts
    1,969

    Default

    How much memory does the system have? For how many devices?

    The UVM that handles all the filtering and rules you set up is a java app. When you kill it, you disable all Untangle's security and logging on the system until it restarts. The networking stack itself is linux, so the internet still works, but nothing is checked or logged. Additionally, the logs that might have told us what caused the problem are now lost.

    It is normal for Untangle to use a certain amount of swap. However, IME, sudden spikes like this are caused by a few things:

    1. Someone started using a torrent client in a way that started hundreds to thousands of sessions very quickly. This is especially likely if your system is designed to block this, as the torrent clients just don't. give. up. ever. They'll keep creating more and more sessions trying more and more ways to get around your filter, eventually creating a DoS situation.

    2. Nightly reports. This doesn't seem like your situation, since the times are wrong, but I'm including it for completeness. The nightly report job can be a beast and overwhelm a system that otherwise is working fine.

    3. A RAM stick or hard drive that is beginning to fail.

    4. Turning on too many of the intrusion prevention rules. Most of them are meant to be left off, but sometimes an admin will go look and think, "Wow, look at all these option I could checking/preventing", and turn on much more than their system can handle. You'll get a similar result if you go turn on logging for all the web filter categories and similar.

    5. A failing network card on a computer somewhere on the same layer-2 network segment can sometimes create a bunch of "noise" on the network, which the uvm must then process.

    6. A real actual denial of service attack

    7. A routing loop
    gravenscroft likes this.
    Five time Microsoft ASP.Net MVP managing a Lenovo RD330 / E5-2420 / 16GB with Untangle 16.5.2 to protect a 1Gbps fiber link for ~450 residential college students and associated staff and faculty

  3. #3
    Untangler
    Join Date
    Jan 2021
    Posts
    94

    Default

    Quote Originally Posted by jcoehoorn View Post
    How much memory does the system have? For how many devices?

    The UVM that handles all the filtering and rules you set up is a java app. When you kill it, you disable all Untangle's security and logging on the system until it restarts. The networking stack itself is linux, so the internet still works, but nothing is checked or logged. Additionally, the logs that might have told us what caused the problem are now lost.

    It is normal for Untangle to use a certain amount of swap. However, IME, sudden spikes like this are caused by a few things:

    1. Someone started using a torrent client in a way that started hundreds to thousands of sessions very quickly. This is especially likely if your system is designed to block this, as the torrent clients just don't. give. up. ever. They'll keep creating more and more sessions trying more and more ways to get around your filter, eventually creating a DoS situation.

    2. Nightly reports. This doesn't seem like your situation, since the times are wrong, but I'm including it for completeness. The nightly report job can be a beast and overwhelm a system that otherwise is working fine.

    3. A RAM stick or hard drive that is beginning to fail.

    4. Turning on too many of the intrusion prevention rules. Most of them are meant to be left off, but sometimes an admin will go look and think, "Wow, look at all these option I could checking/preventing", and turn on much more than their system can handle. You'll get a similar result if you go turn on logging for all the web filter categories and similar.

    5. A failing network card on a computer somewhere on the same layer-2 network segment can sometimes create a bunch of "noise" on the network, which the uvm must then process.

    6. A real actual denial of service attack

    7. A routing loop
    Thanks for your reply. My untangle box has 8GB RAM 64GB SSD and only 15 devices. No Torrenting that day (wouldn't Shield prevent so many sessions?). My IP rules are below, I figured I had enough RAM to handle it. I did make a change recently to IP. I changed "When to scan" to "After other network processing" instead of before. That was a couple days ago. I was getting so many entries in the log, and I have nothing exposed so it wasn't very useful.

    2022-08-29 17_28_45-Untangle - untangle715 - Brave.png

  4. #4
    Untangler
    Join Date
    Jan 2021
    Posts
    94

    Default

    I started to get another spike in swap usage and I changed IP "When to scan" back to "Before other network processing" and swap immediately dropped. Does anyone have IP setting "When to scan" set to "After other network processing" experience this same issue? Or does no one have it set like that?

  5. #5
    Untangle Ninja jcoehoorn's Avatar
    Join Date
    Mar 2010
    Location
    York, NE
    Posts
    1,969

    Default

    That's one of those settings that forces a uvm restart, such that it can have an immediate effect on what swap you're using right now, but might not have any meaning or even make things worse for how much swap you use over time.
    gravenscroft likes this.
    Five time Microsoft ASP.Net MVP managing a Lenovo RD330 / E5-2420 / 16GB with Untangle 16.5.2 to protect a 1Gbps fiber link for ~450 residential college students and associated staff and faculty

  6. #6
    That Which Lurks Below
    Join Date
    Jul 2018
    Posts
    143

    Default

    Quote Originally Posted by MP715 View Post
    Does anyone have IP setting "When to scan" set to "After other network processing" experience this same issue? Or does no one have it set like that?
    I've never seen that setting changed from the default, which is 'before'. When set to 'after', traffic is only fed to IPS once all the other apps have had a chance to scan it. It's possible that this is why you're seeing a spike in resource usage: a lot of traffic that used to be blocked by IPS before it was handled by the UVM is now being processed by all the other applications. Each time an app scans traffic, it's using resources; moreso if it's actively blocking something.

    Passing traffic to the apps when it would normally be blocked by IPS also increases Reports data generation, which further requires processing resources.

    Quote Originally Posted by jcoehoorn View Post
    That's one of those settings that forces a uvm restart, such that it can have an immediate effect on what swap you're using right now, but might not have any meaning or even make things worse for how much swap you use over time.
    This is also an excellent point. How long after changing the setting are you seeing high swap usage?
    Græme Ravenscroft • Technical Marketing Engineer
    ('gram', like the unit of measurement)
    he/him
    Please don't reboot your NGFW.
    How can we make Arista ETM products better?

  7. #7
    Untangler
    Join Date
    Jan 2021
    Posts
    94

    Default

    Quote Originally Posted by gravenscroft View Post
    This is also an excellent point. How long after changing the setting are you seeing high swap usage?
    I saw the spike at least 48 hours after the change. I made the change after reading this All IPS was doing was "blocking" traffic that was already blocked. I have nothing exposed so it was not that useful. I figured changing to after would be more beneficial to me.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2