"Server Load Is Very High" notifications for the last 3 days

[tescophil]

Its an Intel(R) Celeron(R) CPU J1900 @ 1.99GHz, SSD and Dual Intel NIC's.

The hardware is not the problem here, RAM is 41% free, and the Load average varies betrween 0.25 and 1 when the internet connection is fully saturated. Never seen loads above 1.5 before this, now seeing spikes of 30 and 60, or to put it another way a 24,000% increase in CPU load.

I put a ticket in for this, and after analysis by a 'specialist' they concluded it was traffic related, because lots of sessions were created 'about the same time' as the CPU load spike. When I looked at the actual data, I found a 63 second period where all traffic stopped, no new sessions created etc., then a spike in sessions _after_ this lockup.., so the CPU load caused the spike in sessions, not the other way around.

I don't need better hardware, just better software

[sky-knight]

Its an Intel(R) Celeron(R) CPU J1900 @ 1.99GHz, SSD and Dual Intel NIC's.

The hardware is not the problem here, RAM is 41% free, and the Load average varies betrween 0.25 and 1 when the internet connection is fully saturated. Never seen loads above 1.5 before this, now seeing spikes of 30 and 60, or to put it another way a 24,000% increase in CPU load.

I put a ticket in for this, and after analysis by a 'specialist' they concluded it was traffic related, because lots of sessions were created 'about the same time' as the CPU load spike. When I looked at the actual data, I found a 63 second period where all traffic stopped, no new sessions created etc., then a spike in sessions _after_ this lockup.., so the CPU load caused the spike in sessions, not the other way around.

I don't need better hardware, just better software

And, there was someone else in here with similar symptoms that was dealing with a LOG4J scan barrage causing the same thing.

You could, and very likely do have some sort of bot on your network or, a ton of bots outside your network scanning into your network.

Do you have a web server of some kind behind that Untangle?

If you've got a spike in network sessions, you're going to see the CPU go crazy.

[dashpuppy]

Its an Intel(R) Celeron(R) CPU J1900 @ 1.99GHz, SSD and Dual Intel NIC's.

The hardware is not the problem here, RAM is 41% free, and the Load average varies betrween 0.25 and 1 when the internet connection is fully saturated. Never seen loads above 1.5 before this, now seeing spikes of 30 and 60, or to put it another way a 24,000% increase in CPU load.

I put a ticket in for this, and after analysis by a 'specialist' they concluded it was traffic related, because lots of sessions were created 'about the same time' as the CPU load spike. When I looked at the actual data, I found a 63 second period where all traffic stopped, no new sessions created etc., then a spike in sessions _after_ this lockup.., so the CPU load caused the spike in sessions, not the other way around.

I don't need better hardware, just better software

I had one of these boxes ran Untangle on it, it didn't handle it to well, What did help was updating the bios. Is the picture i have attached the box you are using ?fmp07-j1900.jpg

[tescophil]

Hi, no, that's not the PC I'm using....(correction, Quad Intel Gb NIC's), apart from anything else, the system runs perfectly 99.9% of the time, with a very low load average of 0.25. Its only since 16.4.1 that I've seen these spikes in CPU usage.

I don't know how many more times I have to say this, but there is NO change in the network traffic when these spikes happen. I've been running Untangle for over 12 years now, so I'm not unfamiliar with monitoring my own network.., this is not a network traffic / session / load issue, its something in the processing stack very occasionally getting stuck in a very tight loop. This might be related to the _type_ of traffic, but its got nothing to do with the volume.

I appreciate that I'll probably never get a solution to this one, and just have to keep my fingers crossed it doesn't happen too often, and may be 'fixed' in a later update.

[dashpuppy]

Hi, no, that's not the PC I'm using....(correction, Quad Intel Gb NIC's), apart from anything else, the system runs perfectly 99.9% of the time, with a very low load average of 0.25. Its only since 16.4.1 that I've seen these spikes in CPU usage.

I don't know how many more times I have to say this, but there is NO change in the network traffic when these spikes happen. I've been running Untangle for over 12 years now, so I'm not unfamiliar with monitoring my own network.., this is not a network traffic / session / load issue, its something in the processing stack very occasionally getting stuck in a very tight loop. This might be related to the _type_ of traffic, but its got nothing to do with the volume.

I appreciate that I'll probably never get a solution to this one, and just have to keep my fingers crossed it doesn't happen too often, and may be 'fixed' in a later update.

Why not do a wipe and re-load ? Save your config, reload - import. Test...

[sky-knight]

How old is this installation? And what SSD are you using?

[tescophil]

11 Months old, M1 SSD, can't recall the make.. but so what..., the only thing that has changed in my setup is the software version of Untangle to 16.4.1 and that's when this issue started.

[dashpuppy]

11 Months old, M1 SSD, can't recall the make.. but so what..., the only thing that has changed in my setup is the software version of Untangle to 16.4.1 and that's when this issue started.

Why not do a wipe and re-load ? Save your config, reload - import. Test...

[tescophil]

I don't think a wipe and reload is constructive, never had to do than in 12 years of running Untangle. I also have no way to reproduce the issue...

[sky-knight]

11 Months old, M1 SSD, can't recall the make.. but so what..., the only thing that has changed in my setup is the software version of Untangle to 16.4.1 and that's when this issue started.

Because failing SSDs can do this... A reinstall might actually kill it. Untangle's logging is NOT kind to an SSD, and if the garbage collection on the SSD falls behind, you're going to see it as intermittent CPU spikes as the drive IO locks to protect itself.

Formatting the SSD will also flush the tasks the firmware has to perform. Which can reset the system too.