"Server Load Is Very High" notifications for the last 3 days

[tescophil]

OK, this is kinda interesting..., I have machines that have had SSD's installed for over 10 years now, and never had any issues. The disc usage is static on the machine at 15Gb total on a 32Gb SSD, I appreciate this is being written too constantly, but I'm sure the logging level is way lower that the vast majority of installations.

I have not seen any more 'crazy' spikes since the 29th. The only 'regular increased in load I now can see is at 1:30 AM each day from 0.25 to 3.5, this is obviously not excessive, but again it's nothing to do with traffic volume (this is actually a low point for sessions in a 24h period), so this can only be Untangle performing some sort of scheduled task ?

[sky-knight]

1:30 am every morning sounds like the nightly reports generation, it's pretty DB heavy and will show as a substantial load increase while it runs.

If you look closely at your disk usage, you should see a saw tooth pattern, as the data over the day accumulates, and then this process purges stuff that is configured to be removed at the end of the day. The drop in the saw pattern will coincide with the CPU load of the report purge.

Random IO errors are expected on 10 year old SSDs, they too will manifest in strange ways. One of them is CPU load. But it can also act like bad RAM at times... of course bad RAM can also show as a bad disk! Troubleshooting Linux is hard sometimes.

Solid State Drives do NOT live forever, and they fail from being written to. Untangle will generate a substantial write load even when configured conservatively. Still, a properly configured system should still support 20 years of SSD life. But that assumes you have a good SSD, there are a HORDE of cheap ones out there.

[MNTech68]

I appreciate everyone's input here, but taking into account the wide variety of hardware being reported by those of us with issues, I find it really unlikely that it's a BIOS/SSD/CPU issue. This screams a software level issue.

I'm running RAID 10 15K SAS Array, with hardware monitoring and pre-fail detection. Zero errors at the hardware level. Same goes for memory. Fully Registered DIMM's, Advanced ECC features, etc. When there's a failure, the hardware level monitoring almost always picks up the failure before it impacts the OS level. It's not infallible, but it's a heck of a step up from an Alibaba J1900 box.

My install isn't that old either. I believe it was a fresh install at v14, because at the time I changed to Intel NICs, added RAM and moved from RAID 6 to RAID 10.

[donhwyo]

My install isn't that old either. I believe it was a fresh install at v14, because at the time I changed to Intel NICs, added RAM and moved from RAID 6 to RAID 10.

14 was based on debian 9. So a clean install might help? Might not? Doing nothing is not going to help for sure.

[dashpuppy]

14 was based on debian 9. So a clean install might help? Might not? Doing nothing is not going to help for sure.

This thread is going around and around, THe post said he did an upgrade from 14 to 16 and now is seeing issues. We have all suggested SAVE the config, do a fresh install then import config, but he doesn't want to do it. SO It's like trying to get the horse to drink water when it doesn't want to. The thread is a dead end. It will be locked soon.

[sky-knight]

Yeah, because every thread talking about intermittent CPU issues has a different cause. I have no systems doing things like this... I did have a few misbehave on the most recent upgrade but as I said before... they were TINY B1S instances in Azure, that's 1 CPU! Anything that makes those things think sets off that alarm.

V14 is on the wrong side of a kernel upgrade, these are the big upgrades that can cause hardware level compatibility problems that CAN and DO take years to surface sometimes.

Today if your Untangle is doing something strange and random, that's impossible to troubleshoot... AND it started its life v15.0 and prior... You should try a nuke and pave, because it can and does often resolve things that got mucked up in the OS update that came with v15.1.

Assuming v17 is the next Debian OS... we'll be doing this AGAIN.

[dashpuppy]

Yeah, because every thread talking about intermittent CPU issues has a different cause. I have no systems doing things like this... I did have a few misbehave on the most recent upgrade but as I said before... they were TINY B1S instances in Azure, that's 1 CPU! Anything that makes those things think sets off that alarm.

Its like upgrading windows xp to windows 7, sure it works but if there are wierd issues' its a WIPE and reload and start from scratch with new drivers & updates !

[sky-knight]

Its like upgrading windows xp to windows 7, sure it works but if there are wierd issues' its a WIPE and reload and start from scratch with new drivers & updates !

Debian 9 to 10 to 11 is more akin to Windows 10 feature updates, but yeah.

[MNTech68]

This thread is going around and around, THe post said he did an upgrade from 14 to 16 and now is seeing issues. We have all suggested SAVE the config, do a fresh install then import config, but he doesn't want to do it. SO It's like trying to get the horse to drink water when it doesn't want to. The thread is a dead end. It will be locked soon.

Calm down. I have a support ticket open and they are actively looking at it since prior to this thread becoming a "dead end". Wiping the server is poor practice given that there are people involved who want to understand the root cause. Untangle, and Debian for that matter wouldn't be where they are today without people who take the time to actually root out bugs and work through them.

[sky-knight]

Calm down. I have a support ticket open and they are actively looking at it since prior to this thread becoming a "dead end". Wiping the server is poor practice given that there are people involved who want to understand the root cause. Untangle, and Debian for that matter wouldn't be where they are today without people who take the time to actually root out bugs and work through them.

If you've got the time for it, rock on!