I have another fun one!

[sky-knight]

Minimal network traffic is transiting Untangle, the Internal interface and external interface are responding but it takes minutes to load the UI. SSH takes a good three minutes to login.

http://pastebin.com/jg2D3Ezn

Turn off reports, WHAM box is back to life. Turn reports back on, and I'm good for another random length of time. This is the box that is here at Intouch, the hardware is aging, and old mini-PC that was built 5 years ago. I suspect the drive may be going.

Anyone at Untangle got any bright ideas?

Network session count shows between 12-20 sessions during this particular window. So we're not generating a ton of requests for the logging to be a real concern.

Data Retention for Reports? 7 days.
Reports Retention? 30 days.

380 days of uptime, I don't want to reboot her! I don't want to take her down for a drive diagnostic. I know I should, but she's been so happy for so long!

[dwasserman]

du?
Purge the postgres db?

[sky-knight]

root@untangle# ~ # df -h [root @ untangle]
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 74G 4.1G 66G 6% /
tmpfs 474M 0 474M 0% /lib/init/rw
udev 10M 676K 9.4M 7% /dev
tmpfs 474M 0 474M 0% /dev/shm
shmfs 474M 12K 474M 1% /lib/init/rw/splashy
/dev/sda5 71G 180M 68G 1% /data
root@untangle# ~ # [root @ untangle]

Storage isn't the problem.

[dbunyard]

Hard drive problems would make sense as the top output you posted doesn't seem to have anything hitting on the CPU very hard. Did you try an "iotop" by chance to see if there is any clues in there? Wait does "iotop" even exist in Untangle? I didn't check that...

[dmorris]

I use this alot to see what postgres is doing:

watch -n1 "/usr/bin/psql -U postgres uvm -c\"SELECT datname as db,usename as user,procpid as pid,waiting as wait,round(extract('epoch' from (now() - query_start))) as sec,current_query FROM pg_stat_activity WHERE current_query not like '%pg_stat_activity%'\""

[dbunyard]

That picture nearly made me cover my laptop in water...I should know better than to click the links and pictures you post while drinking.

[dbunyard]

Nevermind, I'm an idiot...

[dmorris]

Yeah, but you ran that on your fedora server
which probably isn't running postgres.

That command just shows you what postgres is currently doing. Useful for debugging 'what the hell is going on' if its disk/reports related.

[dbunyard]

Yeah, but you ran that on your fedora server
which probably isn't running postgres.

I realized that...It's been a long week already. Just got back from a 2 week vacation and still trying to drag my mind out of vacation mode... You can just ignore me.

[sky-knight]

Next time I get a load spike I'll take a peek.

Reports have always kicked this machine's butt, hit 2am and I loose internet for a good 30min while it sorts things out. The generation I'm ok with nuking the box, but having the thing just randomly load up in the middle of the day is frustrating.

Still, it happens once, perhaps twice a month. I've been tracking it since 9.0, and it seems to have gotten better with each release. But with such an intermittent issue, troubleshooting it is a real PITA.

I really should just get a better Untangle server, but this little guy has been running Untangle since I found Untangle. It has hit over a year in uptime three times in its life. In alot of ways, it deserves to be in some sort of Untangle museum. All this time and only had to pave it once, and that was just because I wanted some peace of mind. That 7.x to 8.x upgrade was a real whopper. Box just didn't quite make it.