Hard disk filling up, before 9.x had 80%+ free

[far182]

I believe it was 9.0, a SNMP monitoring bug caused the log files to fill up the drive. We worked with support and got a lot of it removed, but not all. Since that time, the drive has been filling up. Currently we are 31% free.

Looking at /var/log I have removed the *.gz files.

Looking at /var/log/uvm I find some VERY huge log files. Is this normal?

-rw-r--r-- 1 root adm 5120068 2011-04-10 03:55 35.log.1
-rw-r--r-- 1 root adm 5120081 2010-11-25 15:29 35.log.2
-rw-r--r-- 1 root adm 5120021 2010-07-09 02:19 35.log.3
-rw-r--r-- 1 root adm 5120009 2010-02-18 16:47 35.log.4
-rw-r--r-- 1 root adm 5128441 2011-06-30 11:19 4.log.1
-rw-r--r-- 1 root adm 5123460 2011-06-20 17:51 4.log.2
-rw-r--r-- 1 root adm 5121435 2011-06-15 08:04 4.log.3
-rw-r--r-- 1 root adm 5121773 2011-06-07 10:44 4.log.4
-rw-r--r-- 1 root adm 5121407 2011-05-26 08:56 4.log.5
-rw-r--r-- 1 root adm 5126183 2011-05-17 13:30 4.log.6
-rw-r--r-- 1 root adm 5124978 2011-05-09 08:06 4.log.7
-rw-r--r-- 1 root adm 5122951 2011-04-29 12:01 4.log.8
-rw-r--r-- 1 root adm 5127143 2011-04-25 09:56 4.log.9
-rw-r--r-- 1 root adm 5120051 2011-05-23 02:53 62.log.1
-rw-r--r-- 1 root adm 5120523 2011-03-20 09:51 62.log.2
-rw-r--r-- 1 root adm 5121163 2011-01-23 14:23 62.log.3
-rw-r--r-- 1 root adm 108219 2011-11-30 10:19 apt.log
-rw-r--r-- 1 root adm 476660 2011-11-30 10:21 console.log
-rw-r----- 1 root adm 126787134 2011-11-11 09:35 events.log
-rw-r--r-- 1 root adm 1749181522 2011-11-30 10:24 gc.log
-rw-r--r-- 1 root adm 78454 2011-06-28 11:31 iptables.log
-rw-rw-r-- 1 root adm 2863659 2011-11-30 10:24 monit.log
-rw-r----- 1 root adm 743879 2011-11-30 08:11 node-13.log
-rw-r----- 1 root adm 322 2011-10-14 13:08 node-17.log
-rw-r----- 1 root adm 11105 2011-07-21 13:45 node-18.log
-rw-r----- 1 root adm 8460 2011-07-17 19:34 node-19.log
-rw-r----- 1 root adm 2598 2011-10-21 15:01 node-1.log
-rw-r----- 1 root adm 131 2011-08-25 07:50 node-24.log
-rw-r----- 1 root adm 6230688 2011-11-30 10:11 node-35.log
-rw-r----- 1 root adm 771984 2011-11-30 08:12 node-36.log
-rw-r----- 1 root adm 1394922 2011-11-29 07:55 node-38.log
-rw-r----- 1 root adm 24464313 2011-11-29 07:55 node-39.log
-rw-r----- 1 root adm 2957 2011-09-30 12:30 node-3.log
-rw-r----- 1 root adm 768902 2011-11-29 19:31 node-40.log
-rw-r----- 1 root adm 585451 2011-11-29 15:09 node-43.log
-rw-r----- 1 root adm 739953 2011-11-30 05:20 node-44.log
-rw-r----- 1 root adm 101207 2011-11-30 00:30 node-46.log
-rw-r----- 1 root adm 368484 2011-11-30 03:31 node-47.log
-rw-r----- 1 root adm 554064 2011-11-30 09:52 node-48.log
-rw-r----- 1 root adm 69419419 2011-11-30 10:15 node-4.log
-rw-r----- 1 root adm 2902 2011-11-14 14:41 node-51.log
-rw-r----- 1 root adm 756261 2011-11-30 08:12 node-52.log
-rw-r----- 1 root adm 1138 2011-10-06 13:06 node-54.log
-rw-r----- 1 root adm 74871 2011-11-02 08:29 node-55.log
-rw-r----- 1 root adm 47362 2011-11-21 10:40 node-56.log
-rw-r----- 1 root adm 25851 2011-11-30 01:52 node-5.log
-rw-r----- 1 root adm 16184 2011-11-30 07:59 node-60.log
-rw-r----- 1 root adm 84989 2011-11-29 16:30 node-61.log
-rw-r----- 1 root adm 872805 2011-11-30 05:45 node-62.log
-rw-r----- 1 root adm 15654 2011-11-28 08:41 node-63.log
-rw-r----- 1 root adm 88417 2011-08-30 08:01 node-7.log
-rw-r----- 1 root adm 243 2011-10-14 13:08 node-81.log
-rw-r----- 1 root adm 9639899808 2011-11-30 10:24 node-82.log
-rw-r----- 1 root adm 516 2011-07-21 13:45 node-83.log
-rw-r----- 1 root adm 308052 2011-11-30 08:03 node-84.log
-rw-r----- 1 root adm 196 2011-07-05 15:15 node-85.log
-rw-r----- 1 root adm 196 2011-07-05 15:16 node-86.log
-rw-r----- 1 root adm 213 2011-09-22 16:49 node-89.log
-rw-r----- 1 root adm 345 2011-09-22 16:28 node-90.log
-rw-r----- 1 root adm 3959 2011-07-24 03:11 node-92.log
-rw-r----- 1 root adm 10684524737 2011-11-30 10:24 nodes.log
-rw-r--r-- 1 root adm 41764 2011-10-04 08:46 openvpn.log
-rw-r--r-- 1 root adm 155133 2011-11-30 03:01 reporter.log
-rw-r--r-- 1 root adm 512009 2009-09-14 02:08 reporter.log.2
-rw-r--r-- 1 root adm 512019 2009-08-31 02:43 reporter.log.3
-rw-r--r-- 1 root adm 512097 2009-08-16 02:28 reporter.log.4
-rw-r--r-- 1 root adm 512122 2009-08-02 01:57 reporter.log.5
-rw-r--r-- 1 root adm 512071 2009-07-18 02:32 reporter.log.6
-rw-r--r-- 1 root adm 512032 2009-07-03 02:32 reporter.log.7
-rw-r--r-- 1 root adm 119051 2010-03-11 15:38 report.log
-rw-r--r-- 1 root adm 3245 2010-08-17 10:35 restore.log
-rw-r--r-- 1 root adm 0 2011-07-25 01:43 schema.log
-rw-r--r-- 1 root adm 182 2009-06-19 08:42 schema.log.0
-rw-r--r-- 1 root adm 180 2009-06-19 08:42 schema.log.2
-rw-r--r-- 1 root adm 574 2009-06-19 08:42 schema.log.3
-rw-r--r-- 1 root adm 181 2009-06-19 08:42 schema.log.4
-rw-r--r-- 1 root adm 178 2009-06-19 08:42 schema.log.5
-rw-r--r-- 1 root adm 179 2009-06-19 08:42 schema.log.6
-rw-r--r-- 1 root adm 179 2009-06-19 08:42 schema.log.7
-rw-r--r-- 1 root adm 1065032 2009-06-19 08:42 schema.log.8
-rw-r--r-- 1 root adm 14357665 2011-11-30 10:24 uvm.log
-rw-r--r-- 1 root adm 526018 2011-06-29 07:32 uvm.log.1
-rw-r--r-- 1 root adm 512094 2011-06-23 01:37 uvm.log.2
-rw-r--r-- 1 root adm 512013 2011-06-14 00:39 uvm.log.3
-rw-r--r-- 1 root adm 512097 2011-06-04 03:51 uvm.log.4
-rw-r--r-- 1 root adm 512035 2011-05-25 02:04 uvm.log.5
-rw-r--r-- 1 root adm 512072 2011-05-14 13:57 uvm.log.6
-rw-r--r-- 1 root adm 512056 2011-05-05 11:10 uvm.log.7
-rw-r--r-- 1 root adm 512091 2011-04-26 15:18 uvm.log.8
-rw-r--r-- 1 root adm 512097 2011-03-22 17:50 uvm.log.9
-rw-r--r-- 1 root adm 398169 2011-11-30 10:24 wrapper.log

[far182]

I took a peak into nodes.log (huge to look through), and I find endless entries that look like this:

Jul 5 15:26:02 localhost node-82: [SplitDImpl] <NT1876832420> WARN No blinger found for interface: 2
Jul 5 15:26:03 localhost node-82: [SplitDImpl] <NT1876832421> WARN No blinger found for interface: 2
Jul 5 15:26:03 localhost node-82: [SplitDImpl] <NT1876832422> WARN No blinger found for interface: 2
Jul 5 15:26:03 localhost node-82: [SplitDImpl] <NT1876832423> WARN No blinger found for interface: 2
Jul 5 15:26:03 localhost node-82: [SplitDImpl] <NT1876832424> WARN No blinger found for interface: 2
Jul 5 15:26:03 localhost node-82: [SplitDImpl] <NT1876832425> WARN No blinger found for interface: 2
Jul 5 15:26:03 localhost node-82: [SplitDImpl] <NT1876832426> WARN No blinger found for interface: 2
Jul 5 15:26:04 localhost node-82: [SplitDImpl] <NT1876832427> WARN No blinger found for interface: 2
Jul 5 15:26:04 localhost node-82: [SplitDImpl] <NT1876832428> WARN No blinger found for interface: 2
Jul 5 15:26:04 localhost node-82: [SplitDImpl] <NT1876832429> WARN No blinger found for interface: 2
Jul 5 15:26:04 localhost node-82: [SplitDImpl] <NT1876832430> WARN No blinger found for interface: 2
Jul 5 15:26:04 localhost node-82: [SplitDImpl] <NT1876832431> WARN No blinger found for interface: 2
Jul 5 15:26:04 localhost node-82: [SplitDImpl] <NT1876832432> WARN No blinger found for interface: 2
Jul 5 15:26:05 localhost node-82: [SplitDImpl] <NT1876832433> WARN No blinger found for interface: 2
Jul 5 15:26:05 localhost node-82: [SplitDImpl] <NT1876832434> WARN No blinger found for interface: 2
Jul 5 15:26:05 localhost node-82: [SplitDImpl] <NT1876832435> WARN No blinger found for interface: 2
Jul 5 15:26:05 localhost node-82: [SplitDImpl] <NT1876832436> WARN No blinger found for interface: 2
Jul 5 15:26:05 localhost node-82: [SplitDImpl] <NT1876832437> WARN No blinger found for interface: 2
Jul 5 15:26:05 localhost node-82: [SplitDImpl] <NT1876832438> WARN No blinger found for interface: 2
Jul 5 15:26:05 localhost node-82: [SplitDImpl] <NT1876832439> WARN No blinger found for interface: 2

[sky-knight]

curl http://www.untangle.com/download/patches/generic/diskuse.sh | dash

Past the output here.

[far182]

Thanks sky. Here is the result:

~ # curl http://www.untangle.com/download/pat...ric/diskuse.sh | dash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 793 100 793 0 0 5685 0 --:--:-- --:--:-- --:--:-- 0
Disk Free Status
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 71G 45G 24G 66% /
tmpfs 1.5G 0 1.5G 0% /lib/init/rw
udev 10M 692K 9.4M 7% /dev
tmpfs 1.5G 0 1.5G 0% /dev/shm
/dev/sda5 16M 1.2M 14M 8% /data
Top 20 Tables
relname | reltuples | relpages
---------------------+-------------+----------
pl_stats | 2.74612e+06 | 65532
pl_endp | 2.76304e+06 | 61922
sessions_2011_11_17 | 896808 | 35218
n_firewall_evt | 2.45782e+06 | 34881
sessions_2011_11_10 | 870432 | 34188
sessions_2011_11_29 | 870455 | 33113
sessions_2011_11_16 | 829379 | 32846
sessions_2011_11_28 | 843072 | 32493
sessions_2011_11_14 | 820523 | 32072
sessions_2011_11_21 | 845622 | 31957
sessions_2011_11_15 | 811962 | 31945
sessions_2011_11_22 | 826167 | 31891
sessions_2011_11_09 | 804723 | 31589
sessions_2011_11_18 | 803483 | 31150
sessions_2011_11_11 | 788747 | 31044
sessions_2011_11_23 | 747033 | 30009
sessions_2011_11_24 | 740797 | 29884
sessions_2011_11_12 | 729990 | 29164
pl_endp_sid_idx | 2.76304e+06 | 28988
sessions_2011_11_13 | 739484 | 28973
(20 rows)

Oldest record in database
2011-11-27 00:00:00.01

Disk Use of /var/log - Top 10 files
21949464 /var/log/uvm
19840 /var/log/apache2
5636 /var/log/auth.log.1
3752 /var/log/atop.log.9
3596 /var/log/atop.log.2
3552 /var/log/atop.log.14
3540 /var/log/atop.log.8
3520 /var/log/atop.log.10
3492 /var/log/atop.log.6
3448 /var/log/atop.log.1
Disk Use of /var/log/uvm - Top 10 files
10459552 /var/log/uvm/nodes.log
9438316 /var/log/uvm/node-82.log
1712792 /var/log/uvm/gc.log
123944 /var/log/uvm/events.log
67956 /var/log/uvm/node-4.log
23920 /var/log/uvm/node-39.log
14056 /var/log/uvm/uvm.log
6104 /var/log/uvm/node-35.log
5024 /var/log/uvm/4.log.1
5020 /var/log/uvm/4.log.9
Disk Use of postgres db files
13G /var/lib/postgresql/8.3
Disk Use of /usr/share/untangle/web/reports/ (old reports)
659M /usr/share/untangle/web/reports
Disk Use of /tmp
7.0M /tmp
Disk Use of /var
4.0K /var/agentx
3.4M /var/backups
8.6M /var/cache
13G /var/lib
4.0K /var/local
12K /var/lock
22G /var/log
8.0K /var/mail
4.0K /var/opt
220K /var/run
3.4G /var/spool
4.0K /var/tmp
43M /var/www

[dmorris]

Hey far,

give this a try: http://wiki.untangle.com/index.php/9...sive_logrotate

I just wrote it up, but basically it just makes the logrotates much more often (since we don't really need that much data anyway)

Please report back about how it works for you as this will probably be the new default in future versions.

[far182]

Hi Dirk. Sorry for the delay. I just upgraded to 9.1 (was on 9.01, and your patch required 9.02). The box was production and we had to wait for a downtime window for the upgrade.

We are still seeing the disk fill up. Is there a new patch like the one above for 9.1?

Thank You for your support

[deleted_account+118970@untangle.com]

There is, on the wiki

[deleted_account+118970@untangle.com]

http://wiki.untangle.com/index.php/Patches#9.1

[dmorris]

Hi Dirk. Sorry for the delay. I just upgraded to 9.1 (was on 9.01, and your patch required 9.02). The box was production and we had to wait for a downtime window for the upgrade.

We are still seeing the disk fill up. Is there a new patch like the one above for 9.1?

Thank You for your support

Hey, yeah just email me your UID and I'll take a look right now. (and turn on remote support if it isn't already)

The more aggressive logrotate is built into 9.1, so if your disk is filling up its something else.
It could be the captive portal issue if this server runs captive portal and has been installed for a long time. If not, its something else.