I found the problem because of “/var” partition are running out of space. Damn!

During my previous VMware ESX setup, the partition setup are configure as  DEFAULT which is 2G for “/var/log”.

[root@esxserver root]# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 4.9G 2.1G 2.6G 45% /
/dev/sda1 99M 28M 67M 30% /boot
/dev/sda6 2.0G 496M 1.4G 27% /var/log

Normally, /var partition won’t be filled up easily(1st time encounter this problem) and for some reasons, total of 36 /var/log/vmkernel* related files are fully occupied the /var partition.

For avoid the similar incident, I make a changes on /etc/logrotate.d/vmkernel file.

/var/log/vmkernel{
create 0600 root root
missingok
compress (originally nocompress)
# keep a history over 1 years.
monthly
rotate 12 (originally 36 for 3 years)
# max log size of 200k (thus limiting total disk usage to under 8megs)
size 200k
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript

Type # logrotate -f /etc/logrotate.d/vmkernel and forced it run on the spot.

Personally I don’t think I need to keep vmkernel log files for more than a year and for space concern, I would prefer compress/zip it. Again, the same logrotate option may apply to others log files.