NSX Manager version: 6.4.1
Appeared in earlier versions too.
Issue: The /var/log is full with 100% utilization for all 3 Controllers.
root@nsx-controller [ /var/log ]# df -h | grep log
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 4.8G 4.8G 0 100% /var/log
Found that syslog.1 file occupied 4 GB of size, as a working scenario syslog file should get rotate the log file, but in our case it didn't happened, this found to be a match of a know issue in NSX 6.4.1
root@nsx-controller [ /var/log ]# ls -lah | grep syslog
-rw-r----- 1 root adm 0 Sep 18 04:03 syslog
-rw-r----- 1 root adm 4.2G Sep 22 02:39 syslog.1
-rw-r----- 1 root adm 3.7M Aug 15 22:22 syslog.10.gz
-rw-r----- 1 root adm 3.7M Aug 15 20:59 syslog.11.gz
-rw-r----- 1 root adm 3.7M Aug 15 19:36 syslog.12.gz
-rw-r----- 1 root adm 3.7M Aug 15 18:13 syslog.13.gz
Solution 1: The permanent fix for this solution is to upgrade to NSX 6.4.2
- Move the syslog.1 file to datastore for a copy of it and made the file to null.
- Make backup of existing rsyslog file cp /etc/logrotate.d/rsyslog /etc/logrotate.d/rsyslog.backup01092018
/etc/logrotate.d/rsyslog existing value "
usr/bin/systemctl reload syslog.service > /dev/null" to "
/usr/bin/systemctl restart syslog.service > /dev/null"
- Restart rsyslog:
systemctl restart rsyslog
- Check syslog now not zerro and has content