NSX Manager version: 6.4.1
Appeared in earlier versions too.
Issue: The /var/log is full with 100% utilization for all 3 Controllers.root@nsx-controller [ /var/log ]# df -h | grep logFilesystem Size Used Avail Use% Mounted on/dev/sda5 4.8G 4.8G 0 100% /var/log
Found that syslog.1 file occupied 4 GB of size, as a working scenario syslog file should get rotate the log file, but in our case it didn't happened, this found to be a match of a know issue in NSX 6.4.1
root@nsx-controller [ /var/log ]# ls -lah | grep syslog-rw-r----- 1 root adm 0 Sep 18 04:03 syslog-rw-r----- 1 root adm 4.2G Sep 22 02:39 syslog.1-rw-r----- 1 root adm 3.7M Aug 15 22:22 syslog.10.gz-rw-r----- 1 root adm 3.7M Aug 15 20:59 syslog.11.gz-rw-r----- 1 root adm 3.7M Aug 15 19:36 syslog.12.gz-rw-r----- 1 root adm 3.7M Aug 15 18:13 syslog.13.gz
Solution 1: The permanent fix for this solution is to upgrade to NSX 6.4.2
Solution 2:
- Move the syslog.1 file to datastore for a copy of it and made the file to null.
- Make backup of existing rsyslog file cp /etc/logrotate.d/rsyslog /etc/logrotate.d/rsyslog.backup01092018
- Edit /etc/logrotate.d/rsyslog existing value "usr/bin/systemctl reload syslog.service > /dev/null" to "/usr/bin/systemctl restart syslog.service > /dev/null"
- Restart rsyslog: systemctl restart rsyslog
- Check syslog now not zerro and has content