Our shop has four indexers with limited storage. This is due to the fact that we wanted fast disk for quicker searching of the most recent data. All servers are RHEL 5.10 x64 running Splunk 6.0. I am planning on upgrading to 6.1.2 soon. Recently I noticed that we are getting the alert stating that there is only 5 GB of disk space left and indexing has been paused. This is happening on all four indexers from time to time. I have a volume configured on the indexers that when it reaches a max size to roll the warm buckets to cold (network storage). This has worked well for about 2 1/2 years until recently. I am guessing that there are other files that are outside of this volume cap that are not getting cleaned up.
I did a search for large files/directories and found the /searchpeers directory with bundles from all of the searchheads. Some of them seem somewhat old.
So enough of the back story. Here are my questions:
Any assistance with this issue would be greatly appreciated.
Thanks Dave. I believe the 5 GB limit has been a requirement for a few major releases now.
Regardless, I need to find what files/directories are growing and are not being cleaned up.
I updated the title to reflect the new issue. I was able to reclaim a good amount of drive space by removing many of the redundant log files that have been rolled. I tried to look for a Splunk logrotate config file in /etc/logrotate.d/ but there is not one.
Does anyone know where the Splunk logrotate config file is located? I would like to update it to only roll a log file one, and not five times.
If you are talking of the splunk logs, not the indexes.
The splunk logs are in $SPLUNK_HOME/var/log/splunk
This folder is also the location of the crashed and coredumps, and have to manually deleted the cores.
The splunk logs are controled by $SPLUNK_HOME/etc/log.cfg, and keep 5 copies of 25 MB each.
Yep. I cleaned up a bunch of logs in $SPLUNK_HOME/var/log/splunk. I just needed to know where the log rotate config was that Splunk used to clean up the logs. Thanks!
@yannK a few followup questions:
is there a way to push this out with the deployment-server?
will an update of splunk erase changes made to this file?
is there a way to set a global setting for any log file and not have to update settings on each log file?
1 - no , dc only write in $SPLUNKHOME/etc/apps not in $SPLUNKHOME/etc/
2 - yes, the file is contained in the installer
3 - no, it seems to be per log file
So what you are saying is that I need to create my own logrotate config and drop it in the logrotate.d directory. I don't want anymore than 1 log file rolled. 5 is way too many and diskspace is an issue. I would rather indexed data fill that space instead of logs. Doesn't the SOS app pull in and index those logs anyway?