Getting Data In
Highlighted

Where is the Splunk logrotate file located?

Builder

Our shop has four indexers with limited storage. This is due to the fact that we wanted fast disk for quicker searching of the most recent data. All servers are RHEL 5.10 x64 running Splunk 6.0. I am planning on upgrading to 6.1.2 soon. Recently I noticed that we are getting the alert stating that there is only 5 GB of disk space left and indexing has been paused. This is happening on all four indexers from time to time. I have a volume configured on the indexers that when it reaches a max size to roll the warm buckets to cold (network storage). This has worked well for about 2 1/2 years until recently. I am guessing that there are other files that are outside of this volume cap that are not getting cleaned up.

I did a search for large files/directories and found the /searchpeers directory with bundles from all of the searchheads. Some of them seem somewhat old.

So enough of the back story. Here are my questions:

  • Do knowledge bundles get cleaned up by some process?
  • If not can I delete them without any issues?
  • I noticed that all of the log files rotate 5 copies. Would there be an issue to modify the logrotate job to only keep one copy?
  • Are there any other files that others have seen that drive disk utilization up?

Any assistance with this issue would be greatly appreciated.

Thanks

Highlighted

Re: Where is the Splunk logrotate file located?

Contributor

You maybe limited by Splunk's latest update about freespace being 5GB
http://docs.splunk.com/Documentation/Splunk/6.1.2/Installation/Systemrequirements#Recommended_hardwa...

Dave

0 Karma
Highlighted

Re: Where is the Splunk logrotate file located?

Builder

Thanks Dave. I believe the 5 GB limit has been a requirement for a few major releases now.

Regardless, I need to find what files/directories are growing and are not being cleaned up.

Thanks

0 Karma
Highlighted

Re: Where is the Splunk logrotate file located?

Builder

Bump. This issue is still happening. I would really appreciate any thoughts.

Thanks

0 Karma
Highlighted

Re: Where is the Splunk logrotate file located?

Builder

I updated the title to reflect the new issue. I was able to reclaim a good amount of drive space by removing many of the redundant log files that have been rolled. I tried to look for a Splunk logrotate config file in /etc/logrotate.d/ but there is not one.

Does anyone know where the Splunk logrotate config file is located? I would like to update it to only roll a log file one, and not five times.

Thanks

0 Karma
Highlighted

Re: Where is the Splunk logrotate file located?

Splunk Employee
Splunk Employee

If you are talking of the splunk logs, not the indexes.

The splunk logs are in $SPLUNK_HOME/var/log/splunk
This folder is also the location of the crashed and coredumps, and have to manually deleted the cores.

The splunk logs are controled by $SPLUNK_HOME/etc/log.cfg, and keep 5 copies of 25 MB each.

View solution in original post

Highlighted

Re: Where is the Splunk logrotate file located?

Builder

Yep. I cleaned up a bunch of logs in $SPLUNK_HOME/var/log/splunk. I just needed to know where the log rotate config was that Splunk used to clean up the logs. Thanks!

0 Karma
Highlighted

Re: Where is the Splunk logrotate file located?

Builder

@yannK a few followup questions:

  1. is there a way to push this out with the deployment-server?

  2. will an update of splunk erase changes made to this file?

  3. is there a way to set a global setting for any log file and not have to update settings on each log file?

Thanks

0 Karma
Highlighted

Re: Where is the Splunk logrotate file located?

Splunk Employee
Splunk Employee

1 - no , dc only write in $SPLUNKHOME/etc/apps not in $SPLUNKHOME/etc/
2 - yes, the file is contained in the installer
3 - no, it seems to be per log file

0 Karma
Highlighted

Re: Where is the Splunk logrotate file located?

Builder

So what you are saying is that I need to create my own logrotate config and drop it in the logrotate.d directory. I don't want anymore than 1 log file rolled. 5 is way too many and diskspace is an issue. I would rather indexed data fill that space instead of logs. Doesn't the SOS app pull in and index those logs anyway?

0 Karma