Getting Data In

Where is the Splunk logrotate file located?

jodros
Builder

Our shop has four indexers with limited storage. This is due to the fact that we wanted fast disk for quicker searching of the most recent data. All servers are RHEL 5.10 x64 running Splunk 6.0. I am planning on upgrading to 6.1.2 soon. Recently I noticed that we are getting the alert stating that there is only 5 GB of disk space left and indexing has been paused. This is happening on all four indexers from time to time. I have a volume configured on the indexers that when it reaches a max size to roll the warm buckets to cold (network storage). This has worked well for about 2 1/2 years until recently. I am guessing that there are other files that are outside of this volume cap that are not getting cleaned up.

I did a search for large files/directories and found the /searchpeers directory with bundles from all of the searchheads. Some of them seem somewhat old.

So enough of the back story. Here are my questions:

  • Do knowledge bundles get cleaned up by some process?
  • If not can I delete them without any issues?
  • I noticed that all of the log files rotate 5 copies. Would there be an issue to modify the logrotate job to only keep one copy?
  • Are there any other files that others have seen that drive disk utilization up?

Any assistance with this issue would be greatly appreciated.

Thanks

1 Solution

yannK
Splunk Employee
Splunk Employee

If you are talking of the splunk logs, not the indexes.

The splunk logs are in $SPLUNK_HOME/var/log/splunk
This folder is also the location of the crashed and coredumps, and have to manually deleted the cores.

The splunk logs are controled by $SPLUNK_HOME/etc/log.cfg, and keep 5 copies of 25 MB each.

View solution in original post

yannK
Splunk Employee
Splunk Employee

If you are talking of the splunk logs, not the indexes.

The splunk logs are in $SPLUNK_HOME/var/log/splunk
This folder is also the location of the crashed and coredumps, and have to manually deleted the cores.

The splunk logs are controled by $SPLUNK_HOME/etc/log.cfg, and keep 5 copies of 25 MB each.

View solution in original post

jodros
Builder

I had to run this as a cron job every hour in order to rotate the log files before Splunk. Hopefully I will not have to increase the frequency.

0 Karma

jodros
Builder

@yannK, I made a logrotate config file to be run against all of the .log files in /opt/splunk/var/log/splunk. Should roll when a file is over 24M and only rotate 1 and compress. Went from 1.2GB to about 120MB. Nice disk size savings. I chose 24M so that my logrotate job would run against logs before the Splunk log rotate job. Do you know how often and at what times Splunk runs it's logrotate job?

0 Karma

jodros
Builder

So what you are saying is that I need to create my own logrotate config and drop it in the logrotate.d directory. I don't want anymore than 1 log file rolled. 5 is way too many and diskspace is an issue. I would rather indexed data fill that space instead of logs. Doesn't the SOS app pull in and index those logs anyway?

0 Karma

yannK
Splunk Employee
Splunk Employee

1 - no , dc only write in $SPLUNK_HOME/etc/apps not in $SPLUNK_HOME/etc/
2 - yes, the file is contained in the installer
3 - no, it seems to be per log file

0 Karma

jodros
Builder

@yannK a few followup questions:

  1. is there a way to push this out with the deployment-server?

  2. will an update of splunk erase changes made to this file?

  3. is there a way to set a global setting for any log file and not have to update settings on each log file?

Thanks

0 Karma

jodros
Builder

Yep. I cleaned up a bunch of logs in $SPLUNK_HOME/var/log/splunk. I just needed to know where the log rotate config was that Splunk used to clean up the logs. Thanks!

0 Karma

jodros
Builder

I updated the title to reflect the new issue. I was able to reclaim a good amount of drive space by removing many of the redundant log files that have been rolled. I tried to look for a Splunk logrotate config file in /etc/logrotate.d/ but there is not one.

Does anyone know where the Splunk logrotate config file is located? I would like to update it to only roll a log file one, and not five times.

Thanks

0 Karma

jodros
Builder

Bump. This issue is still happening. I would really appreciate any thoughts.

Thanks

0 Karma

davebrooking
Contributor

You maybe limited by Splunk's latest update about freespace being 5GB
http://docs.splunk.com/Documentation/Splunk/6.1.2/Installation/Systemrequirements#Recommended_hardwa...

Dave

0 Karma

jodros
Builder

Thanks Dave. I believe the 5 GB limit has been a requirement for a few major releases now.

Regardless, I need to find what files/directories are growing and are not being cleaned up.

Thanks

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!