Monitoring Splunk

Possible to move Splunk's log folder ($SPLUNK_HOME/var/log/splunk) ?

Motivator

I know a couple of Splunk's internal logs in var/log/splunk can grow to 25MB and roll up to 5 times. This can cause hundreds of megs of space to be used.

Is it possible to point Splunk at a different folder (partition, etc) for its logs?

0 Karma
1 Solution

Splunk Employee
Splunk Employee

The paths, sizing, and number of logs generated by splunk are mostly (though not completely) defined in $SPLUNK_HOME/etc/log.cfg Unfortunately there are some log files (eg splunkd_stderr.log) which are effectively hardcoded. Symbolic links will work to redirect the logs, but be very certain that it will be a reasonably performant and reliable storage location.

You can override these settings in $SPLUNK_HOME/etc/log-local.cfg ; for example, you could keep a smaller quantity of files for some categories, or reduce the size of some of them. For a light forwarder on a system with limited space this may be advisable. I do recommend avoiding the temptation to trim them to the bone, especially the main splunkd log, as you may want to review them at a later point to troubleshoot.

There's some somewhat dated information on this topic here: http://www.splunk.com/wiki/Community:MinimizingForwarderFootprint

View solution in original post

New Member

I've succesfully moved only the logs by a symlink to /var/log/splunk pointing at /opt/splunk/var/log/splunk. Our /logs partition is on a separate disk.

0 Karma

Splunk Employee
Splunk Employee

The paths, sizing, and number of logs generated by splunk are mostly (though not completely) defined in $SPLUNK_HOME/etc/log.cfg Unfortunately there are some log files (eg splunkd_stderr.log) which are effectively hardcoded. Symbolic links will work to redirect the logs, but be very certain that it will be a reasonably performant and reliable storage location.

You can override these settings in $SPLUNK_HOME/etc/log-local.cfg ; for example, you could keep a smaller quantity of files for some categories, or reduce the size of some of them. For a light forwarder on a system with limited space this may be advisable. I do recommend avoiding the temptation to trim them to the bone, especially the main splunkd log, as you may want to review them at a later point to troubleshoot.

There's some somewhat dated information on this topic here: http://www.splunk.com/wiki/Community:MinimizingForwarderFootprint

View solution in original post

Splunk Employee
Splunk Employee

Yeah, sorry. There's a bit of chicken-and-egg problem around making this configurable via the bundles/apps layering system. Please do file an Enhancement Request with the background need so we can try to get smarter in the future. I think this is essentially "light forwarders really need to use somewhat less disk space for logging", but hearing how it affects each customer in their own terms really helps product management to prioritize and get things addressed. The interface to file an ER is to file a support ticket that says it is an ER.

0 Karma

Motivator

Argh - etc/log-local.cfg sounds un-editable by deployment server. Not at all ideal for large number of forwarders!

0 Karma

Contributor

Agree with jbslunk.

Symlinks are the way to go.

Just move your splunk home directory to wherever you want and create the symlink in /opt which points to the splunk folder. Symlink has to be called splunk.

I've done it to one or two of my splunk instances.

Hope that helps.

B

Splunk Employee
Splunk Employee

The log files locations are hard coded. You could, however, use symlinks to trick Splunk into thinking it is generating logs in one location while writing files to the symlinked location with more space.

Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!