Possible to move Splunk's log folder ($SPLUNK_HOME/var/log/splunk) ?

[Jason]

I know a couple of Splunk's internal logs in var/log/splunk can grow to 25MB and roll up to 5 times. This can cause hundreds of megs of space to be used.

Is it possible to point Splunk at a different folder (partition, etc) for its logs?

[alsur]

I've succesfully moved only the logs by a symlink to /var/log/splunk pointing at /opt/splunk/var/log/splunk. Our /logs partition is on a separate disk.

[jrodman · Accepted solution]

The paths, sizing, and number of logs generated by splunk are mostly (though not completely) defined in $SPLUNK_HOME/etc/log.cfg Unfortunately there are some log files (eg splunkd_stderr.log) which are effectively hardcoded. Symbolic links will work to redirect the logs, but be very certain that it will be a reasonably performant and reliable storage location.

You can override these settings in $SPLUNK_HOME/etc/log-local.cfg ; for example, you could keep a smaller quantity of files for some categories, or reduce the size of some of them. For a light forwarder on a system with limited space this may be advisable. I do recommend avoiding the temptation to trim them to the bone, especially the main splunkd log, as you may want to review them at a later point to troubleshoot.

There's some somewhat dated information on this topic here: http://www.splunk.com/wiki/Community:MinimizingForwarderFootprint

[jrodman]

Yeah, sorry. There's a bit of chicken-and-egg problem around making this configurable via the bundles/apps layering system. Please do file an Enhancement Request with the background need so we can try to get smarter in the future. I think this is essentially "light forwarders really need to use somewhat less disk space for logging", but hearing how it affects each customer in their own terms really helps product management to prioritize and get things addressed. The interface to file an ER is to file a support ticket that says it is an ER.

[Jason]

Argh - etc/log-local.cfg sounds un-editable by deployment server. Not at all ideal for large number of forwarders!

[balbano]

Agree with jbslunk.

Symlinks are the way to go.

Just move your splunk home directory to wherever you want and create the symlink in /opt which points to the splunk folder. Symlink has to be called splunk.

I've done it to one or two of my splunk instances.

Hope that helps.

B

[jbsplunk]

The log files locations are hard coded. You could, however, use symlinks to trick Splunk into thinking it is generating logs in one location while writing files to the symlinked location with more space.