The $SPLUNK_HOME/var/spool/splunk/ directory is filling up with stash_new Files

[bpaul_splunk]

After upgrading to Splunk version 6.2.4, the $SPLUNK_HOME/var/spool/splunk/ directory starts filling up with files with the extension of .stash_new. This answers post has been reviewed, but the issue should have been fixed in version 5.0.3. Why is this occurring?

[anubhp]

Hi, I am facing a similar issue in version 7.2.3. I don't have a monitor stanza as inputs. Any idea whats the issue

[bpaul_splunk]

I would make sure to search all inputs.conf file to make sure there are no hidden monitor stanzas anywhere. You can also use btool to do this. Then I would look at the ERROR and WARN messages in the $SPLUNK_HOME/var/log/splunk/splunkd.log file to see what useful information you can find there. If you still cannot determine the cause of the directory not emptying, I would file a support ticket to get additional assistance. I would suggest attaching a diag from the instance. Also provide as much information on when the behavior started, and what changed prior to noticing the issue.

[bpaul_splunk · Accepted solution]

As stated in previous answers articles, these files are used for summary indexing. In a default installation, there is an batch stanza in the default inputs.conf file which reads the files and removes them.

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_new]
queue       = stashparsing
sourcetype  = stash_new
move_policy = sinkhole
crcSalt     = <SOURCE>

Check for a stanza which monitors the same files. In this case, it was in $SPLUNK_HOME/etc/system/local/inputs.conf.

[monitor://$SPLUNK_HOME/var/spool/splunk/...stash_new]
disabled = 0

In newer versions of Splunk, having a batch and monitor stanza on the same files will prevent them from being removed. To resolve this, remove the monitor stanza and restart splunk.