topic Data in /opt/splunk/var/spool/splunk filling up disk in Splunk Search

Data in /opt/splunk/var/spool/splunk filling up disk

responsys_cm — Wed, 27 Feb 2013 02:33:09 GMT

I'm seeing a number of very large files building up in /opt/splunk/var/spool/splunk:

drwx------ 2 root root 4096 Feb 27 02:08 .

drwx--x--x 4 root root 4096 Feb 7 23:12 ..

-rw------- 1 root root 360903734 Feb 27 01:28 1400673619_events.stash_new

-rw------- 1 root root 372663350 Feb 27 01:53 1504785327_1400673619_events.stash_new

-rw------- 1 root root 375269359 Feb 27 02:03 157257541_1400673619_events.stash_new

-rw------- 1 root root 373008730 Feb 27 01:43 1750025097_1400673619_events.stash_new

-rw------- 1 root root 359388989 Feb 27 02:08 1874146970_1400673619_events.stash_new

-rw------- 1 root root 355854760 Feb 27 01:38 314379920_1400673619_events.stash_new

-rw------- 1 root root 375817381 Feb 27 01:33 314379920_events.stash_new

-rw------- 1 root root 372663350 Feb 27 01:48 357150606_1400673619_events.stash_new

-rw------- 1 root root 353926431 Feb 27 01:58 378307516_1400673619_events.stash_new

Is there any way I can configure Splunk so it removes them automatically or times them out? I saw an error message in the GUI that says Splunk reached the minimum disk limit for that directory. Is that value configurable? What is the impact on Splunk when that threshold is hit?

Thx.

Craig

Re: Data in /opt/splunk/var/spool/splunk filling up disk

yannK — Wed, 27 Feb 2013 02:52:36 GMT

those are summary indexing results.
the server is not picking up those files probably because they are considered as binary (check splunkd.log)

see this answer http://splunk-base.splunk.com/answers/70072/summary-indexing-blocked-and-binary-file-warning

Re: Data in /opt/splunk/var/spool/splunk filling up disk

responsys_cm — Sat, 27 Jul 2013 20:29:33 GMT

I'm not seeing anything about the binary warning in the logs. I am seeing:

BatchReader - Could not send data to output queue (stashparsing), retrying...

Re: Data in /opt/splunk/var/spool/splunk filling up disk

responsys_cm — Mon, 28 Sep 2020 14:26:47 GMT

I'm also seeing: Metrics - group=queue, name=stashparsing, max_size_kb=500, current_size_kb=449, current_size=7, largest_size=9, smallest_size=3

Is it possible to increase the size of the stashparsing queue?

Re: Data in /opt/splunk/var/spool/splunk filling up disk

yannK — Sat, 27 Jul 2013 21:49:30 GMT

Please do not play with the queues size, it will not solve the root cause.

Your issue is likely that the server (search-head I will bet) is unable to write to it's local indexes, OR is unable to forward to the indexers.
Check the indexing queue (the last one before forwarding / disk writing) on the your search-head, then on the indexers if any for any signs of congestion.