topic Re: Why is the Splunk dispatch directory not getting cleaned up automatically, even after setting the dispatch.ttl in savedsearches.conf? in Reporting

Why is the Splunk dispatch directory not getting cleaned up automatically, even after setting the dispatch.ttl in savedsearches.conf?

Plotkowski — Mon, 23 Nov 2015 09:45:23 GMT

We run into some issues in our Splunk environment.
We have a Splunk 6.3 indexer and search head. The dispatch directory on the search head is constantly growing and Splunk stops working after a few days. We then need to manually restart the search head.
After the restart, the dispatch directory is getting cleaned up automatically and only searches from the last 24h remain.
I already set the dispatch.ttl in savedsearches.conf to 86400. (1 day) But the artifacts in the directory remain for much longer until we restart the system.

We have about 10 scheduled searches with alarms which run every hour, so it is not that much.
Is there any way to fix the automatic clean-up of the directory or what is the best way to restart the search head automatically every night on a Windows system?

Re: Why is the Splunk dispatch directory not getting cleaned up automatically, even after setting the dispatch.ttl in savedsearches.conf?

jrubio1 — Wed, 30 Dec 2015 20:31:34 GMT

I'm having the same problem as well.. Will have to setup a .bat to clear out until solution is provided.

Re: Why is the Splunk dispatch directory not getting cleaned up automatically, even after setting the dispatch.ttl in savedsearches.conf?

jkat54 — Fri, 01 Jan 2016 13:48:47 GMT

Last time this happened to me there was another error causing the problem.

Please check index=_internal for any ERROR or WARN messages, and then fix all of them you can. In my case i had a datamodel that was deleted incorrectly and splunk was pounding the logs with "data model not found" errors. AND my issue completely disabled all scheduled searches, alerts, reports, and the sendemail command. After removing the saved search that referenced the missing data model, 10000 emails went out, performance increased, and the problems went away.

The issue was that Splunk was trying so hard to find the data model, it couldnt do anything else... all the maintenance tasks like emptying the old/obsolete search bundles were failing to execute because "splunk" was too busy.

Re: Why is the Splunk dispatch directory not getting cleaned up automatically, even after setting the dispatch.ttl in savedsearches.conf?

jrubio1 — Thu, 14 Jan 2016 20:55:28 GMT

ahhh I've seen some errors.. I'll look into cleaning that up and report back.

Re: Why is the Splunk dispatch directory not getting cleaned up automatically, even after setting the dispatch.ttl in savedsearches.conf?

jkat54 — Fri, 15 Jan 2016 02:03:07 GMT

I would be suspicious of anything that could hold up a scheduler... Missing searches, data models, infinite loop conditions, extremely long running searches, broken servers missing punctuation in conf files, etc. Good news is you have the best tool for finding needles in haystacks!