How long does my search live in the dispatch directory

[terryjohn]

I am running Splunk 6.3 and I have a search that runs every evening and I can pick up the results the next day. The problem comes that when it runs over the weekend and I come into work on Monday I cannot see the results from Friday and Saturday.

I'm aware that the default time to live for artifacts in the dispatch directory is twice the repeat time of the search. i.e. 2 days.

To extend this I have added a line to the stanza for the search in savedsearches.conf
<pre>
dispatch.ttl = 2592000
</pre> This should extend the life up to 30 days but it seems to ignore this

The search has an email action which I believe gives me a default ttl of 24 hours. I'm not sure if this is overriding my dispatch.ttl setting.

Is there a way of changing the setting on this search that has precedence over all the other ttl defaults.

Thanks

[lguinn2 · Accepted solution]

Yes, the TTL setting for the alert overrides the setting in savedsearches.conf, but you should set the TTL in both places. The TTL in alert_actions.conf only applies if an alert is triggered, otherwise the TTL in savedsearches.conf applies.

In both places, you can use the p notation or just the number of seconds to save.

There are also settings for TTL in limits.conf, but those only apply to ad hoc searches.

[terryjohn]

I have edited $SPLUNK_HOME/etc/system/local/alert_actions.conf and added a ttl value to the email stanza and this has changed the action.email.ttl on the job inspector. Time will tell if that is the solution I need, but I am hopeful.

I could not get this change to take effect until I restarted Splunk. I wonder if there is a less dramatic way of getting Splunk to re-read its conf files?

[terryjohn]

Thanks all. Works now

[somesoni2]

You can try running this from Splunk CLI to reload configuration without restart.

cd /opt/splunk/bin
./splunk _internal call /configs/conf-savedsearches/_reload

and 

./splunk _internal call /configs/conf-limits/_reload

Provide the admin credentials when prompted

[somesoni2]

The default value for the dispatch.ttl is 2p (double the search time range), could you try setting it to 4p and check? (You can change the schedule, run the job and check the job manager to see when it's expiring)

[terryjohn]

I couldn't find an de facto job expiring value. Just several ttl values. In view of the answer below I am looking at the action.email.ttl value