Splunk Search

How long does my search live in the dispatch directory

terryjohn
Path Finder

I am running Splunk 6.3 and I have a search that runs every evening and I can pick up the results the next day. The problem comes that when it runs over the weekend and I come into work on Monday I cannot see the results from Friday and Saturday.

I'm aware that the default time to live for artifacts in the dispatch directory is twice the repeat time of the search. i.e. 2 days.

To extend this I have added a line to the stanza for the search in savedsearches.conf
<pre>
dispatch.ttl = 2592000
</pre> This should extend the life up to 30 days but it seems to ignore this

The search has an email action which I believe gives me a default ttl of 24 hours. I'm not sure if this is overriding my dispatch.ttl setting.

Is there a way of changing the setting on this search that has precedence over all the other ttl defaults.

Thanks

0 Karma
1 Solution

lguinn2
Legend

Yes, the TTL setting for the alert overrides the setting in savedsearches.conf, but you should set the TTL in both places. The TTL in alert_actions.conf only applies if an alert is triggered, otherwise the TTL in savedsearches.conf applies.

In both places, you can use the p notation or just the number of seconds to save.

There are also settings for TTL in limits.conf, but those only apply to ad hoc searches.

View solution in original post

lguinn2
Legend

Yes, the TTL setting for the alert overrides the setting in savedsearches.conf, but you should set the TTL in both places. The TTL in alert_actions.conf only applies if an alert is triggered, otherwise the TTL in savedsearches.conf applies.

In both places, you can use the p notation or just the number of seconds to save.

There are also settings for TTL in limits.conf, but those only apply to ad hoc searches.

terryjohn
Path Finder

I have edited $SPLUNK_HOME/etc/system/local/alert_actions.conf and added a ttl value to the email stanza and this has changed the action.email.ttl on the job inspector. Time will tell if that is the solution I need, but I am hopeful.

I could not get this change to take effect until I restarted Splunk. I wonder if there is a less dramatic way of getting Splunk to re-read its conf files?

0 Karma

terryjohn
Path Finder

Thanks all. Works now

0 Karma

somesoni2
Revered Legend

You can try running this from Splunk CLI to reload configuration without restart.

cd /opt/splunk/bin
./splunk _internal call /configs/conf-savedsearches/_reload

and 

./splunk _internal call /configs/conf-limits/_reload

Provide the admin credentials when prompted

somesoni2
Revered Legend

The default value for the dispatch.ttl is 2p (double the search time range), could you try setting it to 4p and check? (You can change the schedule, run the job and check the job manager to see when it's expiring)

0 Karma

terryjohn
Path Finder

I couldn't find an de facto job expiring value. Just several ttl values. In view of the answer below I am looking at the action.email.ttl value

0 Karma
Get Updates on the Splunk Community!

Discover Powerful New Features in Splunk Cloud Platform: Enhanced Analytics, ...

Hey Splunky people! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2408. In this ...

Splunk Classroom Chronicles: Training Tales and Testimonials

Welcome to the "Splunk Classroom Chronicles" series, created to help curious, career-minded learners get ...

Access Tokens Page - New & Improved

Splunk Observability Cloud recently launched an improved design for the access tokens page for better ...