Splunk Search

How long does my search live in the dispatch directory

terryjohn
Path Finder

I am running Splunk 6.3 and I have a search that runs every evening and I can pick up the results the next day. The problem comes that when it runs over the weekend and I come into work on Monday I cannot see the results from Friday and Saturday.

I'm aware that the default time to live for artifacts in the dispatch directory is twice the repeat time of the search. i.e. 2 days.

To extend this I have added a line to the stanza for the search in savedsearches.conf
<pre>
dispatch.ttl = 2592000
</pre> This should extend the life up to 30 days but it seems to ignore this

The search has an email action which I believe gives me a default ttl of 24 hours. I'm not sure if this is overriding my dispatch.ttl setting.

Is there a way of changing the setting on this search that has precedence over all the other ttl defaults.

Thanks

0 Karma
1 Solution

lguinn2
Legend

Yes, the TTL setting for the alert overrides the setting in savedsearches.conf, but you should set the TTL in both places. The TTL in alert_actions.conf only applies if an alert is triggered, otherwise the TTL in savedsearches.conf applies.

In both places, you can use the p notation or just the number of seconds to save.

There are also settings for TTL in limits.conf, but those only apply to ad hoc searches.

View solution in original post

lguinn2
Legend

Yes, the TTL setting for the alert overrides the setting in savedsearches.conf, but you should set the TTL in both places. The TTL in alert_actions.conf only applies if an alert is triggered, otherwise the TTL in savedsearches.conf applies.

In both places, you can use the p notation or just the number of seconds to save.

There are also settings for TTL in limits.conf, but those only apply to ad hoc searches.

terryjohn
Path Finder

I have edited $SPLUNK_HOME/etc/system/local/alert_actions.conf and added a ttl value to the email stanza and this has changed the action.email.ttl on the job inspector. Time will tell if that is the solution I need, but I am hopeful.

I could not get this change to take effect until I restarted Splunk. I wonder if there is a less dramatic way of getting Splunk to re-read its conf files?

0 Karma

terryjohn
Path Finder

Thanks all. Works now

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You can try running this from Splunk CLI to reload configuration without restart.

cd /opt/splunk/bin
./splunk _internal call /configs/conf-savedsearches/_reload

and 

./splunk _internal call /configs/conf-limits/_reload

Provide the admin credentials when prompted

somesoni2
SplunkTrust
SplunkTrust

The default value for the dispatch.ttl is 2p (double the search time range), could you try setting it to 4p and check? (You can change the schedule, run the job and check the job manager to see when it's expiring)

0 Karma

terryjohn
Path Finder

I couldn't find an de facto job expiring value. Just several ttl values. In view of the answer below I am looking at the action.email.ttl value

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...