What exactly does the ttl mechanism do?

[ddrillic]

Sorry, but I don't understand how ttl is used and the reason for this design paradigm. Any ideas?

[harsmarvania57]

Hi @ddrillic,

TTL means Time To Live, generally I have seen ttl with Splunk search artifacts. When any search run there is ttl associated with that Job for example when you run adhoc search it's default ttl is 10 min which means that once job will finish their execution, it will live for another 10 min and that search artifact will be available in dispatch directory for 10min and after that it will be removed from dispatch directory so you can't access that job result after 10 min.

In Splunk when you run adhoc or schedule search there are different ttl, with schedule search ttl is depend on alert_actions as well you can check ttl for different alert actions in alert_actions.conf

Splunk Doc : https://docs.splunk.com/Documentation/Splunk/7.2.4/Search/Dispatchdirectoryandsearchartifacts#Dispat...

[ddrillic]

Ok, so, let's say I run an ad-hoc search at 1:00 pm and the search interval is for 10 minutes. Then, I run the same search at 1:03 pm - how can the original search results assist me in my second search?

[harsmarvania57]

When you run ad-hoc search at 01:00 PM and let's say it will finish in 30 seconds then that job will be available in Job Inspector until 01:10 PM and job 01:03PM will be available till 01:13PM because those are adhoc searches but here is exception if adhoc job which you ran and splunk UI is open for that job then it will not expire in 10 minute.

ttl = <integer>
* How long, in seconds, the search artifacts should be stored on disk after
  the job completes. The ttl is computed relative to the modtime of the 
  status.csv file of the job, if the file exists, or the modtime of the 
  artifact directory for the search job. 
* If a job is being actively viewed in the Splunk UI then the modtime of 
  the status.csv file is constantly updated such that the reaper does not
  remove the job from underneath.
* Default: 600 (10 minutes)

[ddrillic]

I see, so for ad-hoc searches, its usage is primarily for actively viewed searches, so the user can navigate through the pages and export the data. Right?

[harsmarvania57]

ttl applies to all searches but their values vary for different scenario as I explained earlier.

[ddrillic]

I guess that what bothers me is about the fact that these data sets that by other software applications, would be stored in memory, completely transparent to the app administrators.

Here, there is an entire infrastructure surrounding this memory set, including disk storage for this memory based data, which truly confuses us.

Maybe I miss something ; -)

[harsmarvania57]

Whenever you run any search adhoc or schedule, it generates job directory for every single job in $SPLUNK_HOME/var/run/splunk/dispatch and that directory for each job contains few files, one of them is compressed results file. Now this directory occupies storage on search head but how much, it depends on your search query, if you are displaying raw data in your query output then it will occupy more space on disk OR if you are doing statistics on raw data then it will occupy less disk storage.

So based on my understanding your confusion is that where this job data stores, in RAM (memory) or Disk ? Then answer is it is on disk. Now think about if ttl does not exist for any job then $SPLUNK_HOME/var/run/splunk/dispatch directory will continuously grow on your search head and you need to add more storage at every few days. As you will not face disk space issue on longer term for dispatch directory, ttl comes into play. It will remove jobs directory and associated files from dispatch directory after TTL expires.

[ddrillic]

Right, what I'm trying to say is that in many software applications the objects are being created in memory and when not needed, they are either being destroyed or the garbage collection process would take care of it. For me, pushing this storage to disk is a bit confusing as I haven't encountered something like that in my enterprise days and they are many ; -)

[harsmarvania57]

Yes I totally understand you but splunk does not store Job results in memory and it writes to disk & when there are too many jobs in dispatch directory it raise this error https://answers.splunk.com/answers/558452/too-many-search-jobs-found-in-the-dispatch-directo-3.html (I am not sure what is threshold in new version of Splunk)

[harsmarvania57]

Additionally for scheduled search default ttl is 2P (that is, 2 x the period of the scheduled search For example if schedule search run at every 1 hour then dispatch.ttl will be 2 hours) but this ttl will override by alert_actions.conf ttl if any alert(like email) has been fired by that scheduled search.