This seems to imply that if a user submits a long running (ie. 2 hour report) ah-hoc search via the gui, then performs a "send to background" (ie. adjusts job ttl to 7 days) that it will be restarted IF the search head restarts.
I was under the impression that the search head will wait for 6 mins or so before it starts killing off in process jobs.
In the server.conf there is the splunkd stop timeout.
* The maximum time, in seconds, that splunkd waits for a graceful shutdown to
complete before splunkd forces a stop.
* Default: 360 (6 minutes)
What actually happens to this (and other) long running jobs when the admin issues a "splunk restart" and users have this setting enabled?
Is the search head storing this job somewhere between shutdown and restart?
What about paused jobs with long ttl's?
I've tried pausing a search and then a search head restart.
So it appears that in certain circumstances this actually works?
Why wouldn't this be, or is it, normal behavior on shutdown for all jobs regardless of the splunkd stop timeout?
Pause all running jobs.
Restart all previously running but paused jobs?