topic Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work? in Splunk Search

When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

splunker12er — Sat, 11 Jul 2015 14:53:09 GMT

When my search runs for more than 10 min, 'job-id' expires since the default TTL value is 600 (10 min), so I get "unknown sid" in the search view.

I tried to recover the job by the option "Send job to background" - which should ideally extend the particular job-id's TTL value under the metadata.csv in dispatch folder.

However, since the folder itself gets removed, there is no possibility to reset the value, whereby there is no action in the search head when I click "Send job to background".

Is this a known issue ?

Please advise

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

woodcock — Sun, 12 Jul 2015 06:53:36 GMT

When you click on "Send Job to Background, you should get a popup that asks you if you would like to be notified by email when the job is complete. Are you getting this dialog?

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

splunker12er — Sun, 12 Jul 2015 08:19:36 GMT

Yes , i am getting this dialog. I choose the option "Email when complete" and provided my mail-id.
But no response after i submit.

When i close the dialog , it takes me back to the "search" window . No traces found for that particular search_id

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

splunker12er — Sun, 12 Jul 2015 08:21:22 GMT

Inspecting the job:

ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/services/search/jobs/1436687594.215?message_level=debug; [{'type': 'FATAL', 'text': 'Unknown sid.', 'code': None}]

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

woodcock — Mon, 13 Jul 2015 01:29:33 GMT

Open a case with support.Splunk.com.

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

srisahitya_v — Mon, 13 Jul 2015 05:37:43 GMT

did you delete dispatch folder? or cleared it?

if dispatch folder fills , then search takes time.

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

splunker12er — Mon, 13 Jul 2015 06:00:34 GMT

no I do have enough free space , also i haven't deleted anything in dispatch folder.

I hope so , since the default TTL is 600 seconds , splunk automatically clears the "search-id" folder , and shows me "Unknown Sid" , so eventhough I send job to background , there is no traces of my job , so i get no response.

TO resolve this , eventhough the TTL is 600 , that search-id folder in dispatch directory should exist for more time, or else "Send job to background" should be disabled in splunkweb ??!

does this make sense ?

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

alacercogitatus — Mon, 13 Jul 2015 23:12:26 GMT

The problem here is that your job expired, just as you stated. Once the Job expires, it gets deleted from the system. So if you try to send a non-existent job to background, it won't know the job, because it doesn't exists. You get the popup because you haven't refreshed the page in a while, and it's just javascript. It doesn't know the job expired. This can be considered a bug in the UI. The "Send to background" button should check first if the job is expired, and if not, then display the modal.

Opening a support case won't actually help, since it is very easy to determine the cause of this problem. I instead would open a support case to report the UI bug that the model displayed without checking the state of the job.

Additionally, don't wait 10 minutes in the UI. Send the job to background immediately if you want it to continue past 10 minutes.

Conversely, try some search optimizations and enhancements to see if you can speed up the search it self.