Splunk Search
Highlighted

When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

Motivator

When my search runs for more than 10 min, 'job-id' expires since the default TTL value is 600 (10 min), so I get "unknown sid" in the search view.

I tried to recover the job by the option "Send job to background" - which should ideally extend the particular job-id's TTL value under the metadata.csv in dispatch folder.

However, since the folder itself gets removed, there is no possibility to reset the value, whereby there is no action in the search head when I click "Send job to background".

Is this a known issue ?

Please advise

Tags (5)
0 Karma
Highlighted

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

Esteemed Legend

When you click on "Send Job to Background, you should get a popup that asks you if you would like to be notified by email when the job is complete. Are you getting this dialog?

0 Karma
Highlighted

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

Motivator

Yes , i am getting this dialog. I choose the option "Email when complete" and provided my mail-id.
But no response after i submit.

When i close the dialog , it takes me back to the "search" window . No traces found for that particular search_id

0 Karma
Highlighted

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

Motivator

Inspecting the job:

ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/services/search/jobs/1436687594.215?message_level=debug; [{'type': 'FATAL', 'text': 'Unknown sid.', 'code': None}]
0 Karma
Highlighted

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

Esteemed Legend

Open a case with support.Splunk.com.

0 Karma
Highlighted

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

Communicator

did you delete dispatch folder? or cleared it?

if dispatch folder fills , then search takes time.

0 Karma
Highlighted

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

Motivator

no I do have enough free space , also i haven't deleted anything in dispatch folder.

I hope so , since the default TTL is 600 seconds , splunk automatically clears the "search-id" folder , and shows me "Unknown Sid" , so eventhough I send job to background , there is no traces of my job , so i get no response.

TO resolve this , eventhough the TTL is 600 , that search-id folder in dispatch directory should exist for more time, or else "Send job to background" should be disabled in splunkweb ??!

does this make sense ?

0 Karma
Highlighted

Re: When a search job runs for more than 10 minutes and the job-id expires, why doesn't the "Send job to background" option work?

SplunkTrust
SplunkTrust

The problem here is that your job expired, just as you stated. Once the Job expires, it gets deleted from the system. So if you try to send a non-existent job to background, it won't know the job, because it doesn't exists. You get the popup because you haven't refreshed the page in a while, and it's just javascript. It doesn't know the job expired. This can be considered a bug in the UI. The "Send to background" button should check first if the job is expired, and if not, then display the modal.

Opening a support case won't actually help, since it is very easy to determine the cause of this problem. I instead would open a support case to report the UI bug that the model displayed without checking the state of the job.

Additionally, don't wait 10 minutes in the UI. Send the job to background immediately if you want it to continue past 10 minutes.

Conversely, try some search optimizations and enhancements to see if you can speed up the search it self.