Splunk loadjob failing when using a scheduled report?

[mbasharat]

Hi,

I have a scheduled report that is using loadjob and saved search stanza. It runs perfectly fine when running in a search, but when I schedule that using a scheduled report, it fails.

Why and how do I fix that?

Thanks in-advance.

[woodcock]

The problem is that the Time To Live ( TTL ) of the search job is too short, probably the default. You need to click on Edit -> Advanced edit and change the dispatch.ttl to something much longer.

[woodcock]

The problem might be that the user running the scheduled search is not the same as the user that owns the loadjob search. Go the the loadjob search and make sure that it has permissions set to global and read set to everyone. That should do it.

[mbasharat]

@woodcock & @burwell, thanks for prompt responses.

So I updated the permission of Job that is running for this load job to Read by Everyone and also changed the permission of load job's scheduled report referenced in load job to likewise. Failed again.

I checked the error log and it is only saying "cannot find artifacts for saved search_ident '"User:Application:ABC Report"'

When searched in Splunk search, it return results fine. Report is scheduled to run nightly so it cant expire that quick and artifacts are still there. Only above query when scheduled thru report, it fails. This is odd.

[woodcock]

My answer STINKS and is totally wrong. My answer would apply if you were using | savedsearch which you definitely could do and it would work.

[mbasharat]

What would be the correct syntax of saved search? I just wanna make sure.

Is below correct? If yes then it also did not produce any results for me

| savedsearch "My ABC Report"

I will try the TTL adjustment as well and update back.

[burwell]

Hey @mbasharat

What version of Splunk are you using?

Does your savedsearch name have spaces or other odd characters in the name?

In the past people had problems with special characters in the savedsearch name.

And I think @woodcock suggestion about ttl and the time range is important. I scheduled a job that only looked in the past minute for results with the loadjob and got the error

Error in 'SearchOperator:loadjob': Cannot find artifacts within the search time range for savedsearch_ident 'burwell:search:ABC Report'.

So

1. the search that is doing the loadjob has to be spanning a time when the ABC Report ran in.
2. If the ABC Report ttl is not current then it won't be able to find the artifacts.

[mbasharat]

Hi folks,

So SBX environment is @ 6.6.5 and another project group is @ 6.6.9. We will recently be upgrading all to 7.2.1.

Yes, the schedule search report name has spaces. Schedule report name is Sample Report 7 Days

This was the exact error I am getting like yours.

So this is what I did to test:

• Created two reports to test, one with spaced name like Report Name and second with special characters like Report_Name
• Spanned the loadjob to run for same 1 week just like report.
• Changed dispatch.ttl to same value as original report loadjob is referencing to

And both reports DID NOT FAIL this time ...... !!!!

Seems like above two issues were the culprits. Made my head spin ...... lol

You @burwell and @woodcock are AWESOME!!! TY 🙂

[woodcock]

Perfect. So pick the answer that was the most help and click Accept to close the question. If other answers or comments helped, you can should UpVote on them, too.

[woodcock]

Show us the entire search SPL.

[mbasharat]

Hi @woodcock and @burwell,

Here is the query. It runs fine in search. It fails when I schedule to run it thru scheduled report.

| loadjob savedsearch="User:Application:ABC Report"

| stats dc(BusinessUnit) AS count by BusinessUnit, Group, Project, Category

| stats dc(BusinessUnit) as count by BusinessUnit

| outputlookup business_unit.csv

[burwell · Accepted solution]

@mbasharat can you share the error?

Is it the same user that created the scheduled report that is trying to use the loadjob?

If the scheduled report is not available to all apps/users, perhaps the user cannot see the report? That is a common problem. You or Admin can change the permissions.

[mbasharat]

Hi @burwell,

Yes, it is the same user that created the job.
Report's permissions are Read for everyone.
It runs fine in Splunk search. It Fails only when the exact same query is scheduled via report.
Report query is a simple query with | outputlookup ABC.csv at the end for results to be written to a csv.

[burwell]

Hi @mbasharat is there an error in source="/opt/splunk/var/log/splunk/scheduler.log"

I tried a scheduled search with a bad loadjob command and found

02-13-2019 18:02:00.700 +0000 ERROR SavedSplunker - savedsearch_id="nobody;search;wontwork", message="Error in 'SearchOperator:loadjob': Cannot find job_id 'thiswontwork:thiswontwork:thiswontwork'.". No actions executed

[mbasharat]

Hi @woodcock and @burwell,

Here is the query. It runs fine in search. It fails when I schedule to run it thru scheduled report.

| loadjob savedsearch="User:Application:ABC Report"

| stats dc(BusinessUnit) AS count by BusinessUnit, Group, Project, Category

| stats dc(BusinessUnit) as count by BusinessUnit

| outputlookup business_unit.csv

[burwell]

And what is the error in your logs?

[mbasharat]

@woodcock & @burwell, thanks for prompt responses.

So I updated the permission of Job that is running for this load job to Read by Everyone and also changed the permission of load job's scheduled report referenced in load job to likewise. Failed again.

I checked the error log and it is only saying "cannot find artifacts for saved search_ident '"User:Application:ABC Report"'

When searched in Splunk search, it return results fine. Report is scheduled to run nightly so it cant expire that quick and artifacts are still there. Only above query when scheduled thru report, it fails. This is odd.