Reporting

Splunk loadjob failing when using a scheduled report?

mbasharat
Contributor

Hi,

I have a scheduled report that is using loadjob and saved search stanza. It runs perfectly fine when running in a search, but when I schedule that using a scheduled report, it fails.

Why and how do I fix that?

Thanks in-advance.

0 Karma
1 Solution

burwell
SplunkTrust
SplunkTrust

@mbasharat can you share the error?

Is it the same user that created the scheduled report that is trying to use the loadjob?

If the scheduled report is not available to all apps/users, perhaps the user cannot see the report? That is a common problem. You or Admin can change the permissions.

View solution in original post

woodcock
Esteemed Legend

The problem is that the Time To Live ( TTL ) of the search job is too short, probably the default. You need to click on Edit -> Advanced edit and change the dispatch.ttl to something much longer.

0 Karma

woodcock
Esteemed Legend

The problem might be that the user running the scheduled search is not the same as the user that owns the loadjob search. Go the the loadjob search and make sure that it has permissions set to global and read set to everyone. That should do it.

0 Karma

mbasharat
Contributor

@woodcock & @burwell, thanks for prompt responses.

So I updated the permission of Job that is running for this load job to Read by Everyone and also changed the permission of load job's scheduled report referenced in load job to likewise. Failed again.

I checked the error log and it is only saying "cannot find artifacts for saved search_ident '"User:Application:ABC Report"'

When searched in Splunk search, it return results fine. Report is scheduled to run nightly so it cant expire that quick and artifacts are still there. Only above query when scheduled thru report, it fails. This is odd.

0 Karma

woodcock
Esteemed Legend

My answer STINKS and is totally wrong. My answer would apply if you were using | savedsearch which you definitely could do and it would work.

0 Karma

mbasharat
Contributor

What would be the correct syntax of saved search? I just wanna make sure.

Is below correct? If yes then it also did not produce any results for me

| savedsearch "My ABC Report"

I will try the TTL adjustment as well and update back.

0 Karma

burwell
SplunkTrust
SplunkTrust

Hey @mbasharat

What version of Splunk are you using?

Does your savedsearch name have spaces or other odd characters in the name?

In the past people had problems with special characters in the savedsearch name.

And I think @woodcock suggestion about ttl and the time range is important. I scheduled a job that only looked in the past minute for results with the loadjob and got the error

Error in 'SearchOperator:loadjob': Cannot find artifacts within the search time range for savedsearch_ident 'burwell:search:ABC Report'.

So

  1. the search that is doing the loadjob has to be spanning a time when the ABC Report ran in.
  2. If the ABC Report ttl is not current then it won't be able to find the artifacts.

mbasharat
Contributor

Hi folks,

So SBX environment is @ 6.6.5 and another project group is @ 6.6.9. We will recently be upgrading all to 7.2.1.

Yes, the schedule search report name has spaces. Schedule report name is Sample Report 7 Days

This was the exact error I am getting like yours.

So this is what I did to test:

  • Created two reports to test, one with spaced name like Report Name and second with special characters like Report_Name
  • Spanned the loadjob to run for same 1 week just like report.
  • Changed dispatch.ttl to same value as original report loadjob is referencing to

And both reports DID NOT FAIL this time ...... !!!!

Seems like above two issues were the culprits. Made my head spin ...... lol

You @burwell and @woodcock are AWESOME!!! TY 🙂

0 Karma

woodcock
Esteemed Legend

Perfect. So pick the answer that was the most help and click Accept to close the question. If other answers or comments helped, you can should UpVote on them, too.

0 Karma

woodcock
Esteemed Legend

Show us the entire search SPL.

0 Karma

mbasharat
Contributor

Hi @woodcock and @burwell,

Here is the query. It runs fine in search. It fails when I schedule to run it thru scheduled report.

| loadjob savedsearch="User:Application:ABC Report"

| stats dc(BusinessUnit) AS count by BusinessUnit, Group, Project, Category

| stats dc(BusinessUnit) as count by BusinessUnit

| outputlookup business_unit.csv

0 Karma

burwell
SplunkTrust
SplunkTrust

@mbasharat can you share the error?

Is it the same user that created the scheduled report that is trying to use the loadjob?

If the scheduled report is not available to all apps/users, perhaps the user cannot see the report? That is a common problem. You or Admin can change the permissions.

mbasharat
Contributor

Hi @burwell,

Yes, it is the same user that created the job.
Report's permissions are Read for everyone.
It runs fine in Splunk search. It Fails only when the exact same query is scheduled via report.
Report query is a simple query with | outputlookup ABC.csv at the end for results to be written to a csv.

0 Karma

burwell
SplunkTrust
SplunkTrust

Hi @mbasharat is there an error in source="/opt/splunk/var/log/splunk/scheduler.log"

I tried a scheduled search with a bad loadjob command and found

02-13-2019 18:02:00.700 +0000 ERROR SavedSplunker - savedsearch_id="nobody;search;wontwork", message="Error in 'SearchOperator:loadjob': Cannot find job_id 'thiswontwork:thiswontwork:thiswontwork'.". No actions executed
0 Karma

mbasharat
Contributor

Hi @woodcock and @burwell,

Here is the query. It runs fine in search. It fails when I schedule to run it thru scheduled report.

| loadjob savedsearch="User:Application:ABC Report"

| stats dc(BusinessUnit) AS count by BusinessUnit, Group, Project, Category

| stats dc(BusinessUnit) as count by BusinessUnit

| outputlookup business_unit.csv

0 Karma

burwell
SplunkTrust
SplunkTrust

And what is the error in your logs?

0 Karma

mbasharat
Contributor

@woodcock & @burwell, thanks for prompt responses.

So I updated the permission of Job that is running for this load job to Read by Everyone and also changed the permission of load job's scheduled report referenced in load job to likewise. Failed again.

I checked the error log and it is only saying "cannot find artifacts for saved search_ident '"User:Application:ABC Report"'

When searched in Splunk search, it return results fine. Report is scheduled to run nightly so it cant expire that quick and artifacts are still there. Only above query when scheduled thru report, it fails. This is odd.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...