I have a dashboard (within an app) that uses the loadjob command that usually works, but now returns:
Error in 'SearchOperator:loadjob': error accessing https:///saved/searches/External API Usage/?output_mode=json, statusCode=502, description=Not Found
Any ideas what the issue could be?
I have the saved search saved in savedsearches.conf and its scheduled and running successfully
As we have a search head cluster and I wanted to use loadjob, I ran a bunch of experiments. Here are the results:
Version 6.3.3.0.1 loadjob command will only work if the 3 following requirements are respected:
1. saved search name must not include spaces
2. saved search must be added to the search app (I had to go around through many test to make sure of this issue, but if your searches are in a custom app you will get this error message: "Error in 'SearchOperator:loadjob': Cannot find artifacts for savedsearch_ident 'user:custom_app:saved_search'.
3. saved search permission must be public if you want others to see the results retrieved by this command
Also, keep in mind that the saved search must be scheduled and the loadjob call should happen within a time range where the job results haven't been automatically deleted in order for you to see any previously collected results.
As we have a search head cluster and I wanted to use loadjob, I ran a bunch of experiments. Here are the results:
Version 6.3.3.0.1 loadjob command will only work if the 3 following requirements are respected:
1. saved search name must not include spaces
2. saved search must be added to the search app (I had to go around through many test to make sure of this issue, but if your searches are in a custom app you will get this error message: "Error in 'SearchOperator:loadjob': Cannot find artifacts for savedsearch_ident 'user:custom_app:saved_search'.
3. saved search permission must be public if you want others to see the results retrieved by this command
Also, keep in mind that the saved search must be scheduled and the loadjob call should happen within a time range where the job results haven't been automatically deleted in order for you to see any previously collected results.
The search context issue is confirmed!
Loadjob is not able to pickup any saves searches outside of search app!!
Below is the issue:
Error in 'SearchOperator:loadjob': Cannot find artifacts for savedsearch_ident 'admin:JLPOperations:birdseye_rtr_base'.
The search job has failed due to an error. You may be able view the job in the Job Inspector.
Splunk 6.3.4 (build cae2458f4aef)
Yes, absolutely. Haven't been able to get back to update this but this is what I found as well. All 3 points MUST be met or it will fail. Good testing!
So for that, I give you the answer credit!
Thank you!
Glad to see that I am not the only one to reach this conclusion.
I've encountered this as well, and I believe it's related to search head clustering as it only started happening after we upgraded to 6.3. Using |savedsearch was not an option for me, because it negates the primary effect of using |loadjob, namely, that I can schedule a long-running search to execute in the background and then pick up the results when my report or dashboard needs it.
I am able to use this as a workaround:
|loadjob [rest /services/saved/searches/mysearchname/history splunk_server=local |where isDone==1 |eventstats max(ttl) as maxttl |where ttl==maxttl |return $title]
Replace mysearchname with the name of your saved search.
The path "/services/saved/searches/xx_xxx_ENV-All_Report_allT_ENV/history" did not work.
=> Permission denied. Cannot access artifacts of job_id ''.
The path "/servicesNS/nobody/xx_rp_xxx_portale/saved/searches/xx_xxx_ENV-All_Report_allT_ENV/history" worked.
|loadjob [rest /servicesNS/nobody/xx_rp_xxx_portale/saved/searches/xx_xxx_ENV-All_Report_allT_ENV/history splunk_server=local |where isDone==1 |eventstats max(ttl) as maxttl |where ttl==maxttl |return $title]
Splunk 6.3.4
This workaround works , Excellent czeller!
However you need to set global read permission for the savedsearch for rest to access it, atleast in my case .
Below errors are seen but dont affect the resultset or graph..
8 errors occurred while the search was
executing. Therefore, search results
might be incomplete. Hide errors.
[subsearch]: [SPLUNK_SERVER01] Failed
to fetch REST endpoint
uri=https://127.0.0.1:8089/services/saved/searches/my_saved_search/history?count=0
from server=https://127.0.0.1:8089
[subsearch]: [SPLUNK_SERVER02] Failed
to fetch REST endpoint
uri=https://127.0.0.1:8089/services/saved/searches/my_saved_search/history?count=0
from server=https://127.0.0.1:8089
[subsearch]: [SPLUNK_SERVER03] Failed
to fetch REST endpoint
uri=https://127.0.0.1:8089/services/saved/searches/my_saved_search/history?count=0
from server=https://127.0.0.1:8089
OK so here's the deal:
Apparently in 6.3.2 "| loadjob" for whatever reason doesn't work in the clustered search head environment. I tested this by changing the search string from:
| loadjob savedsearch="tkwaller:api-gateway-tools:External API Usage" | where appName="XXXX" | stats count by apiTarget | rename count AS NumberOfCalls | sort -NumberOfCalls
to
| savedsearch "External API Usage" | where appName="XXXX" | stats count by apiTarget | rename count AS NumberOfCalls | sort -NumberOfCalls
Everything works good
Actually this will just rerun your saved search. The whole idea of loadjob is to retrieve results from the last run of the saved search.
You may need to URL-encode your url, because it has spaces in it. Take a look here for documentation on the urlencode eval function.
It is useful but there is no urlencode there, just urldecode
So I created a new scheduled search(with no spaces) the search is:
| loadjob savedsearch="tkwaller:api-gateway-tools:External_API_Usage"
it now returns:
Error in 'SearchOperator:loadjob': Cannot find artifacts for savedsearch_ident 'tkwaller:api-gateway-tools:External_API_Usage'.
If I omit the "| loadjob" command and just use "savedsearch="tkwaller:api-gateway-tools:External_API_Usage"" I get back:
Savedsearch 'tkwaller:api-gateway-tools:External_API_Usage' does not exist or is disabled.
but it does exist and is readable and writable to Everyone