topic Re: Why do these sub-searches error when part of a dashboard or report? in Splunk Search

Why do these sub-searches error when part of a dashboard or report?

lycollicott — Thu, 27 Oct 2016 13:08:30 GMT

We have separate indexes for 3 different applications and there are multiple instances of each application. I run the SPL below in search to list the 5 instances with the most errors in each application.

index=app_1_logs error | `instance` | stats count by instance_appl | sort 5 - count 
 | append [search index=app_2_logs error | `instance` | stats count by instance_appl | sort 5 - count]
 | append [search index=app_3_logs error | `instance` | stats count by instance_appl | sort 5 - count]
 | sort - count
| rename count as Count, instance_appl as "Instance Appl"

This works fine when I just run it in search, but when I add it to a dashboard or a report, it only returns the 5 results for the main search and nothing for the sub-searches. One of my favorite errors occurs for each subsearch Search process did not exit cleanly, exit_code=-1, description="exited with code -1".

Re: Why do these sub-searches error when part of a dashboard or report?

skoelpin — Thu, 27 Oct 2016 13:13:04 GMT

What's the job inspector say?

Re: Why do these sub-searches error when part of a dashboard or report?

lycollicott — Thu, 27 Oct 2016 13:27:43 GMT

Not really anything. I go into it and click the search.log link and nothing ever happens.

Re: Why do these sub-searches error when part of a dashboard or report?

lycollicott — Thu, 27 Oct 2016 14:03:24 GMT

Ok, I was able to get search.log to open by jumping behind the ^&%$# network ^&%$# things that ^&%$# me over every time I troubleshoot his sort of ^&%$#.

There were no errors and there was not even a mention of either sub-search

Re: Why do these sub-searches error when part of a dashboard or report?

skoelpin — Thu, 27 Oct 2016 14:13:09 GMT

Very strange.. Are you running a distributed search? If so, do you have any old versions of Splunk running? I'm also wondering if the knowledge bundles didn't get copied to the search peer.

http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/Limittheknowledgebundlesize

Re: Why do these sub-searches error when part of a dashboard or report?

lycollicott — Thu, 27 Oct 2016 14:25:33 GMT

It is distributed and everything is 6.5. I don't see any bundle errors, so I think it's getting copied.

Re: Why do these sub-searches error when part of a dashboard or report?

somesoni2 — Thu, 27 Oct 2016 14:57:35 GMT

What's the behavior when you run this query?

(index=app_1_logs error ) OR (index=app_2_logs error) OR  index=app_3_logs error ) | `instance` | top 5 instance_appl by index showperc=f | table instance_appl count |sort - count | rename count as Count, instance_appl as "Instance Appl"

Re: Why do these sub-searches error when part of a dashboard or report?

lycollicott — Thu, 27 Oct 2016 16:39:05 GMT

So much better. Thanks, dude.