Why do these sub-searches error when part of a dashboard or report?

[lycollicott]

We have separate indexes for 3 different applications and there are multiple instances of each application. I run the SPL below in search to list the 5 instances with the most errors in each application.

index=app_1_logs error | `instance` | stats count by instance_appl | sort 5 - count 
 | append [search index=app_2_logs error | `instance` | stats count by instance_appl | sort 5 - count]
 | append [search index=app_3_logs error | `instance` | stats count by instance_appl | sort 5 - count]
 | sort - count
| rename count as Count, instance_appl as "Instance Appl"

This works fine when I just run it in search, but when I add it to a dashboard or a report, it only returns the 5 results for the main search and nothing for the sub-searches. One of my favorite errors occurs for each subsearch Search process did not exit cleanly, exit_code=-1, description="exited with code -1".

[somesoni2 · Accepted solution]

What's the behavior when you run this query?

(index=app_1_logs error ) OR (index=app_2_logs error) OR  index=app_3_logs error ) | `instance` | top 5 instance_appl by index showperc=f | table instance_appl count |sort - count | rename count as Count, instance_appl as "Instance Appl"

[lycollicott]

So much better. Thanks, dude.

[skoelpin]

What's the job inspector say?

[lycollicott]

Ok, I was able to get search.log to open by jumping behind the ^&%$# network ^&%$# things that ^&%$# me over every time I troubleshoot his sort of ^&%$#.

There were no errors and there was not even a mention of either sub-search

[skoelpin]

Very strange.. Are you running a distributed search? If so, do you have any old versions of Splunk running? I'm also wondering if the knowledge bundles didn't get copied to the search peer.

http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/Limittheknowledgebundlesize

[lycollicott]

It is distributed and everything is 6.5. I don't see any bundle errors, so I think it's getting copied.

[lycollicott]

Not really anything. I go into it and click the search.log link and nothing ever happens.