Splunk Search
Highlighted

Why do these sub-searches error when part of a dashboard or report?

Motivator

We have separate indexes for 3 different applications and there are multiple instances of each application. I run the SPL below in search to list the 5 instances with the most errors in each application.

index=app_1_logs error | `instance` | stats count by instance_appl | sort 5 - count 
 | append [search index=app_2_logs error | `instance` | stats count by instance_appl | sort 5 - count]
 | append [search index=app_3_logs error | `instance` | stats count by instance_appl | sort 5 - count]
 | sort - count
| rename count as Count, instance_appl as "Instance Appl"

This works fine when I just run it in search, but when I add it to a dashboard or a report, it only returns the 5 results for the main search and nothing for the sub-searches. One of my favorite errors occurs for each subsearch Search process did not exit cleanly, exit_code=-1, description="exited with code -1".

0 Karma
Highlighted

Re: Why do these sub-searches error when part of a dashboard or report?

SplunkTrust
SplunkTrust

What's the job inspector say?

0 Karma
Highlighted

Re: Why do these sub-searches error when part of a dashboard or report?

Motivator

Not really anything. I go into it and click the search.log link and nothing ever happens.

0 Karma
Highlighted

Re: Why do these sub-searches error when part of a dashboard or report?

Motivator

Ok, I was able to get search.log to open by jumping behind the ^&%$# network ^&%$# things that ^&%$# me over every time I troubleshoot his sort of ^&%$#.

There were no errors and there was not even a mention of either sub-search

0 Karma
Highlighted

Re: Why do these sub-searches error when part of a dashboard or report?

SplunkTrust
SplunkTrust

Very strange.. Are you running a distributed search? If so, do you have any old versions of Splunk running? I'm also wondering if the knowledge bundles didn't get copied to the search peer.

http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/Limittheknowledgebundlesize

0 Karma
Highlighted

Re: Why do these sub-searches error when part of a dashboard or report?

Motivator

It is distributed and everything is 6.5. I don't see any bundle errors, so I think it's getting copied.

0 Karma
Highlighted

Re: Why do these sub-searches error when part of a dashboard or report?

SplunkTrust
SplunkTrust

What's the behavior when you run this query?

(index=app_1_logs error ) OR (index=app_2_logs error) OR  index=app_3_logs error ) | `instance` | top 5 instance_appl by index showperc=f | table instance_appl count |sort - count | rename count as Count, instance_appl as "Instance Appl"

View solution in original post

0 Karma
Highlighted

Re: Why do these sub-searches error when part of a dashboard or report?

Motivator

So much better. Thanks, dude.

0 Karma