Splunk Search

Why do these sub-searches error when part of a dashboard or report?

lycollicott
Motivator

We have separate indexes for 3 different applications and there are multiple instances of each application. I run the SPL below in search to list the 5 instances with the most errors in each application.

index=app_1_logs error | `instance` | stats count by instance_appl | sort 5 - count 
 | append [search index=app_2_logs error | `instance` | stats count by instance_appl | sort 5 - count]
 | append [search index=app_3_logs error | `instance` | stats count by instance_appl | sort 5 - count]
 | sort - count
| rename count as Count, instance_appl as "Instance Appl"

This works fine when I just run it in search, but when I add it to a dashboard or a report, it only returns the 5 results for the main search and nothing for the sub-searches. One of my favorite errors occurs for each subsearch Search process did not exit cleanly, exit_code=-1, description="exited with code -1".

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

What's the behavior when you run this query?

(index=app_1_logs error ) OR (index=app_2_logs error) OR  index=app_3_logs error ) | `instance` | top 5 instance_appl by index showperc=f | table instance_appl count |sort - count | rename count as Count, instance_appl as "Instance Appl"

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

What's the behavior when you run this query?

(index=app_1_logs error ) OR (index=app_2_logs error) OR  index=app_3_logs error ) | `instance` | top 5 instance_appl by index showperc=f | table instance_appl count |sort - count | rename count as Count, instance_appl as "Instance Appl"
0 Karma

lycollicott
Motivator

So much better. Thanks, dude.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

What's the job inspector say?

0 Karma

lycollicott
Motivator

Ok, I was able to get search.log to open by jumping behind the ^&%$# network ^&%$# things that ^&%$# me over every time I troubleshoot his sort of ^&%$#.

There were no errors and there was not even a mention of either sub-search

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Very strange.. Are you running a distributed search? If so, do you have any old versions of Splunk running? I'm also wondering if the knowledge bundles didn't get copied to the search peer.

http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/Limittheknowledgebundlesize

0 Karma

lycollicott
Motivator

It is distributed and everything is 6.5. I don't see any bundle errors, so I think it's getting copied.

0 Karma

lycollicott
Motivator

Not really anything. I go into it and click the search.log link and nothing ever happens.

0 Karma
Get Updates on the Splunk Community!

Streamline Data Ingestion With Deployment Server Essentials

REGISTER NOW!Every day the list of sources Admins are responsible for gets bigger and bigger, often making the ...

Remediate Threats Faster and Simplify Investigations With Splunk Enterprise Security ...

REGISTER NOW!Join us for a Tech Talk around our latest release of Splunk Enterprise Security 7.2! We’ll walk ...

Introduction to Splunk AI

WATCH NOWHow are you using AI in Splunk? Whether you see AI as a threat or opportunity, AI is here to stay. ...