We have separate indexes for 3 different applications and there are multiple instances of each application. I run the SPL below in search to list the 5 instances with the most errors in each application.
index=app_1_logs error | `instance` | stats count by instance_appl | sort 5 - count | append [search index=app_2_logs error | `instance` | stats count by instance_appl | sort 5 - count] | append [search index=app_3_logs error | `instance` | stats count by instance_appl | sort 5 - count] | sort - count | rename count as Count, instance_appl as "Instance Appl"
This works fine when I just run it in search, but when I add it to a dashboard or a report, it only returns the 5 results for the main search and nothing for the sub-searches. One of my favorite errors occurs for each subsearch
Search process did not exit cleanly, exit_code=-1, description="exited with code -1".
Not really anything. I go into it and click the search.log link and nothing ever happens.
Ok, I was able to get search.log to open by jumping behind the ^&%$# network ^&%$# things that ^&%$# me over every time I troubleshoot his sort of ^&%$#.
There were no errors and there was not even a mention of either sub-search
Very strange.. Are you running a distributed search? If so, do you have any old versions of Splunk running? I'm also wondering if the knowledge bundles didn't get copied to the search peer.
It is distributed and everything is 6.5. I don't see any bundle errors, so I think it's getting copied.
What's the behavior when you run this query?
(index=app_1_logs error ) OR (index=app_2_logs error) OR index=app_3_logs error ) | `instance` | top 5 instance_appl by index showperc=f | table instance_appl count |sort - count | rename count as Count, instance_appl as "Instance Appl"