Splunk Panel delayed update with no indicator

[Anubis]

I have a search on my dashboard that takes ~20 seconds to complete.  This search is a member of a chain. base:

index=yum sourcetype=woohoo earliest=-12h@h
| table device_name field1 field2 field3

Chained search:

| search field1="$field1_tok$" AND field2="$field2_tok$"

Panel:

| stats sum(field3) as field3 by device_name
| sort - field3
| table device_name field3
| head 10

Everything works fine but when I change tokens the panel loads  a cached version of the table with incorrect values. 5 to 10 seconds later the panel updates with the correct values but without any indication. So, Is there a setting to turn off these cached results?

[bowesmana]

This is not really what chained searches are good for - what you are doing is basically loading the entire dataset into memory in your index= base search. How many events do you have - there is a limit to the number of results a base search can retain.

For this type of usage, you will often make the search slower because all the processing is done on the search head rather than using the benefit of search distribution.

The issue you are seeing I suspect will be related to the data volumes you are trying to manage through your base search.

Depending on what other searches you have, your search chain might be better expressed by this

Base search

index=yum sourcetype=woohoo earliest=-12h@h
| stats sum(field3) as field3 by field1 field2

Chain search

| search field1="$field1_tok$" AND field2="$field2_tok$"

Panel

Your original search makes no sense, because you are doing a stats command and then trying to use device_name which does not exist after the stats. Also you only have a single row after the stats, so the sort and head are pointless....

| sort - field3
| head 10
| table device_name field3

Note, you should ALWAYS put your transforming commands as late as possible in any pipeline - i.e. it is better to put the head BEFORE the table, so in a normal search you would only get 10 results from the indexer to the search head rather than sending all results and discarding all but 10.

The above will depend on what other usage you are making from your base search, but please come back with more details if you still need advice.

[Anubis]

Hello, thanks for the response. I forgot to include the 'by device_name' in my post. Sorry about that. 

1. how many events in base search: 1.6 million

2. I used the tokens in the chained search to not call the index every time a token is changed. Seemed logical

3. Putting head infront of table is better. honest mistake

[bowesmana · Accepted solution]

---

@Anubis wrote:

1. how many events in base search: 1.6 million

---

This is your problem. Although not specifically about dashboard studio, which you seem to be using as you talk about chained searches, the limit is 500,000 events.

What you are intending to do, i.e. post filter a base, is indeed logical, but there is no way you can manage 1.6 million event in a base search - see this link for a discussion on base searches - a chained search is what is referred to as a post-processing search.

https://docs.splunk.com/Documentation/Splunk/9.3.2/Viz/Savedsearches

with particular reference to Event retention and Limit base search results and post-process complexity and do not think about increasing limits.conf, that will not make things bette

You can still bump the post processing/chained search out of the base, but you need to consider each use case of your base search to work out how that post filtering can work.

If your panel searches are all doing things like stats, then move the stats down to the base search.

You can always do something like

| stats count by a b c d

and if you only want a count by c, you can then do this in your panel search.

| stats sum(c) as c

[Anubis]

Thanks for the response. I've tweaked my logic to reduce the number of lines I need in my base search making sure I do a stats in my base search before the chain. Closing this out